Google Colab: how to read data from my google drive?

Question

I am having some data on my google drive, let's say at /projects/my_project/my_data*. And also, I have a simple notebook file in google colab.

So, I would like to do something like:

for file in glob.glob("/projects/my_project/my_data*"):

    do_something(file)

Unfortunately, all examples suggest only mainly loading all necessary data to the notebook.

But, if I am having a huge amount of data, it can be quite complicated. Are there any opportunities to solve this issue?

Answer 1

 Kindly be informed that the PyDrive is now having first-class support on CoLab! PyDrive is a cover for the Google Drive python client. Here is an example of how you would download ALL files from a folder, similar to using glob + *:

!pip install -U -q PyDrive

import os

from pydrive.auth import GoogleAuth

from pydrive.drive import GoogleDrive

from google.colab import auth

from oauth2client.client import GoogleCredentials

# 1. Authenticate and create the PyDrive client.

auth.authenticate_user()

gauth = GoogleAuth()

gauth.credentials = GoogleCredentials.get_application_default()

drive = GoogleDrive(gauth)

# choose a local (colab) directory to store the data.

local_download_path = os.path.expanduser('~/data')

try:

  os.makedirs(local_download_path)

except: pass

# 2. Auto-iterate using the query syntax

#    https://developers.google.com/drive/v2/web/search-parameters

file_list = drive.ListFile(

    {'q': "'1SooKSw8M4ACbznKjnNrYvJ5wxuqJ-YCk' in parents"}).GetList()

for f in file_list:

  # 3. Create & download by id.

  print('title: %s, id: %s' % (f['title'], f['id']))

  fname = os.path.join(local_download_path, f['title'])

  print('downloading to {}'.format(fname))

  f_ = drive.CreateFile({'id': f['id']})

  f_.GetContentFile(fname)

with open(fname, 'r') as f:

  print(f.read())

See that the parameters to drive.ListFile is a dictionary format data that coincides with the arguments used by Google Drive HTTP API.

Looking for a Python Tutorial? Join the Intellipaat's Python Course to gain more knowledge on Python.