Reading data from S3 using Lambda

Question

I have a range of JSON files stored in an S3 bucket on AWS.

I wish to use AWS lambda python service to parse this JSON and send the parsed results to an AWS RDS MySQL database.

I have a stable python script for doing the parsing and writing to the database. I need to lambda script to iterate through the JSON files (when they are added).

Each json file contains a list, simple consisting of results = [content]

In pseudo-code what I want is:

Connect to the S3 bucket (jsondata)

Read the contents of the JSON file (results)

Execute my script for this data (results)

I can list the buckets I have by:

import boto3

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():

    print(bucket.name)

Giving:

jsondata

But I cannot access this bucket to read its results.

There doesn't appear to be a read or load function.

I wish for something like

for bucket in s3.buckets.all():

   print(bucket.contents)

 

I am misunderstanding something. Rather than reading the file in S3, lambda must download it itself.

From here it seems that you must give lambda a download path, from which it can access the files itself

import libraries

s3_client = boto3.client('s3')

def function to be executed:

   content

def handler(event, context):

    for record in event['Records']:

        bucket = record['s3']['bucket']['name']

        key = record['s3']['object']['key'] 

        download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)

        s3_client.download_file(bucket, key, download_path)

Answer 1

To get a list of the buckets you can use bucket.objects.all(). Also, these are some alternative methods - filter, page_size and limit. These methods will return an iterator with S3.ObjectSummary objects in it. You can use object.get to retrieve the file after that.

You can learn more about AWS Lambda and Amazon Web Services on AWS Tutorial.