Should I run Glue crawler everytime to fetch latest data?

Question

I have an S3 bucket, every 3 hours I get a file in the bucket with a timestamp attached to it. I'm using Glue job for moving the files from S3 to Redshift with some transformations. The Glue job uses the table created in Data Catalog via crawler as he input.

First, run: 

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "test", table_name = "employee_623215", transformation_ctx = "datasource0")

After three hours if I get one more file on the S3 bucket should I crawl it again?

Is there any way to have a single table in Data Catalog and update the table with the latest S3 file which can be used by Glue Job for processing or I need to run crawler every time to get the latest data?

Answer 1

As per my understanding, an alternative approach can be, instead of reading from the catalog read directly from S3 and process data in the Glue job.

Use the below command to run the crawler again:

from_options(connection_type, connection_options={}, format=None, format_options={}, transformation_ctx="")

For more details, you can read the documentation here. 

Do you want to master AWS, then do checkout the AWS Course by Intellipaat.