AWS Kinesis Connector Library

Question

I am working on a real-time streaming application that needs to send information to AWS Kinesis streams and from there to AWS Redshift. Based on my understanding as per the documentation, the following are the option to push information from Kinesis Stream to Redshift:

1. Kinesis Streams -> Lambda Function -> Redshift

2. Kinesis Streams -> Lambda Function -> Kinesis Firehose -> Redshift

3. Kinesis Streams -> Kinesis Connector Library -> Redshift

It seems the Kinesis Connector option to be the best option for moving information from Streams to Redshift. But, I am not able to understand where do we deploy this library and how does this runs? Does this need to run as a lambda function or as a java function on an EC2 instance? In case anyone can help me with this!!

Answer 1

If you're using the Kinesis Connector Library and then you want to deploy it on an EC2 instance, but using a Lambda function without the Connector Library is a lot easier and would be a better option as per my opinion. As it handles batching, scaling up your instance invocation and retries. Dead Letter Queues are probably coming soon too for Lambda + Kinesis.

Basically, it's a lot easier to scale and deal with failures in Lambda.

Want to learn more about AWS, then do check out AWS Course offered by Intellipaat.