Append to a log file in S3 (or any other AWS service)

Question

I need to create a log file in AWS S3 (or any other AWS service that can help here). AFAIU there is no way to append a line to an existing log file in S3. This means that I would either need to retrieve and resend the whole log each time a new message comes, or that I will need to create a new object per message. The latter option is complicated when retrieving data.

I have lots of log messages and it is important not to lose them so it is not an option to buffer them in my server memory and send to S3 once in a while.

Which AWS service would be my best option (also in terms of saving costs).

Answer 1

What you need is an Elastic Block Store (EBS). EBS is a block storage service offered by Amazon. You can think of it as a hard disk that you can connect to your system or the virtual machine.

So, what you can do is, create an EBS volume, attach it to your instance and start logging there. if your instance is shut down or stopped, the EBS volume will still be around and you can access the data stored in it later.

When your EBS volume gets filled up to it's maximum size, you can archive the data into S3 logs.