What's a good way to collect logs from Amazon EC2 instances?

Question

My app is hosted on an Amazon EC2 cluster. Each instance writes events to log files. I need to collect (and data mine) over these logs at the end of each day. What's a recommended way to collect these logs in a central location? I have thought of several options, not sure which way to go:

1. scp them to an instance using a cron job
2. Log all events over TCP/IP to an instance

Answer 1

We use Logstash on every host (deployed via Puppet) to collect and ship log events to a message queue (RabbitMQ, but can be Redis) on a central host, while another Logstash instance will retrieve the events, processes them and stuffs the result into ElasticSearch. A Kibana web interface is used to search through this database.

It's very capable, scales easily and is incredibly flexible. Logstash has a lot of filters to process events from various inputs and can output to plenty of services, ElasticSearch being one among them. On light hardware, we currently ship about 1 or 2 million log events per day from the EC2 instances. In our setup, the latency for a log event from event to searchable is 1 second.

Here's some documentation on this sort of setup: https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html, and a demo of the Kibana search interface with some live data.