For better or worse, we have migrated our whole LAMP web application from dedicated machines to the cloud (Amazon EC2 machines). It's going great so far but the way we do crons is sub-optimal. I have an Amazon-specific question about how to best manage cron jobs in the cloud using "the Amazon way".
The problem: We have multiple web servers, and need to run crons for batch jobs such as creating RSS feeds, triggering emails, many different things actually. BUT the cron jobs need to only run on one machine because they often write to the database so would duplicate the results if run on multiple machines.
So far, we designated one of the webservers as the "master-webserver" and it has a few "special" tasks that the other web servers don't have. The trade-off for cloud computing is reliability - we don't want a "master-webserver" because it's a single point of failure. We want them to all be identical and to be able to upscale and downscale without remembering not to take the master-webserver out of the cluster.
How can we redesign our application to convert Linux cron jobs into transitory work items that don't have a single point of failure?
My ideas so far:
- Have a machine dedicated to only running crons. This would be a little more manageable but would still be a single-point-of-failure, and would waste some money having an extra instance.
- Some jobs could conceivably be moved from Linux crons to MySQL Events however I'm not a big fan of this idea as I don't want to put application logic into the database layer.
- Perhaps we can run all crons on all machines but change our cron scripts so they all start with a bit of logic that implements a locking mechanism so only one server actually takes action and the others just skip. I'm not a fan of this idea as it sounds potentially buggy and I would prefer to use an Amazon best-practice rather than rolling our own.
- I'm imagining a situation where jobs are scheduled somewhere, added to a queue and then the web servers could each be a worker, that can say "hey, I'll take this one". Amazon Simple Workflow Service sounds exactly this kind of thing but I don't currently know much about it so any specifics would be helpful. Does it seem kind of heavy-weight for something as simple as a cron? Is it the right service or is there a more suitable Amazon service?
... > Cron jobs In this sample, a long-running workflow periodically executes an activity. The ability to continue executions as new executions so that execution can run for very extended periods of time is demonstrated. ...
I downloaded the AWS SDK for Java (http://aws.amazon.com/sdkforjava/) and sure enough buried within ridiculous layers of folders there is some java code (aws-java-sdk-1.3.6/samples/AwsFlowFramework/src/com/amazonaws/services/simpleworkflow/flow/examples/periodicworkflow).
The problem is, if I'm honest, this doesn't really help as it's not something I can easily digest with my skillset. The same sample is missing from the PHP SDK and there doesn't seem to be a tutorial that walks through the process. So basically, I'm still hunting for advice
Back