Best practices for developing scalable video transcoding server on Amazon Web Services?

Question

What do people think are the most important issues when developing an application that is going to allow users to upload video and images to a server and have them transcoded by FFMPEG and stored in Amazon S3? I have a couple of options;

1) install FFMPEG on the same server that handles file uploads, when a video is uploaded and stored on EC2 instance, call FFMPEG to convert it then when done, write the file to S3 bucket and dispose of the original.

How scalable is this? What happens when many users upload at the same time? How do I manage multiple processes at once? How do I know when to start another instance and load balance this configuration?

2) Have one server for processing uploads (updating database, renaming files etc) and one server for doing transcoding. Again what is the best way to manage multiple processes? should I be looking at Amazon SQS for this? Can I tell the transcoding server to get the file from the upload server or should I copy the file to the transcoding server? Should I just store all files on S3 and SQS can read from there. I am trying to have as little traffic as possible.

I am running a Linux box as the upload server and have FFMPEG running on this.

Any advice on best practices for setting up such a configuration would be appreciated.

Answer 1

I don't suppose you will need to start a new FFMPEG instance whenever somebody uploads a file for transcoding. Instead, you'll probably need to start the same number of FFMPEG processes as the number of CPUs you've got, then queue the input files you wish to transcode and do them in the order they were received. You could do this all on one computer, I don't think the server that accepts the uploads and puts them in the queue will need to take much CPU and can probably coexist just fine with the FFMPEG processes.

Depending on how huge you wish to scale to (if you want to do more than just a number of FFMPEG processes on a single machine) you'll easily make this distributed, and this is where SQS would come in handy. You could run one FFMPEG process per core, and instead of looking in a local queue for the data, it could look to the SQS. Then you could instantiate as many transcoding processes as you need, on different machines.

The downside to this is that you will need to transfer the raw videos from the server that accepts them to the server that needs to transcode them. You could put them in S3 then grab them out of S3, but I don't remember off the top of my head if you have to pay for that. Alternatively, you'll just keep them on the disk of the machine that received them, and have the transcoding process go there to get the raw files.