des services web d'Amazon - Auto mise à l'échelle de traitements par lots sur Elastic Beanstalk - Stack Overflow

I am trying to set up a scalable background image processing using beanstalk. My setup is the following:

• Application server (running on Elastic Beanstalk) receives a file, puts it on S3 and sends a request to process it over SQS.


• Worker server (also running on Elastic Beanstalk) polls the SQS queue, takes the request, load original image from S3, processes it resulting in 10 different variants and stores them back on S3.


• These upload events are happening at a rate of about 1-2 batches per day, 20-40 pics each batch, at unpredictable times.

Problem: I am currently using one micro-instance for the worker. Generating one variant of the picture can take anywhere from 3 seconds to 25-30 (it seems first ones are done in 3, but then micro instance slows down, I think this is by its 2 second bursty workload design). Anyway, when I upload 30 pictures that means the job takes: 30 pics * 10 variants each * 30 seconds = 2.5 hours to process??!?!

Obviously this is unacceptable, I tried using "small" instance for that, the performance is consistent there, but its about 5 seconds per variant, so still 30*10*5 = 26 minutes per batch. Still not really acceptable.

What is the best way to attack this problem which will get fastest results and will be price efficient at the same time?

Solutions I can think of:

1. Rely on beanstalk auto-scaling. I've tried that, setting up auto scaling based on CPU utilization. That seems very slow to react and unreliable. I've tried setting measurement time to 1 minute, and breach duration at 1 minute with thresholds of 70% to go up and 30% to go down with 1 increments. It takes the system a while to scale up and then a while to scale down, I can probably fine tune it, but it still feels weird. Ideally I would like to get a faster machine than micro (small, medium?) to use for these spikes of work, but with beanstalk that means I need to run at least one all the time, since most of the time the system is idle that doesn't make any sense price-wise.




2. Abandon beanstalk for the worker, implement my own monitor of of the SQS queue running on a micro, and let it fire up larger machine(or group of larger machines) when there are enough pending messages in the queue, terminate them the moment we detect queue is idle. That seems like a lot of work, unless there is a solution for this ready out there. In any case, I lose the benefits of beanstalk of deploying the code through git, managing environments etc.

I don't like any of these two solutions Is there any other nice approach I am missing?

Thanks

CPU utilization on a micro instance is probably not the best metric to use for autoscaling in this case.

Length of the SQS queue would probably be the better metric to use, and the one that makes the most natural sense.

Needless to say, if you can budget for a bigger base-line machine everything would run that much faster.

I am trying to set up a scalable background image processing using beanstalk. My setup is the following:

• Application server (running on Elastic Beanstalk) receives a file, puts it on S3 and sends a request to process it over SQS.


• Worker server (also running on Elastic Beanstalk) polls the SQS queue, takes the request, load original image from S3, processes it resulting in 10 different variants and stores them back on S3.


• These upload events are happening at a rate of about 1-2 batches per day, 20-40 pics each batch, at unpredictable times.

Problem: I am currently using one micro-instance for the worker. Generating one variant of the picture can take anywhere from 3 seconds to 25-30 (it seems first ones are done in 3, but then micro instance slows down, I think this is by its 2 second bursty workload design). Anyway, when I upload 30 pictures that means the job takes: 30 pics * 10 variants each * 30 seconds = 2.5 hours to process??!?!

Obviously this is unacceptable, I tried using "small" instance for that, the performance is consistent there, but its about 5 seconds per variant, so still 30*10*5 = 26 minutes per batch. Still not really acceptable.

What is the best way to attack this problem which will get fastest results and will be price efficient at the same time?

Solutions I can think of:

1. Rely on beanstalk auto-scaling. I've tried that, setting up auto scaling based on CPU utilization. That seems very slow to react and unreliable. I've tried setting measurement time to 1 minute, and breach duration at 1 minute with thresholds of 70% to go up and 30% to go down with 1 increments. It takes the system a while to scale up and then a while to scale down, I can probably fine tune it, but it still feels weird. Ideally I would like to get a faster machine than micro (small, medium?) to use for these spikes of work, but with beanstalk that means I need to run at least one all the time, since most of the time the system is idle that doesn't make any sense price-wise.




2. Abandon beanstalk for the worker, implement my own monitor of of the SQS queue running on a micro, and let it fire up larger machine(or group of larger machines) when there are enough pending messages in the queue, terminate them the moment we detect queue is idle. That seems like a lot of work, unless there is a solution for this ready out there. In any case, I lose the benefits of beanstalk of deploying the code through git, managing environments etc.

I don't like any of these two solutions Is there any other nice approach I am missing?

Thanks

CPU utilization on a micro instance is probably not the best metric to use for autoscaling in this case.

Length of the SQS queue would probably be the better metric to use, and the one that makes the most natural sense.

Needless to say, if you can budget for a bigger base-line machine everything would run that much faster.

Source