Why?
As part of our cost reduction at Cisco Cloud Security, we are always expected to find ways to reduce our expenses from the obvious reasons.
Circumstances
We normally use CloudWatch metrics to increase and decrease the number of running containers. This is fine as long as the load distribution over time is unpredictable, and the peaks are long enough to actually make use of the new allocated resources.
The problem Short predictable loads over API endpoints, cause more containers to be launched, but the time it takes them to be added to the cluster is too long for them to join the effort.
The solution AWS released in late 2017 application scheduled scaling for ECS, we decided to give it a go.
The first service we used it for was quite simple to implement,
we created a new terraform resource that will be used across all of our services using aws_appautoscaling_scheduled_action:
when scheduling takes ordinary crontab expression such as:
schedule_up = "cron(30 5 * * ? *)"
Just like this new resource that we created, we created a new resource for schedule_down that takes crontab expression to change the resource back to the default
Improving the solution for multiple predictable high load points during the day with varying capacity
We reached a new challenge - multiple times a day where there are peak in API usage, where we want to add more containers to the cluster.
Here we used an array of maps:
As you can see, for each for the points where we scale we have different target capacity.
At the resource we handle the array of maps:
I would like to thanks Keitaroh Kobayashi and her following blog post