Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Auto scaling Matillion instance

We have uneven NRT workload throughout the day which may pick up in the afternoon and slows down after 6:00 PM. During pick hours, we are seeing some of the jobs (mainly orchestration jobs) are queued up because only 4 processes run in parallel (we are on m4.large). Due to this we are not able to keep up with data arrival rate. After 6:00 PM the load starts to slow down and we can process data faster than arrival rate.

My questions -
(1) How do we go about solving this issues without having to go to largest instance size, specially knowing that we need more resources only for about 6 hours out of 24 hour window ?
(2) Is there auto scaling functionality currently available or in the future road map ?
(3) Does high availability (2 - m4.large Matillion instances ) setup help with performance by distributing the workload or does it only address high availability (no workload distribution) ?

Any pointers to above are highly appreciated.

Thank you,
Devang.

1 Community Answers

Matillion Agent  

Laura Malins —

Hi Devang

My first thought is you could use the AWS cli in a bash script to run a command which will stop the instance and change the type at certain times of the day. You’d need to run this as a script outside of Matillion as it invovles switching Matillion on and off. However this could be scheduled to meet your time frames.

You will need to use the modify-instance-attirbute:
https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html

To my knowledge there is no auto scaling functionailty on the roadmap. Certainly nothing is available now. I will speak to our development team about this.

High availability will help with your issue as some jobs will be managed by different servers.

Thanks
Laura

Post Your Community Answer

To add an answer please login