Auto scaling Matillion instance
We have uneven NRT workload throughout the day which may pick up in the afternoon and slows down after 6:00 PM. During pick hours, we are seeing some of the jobs (mainly orchestration jobs) are queued up because only 4 processes run in parallel (we are on m4.large). Due to this we are not able to keep up with data arrival rate. After 6:00 PM the load starts to slow down and we can process data faster than arrival rate.
My questions -
(1) How do we go about solving this issues without having to go to largest instance size, specially knowing that we need more resources only for about 6 hours out of 24 hour window ?
(2) Is there auto scaling functionality currently available or in the future road map ?
(3) Does high availability (2 - m4.large Matillion instances ) setup help with performance by distributing the workload or does it only address high availability (no workload distribution) ?
Any pointers to above are highly appreciated.