Say I am running Matillion for Amazon Redshift, hosted on EC2. I am aware that there is 2 options for billing: on-demand and annual.
I like the idea of the on-demand basis. However, I'm not sure how I will be billed if I am using Matillion scheduling. E.g. If I schedule a process/job daily, and it takes 2 hours to run the job, will Matillion technically be running for 24 hours per day, and hence charge me for 24 hours? or will I just be charged for the 2 hours that Matillion was actually processing data?
4 Community Answers
Kevin Havice —
I've used on-demand billing. You will be charged for 24 hours, unless you go into the AWS console and "Stop" the EC2 instance.
Then, when you want to use Matillion again, you go into the AWS console and manually start it back up again.
I found that starting/stopping the instance, so that it was "on" (and billable), worked fine during early development stages. Once I had created some Matillion jobs, and I was scheduling the jobs to run day and night, it was not feasible any more to stop/start the EC2 instance; I needed it to be on all the time.
With the on-demand option, you can choose exactly when you want to use Matillion. From the EC2 console you can choose to Stop the instance with one click. Then it will shut down gracefully, and there is no more runtime charge.
Later, you can choose to Start the instance again from the EC2 console. It will take a couple of minutes to resume, and there will be no loss of state. I.e. all the jobs you built beforehand will still be there. The hourly on-demand charge will resume.
If you have scheduled a job to run at a specific time, and Matillion is in a Stopped state when the scheduled time arrives, the job will simply not run.
If you shut down the Matillion instance while a job is running, then it will be killed partway through. Database connections will be terminated abruptly and any executing SQL statements or transactions will rollback.
Note that Matillion has an alternative scheduling mechanism, using SQS
In this case, if a job request is enqueued while Matillion is Stopped, the request will simply remain in the queue and will start to run the next time Matillion is started again.
(So, that is to say, if you schedule a 2-hour job, you have to ensure the EC2 instance is started and running for the duration of that job. Then, to avoid hourly charges when you aren't using it, you have to manually stop the instance after the job is done. Then, you have to manually start the instance again before the job is scheduled to begin, again.
As far as I know, there is no way to automatically "schedule the EC2 instance" to be running during certain times, and stopped during certain times. But maybe there is some utility out there for that?)
AWS Lambda has the ability to run a serverless task on a regular, scheduled basis so I think you would be able to use it to automatically start and stop your Matillion EC2 instance at predefined times.