Just looking for some clarification regarding scheduled jobs in Matillion.
Consider this scenario:
I have a job that is scheduled to run every hour. However for one job run it takes over an hour and is still running.
What would be the behaviour of Matillion? Would it try to run the job even though it is still running or would it not bother running the job as it can see it is already running and await the next time it is due to run?
5 Community Answers
Laura Malins —
Matillion would actually queue the second occurrence of the job to start as soon as the first run finished.
Thanks Laura. So, theoretically, the queue could grow exponentially. Is that correct?
If so would it be better for the scheduler to not bother queuing the second occurrence of the job then await to see if the job is running when it tries running it the next time? I think that is how SQL Agent works.
There is a finite limit so the queued jobs would not go on stacking forever, but they would accumulate, yes.
If you want a job to run more or less continually – i.e. start again as soon as it has finished, then you could take a look at Matillion’s integration with Amazon SQS. This document describes how to make Matillion listen for “run this job” requests via SQS.
Then you could have a job run itself again as a final step, as shown in this document on microbatching.