Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Matillion Job Scheduling

Hello

Just looking for some clarification regarding scheduled jobs in Matillion.

Consider this scenario:

I have a job that is scheduled to run every hour. However for one job run it takes over an hour and is still running.

What would be the behaviour of Matillion? Would it try to run the job even though it is still running or would it not bother running the job as it can see it is already running and await the next time it is due to run?

5 Community Answers

Matillion Agent  

Laura Malins —

Hi Mark

Matillion would actually queue the second occurrence of the job to start as soon as the first run finished.

Thanks
Laura


Mark Stone —

Thanks Laura. So, theoretically, the queue could grow exponentially. Is that correct?

If so would it be better for the scheduler to not bother queuing the second occurrence of the job then await to see if the job is running when it tries running it the next time? I think that is how SQL Agent works.


Matillion Agent  

Ian Funnell —

Hi Mark,

There is a finite limit so the queued jobs would not go on stacking forever, but they would accumulate, yes.

If you want a job to run more or less continually – i.e. start again as soon as it has finished, then you could take a look at Matillion’s integration with Amazon SQS. This document describes how to make Matillion listen for “run this job” requests via SQS.

Then you could have a job run itself again as a final step, as shown in this document on microbatching.

Best regards,
Ian


Mark Stone —

Thanks Ian, what is the limit on the queued jobs?


Matillion Agent  

Ian Funnell —

Hi Mark,

The limit is currently 16, although it’s the kind of thing we may tweak occasionally in line with other performance tuning and updates.

Best regards,
Ian

Post Your Community Answer

To add an answer please login