Scheduling Jobs

Scheduling Jobs


Matillion ETL features a scheduler that will launch orchestration jobs automatically at a pre-defined regular time interval. Schedules are set up against a project and you can have multiple schedules set up. The scheduler is straight forward to set and based on Quartz technology.

 

Creating a Schedule

The scheduler is available from the ProjectManage Schedules menu.




Scheduled jobs can be added and removed using the + and - buttons and each can be enabled/disabled with a checkbox beside each schedule.

After giving your job a name, the time of execution can be entered by giving Hours and Minutes. For example, 4:15pm would be 16 hours and 15 minutes. A job can be run multiple times by entering multiple hours or minutes separated by commas. The wildcard character, *, can be used to signify every hour or every minute in their respective fields.

The job itself is run by Matillion ETL, which is always based on UTC and is offset to local time using the Timezone dropdown menu. The 'Next Run' for all job times shows a similar time offset but in the timezone of the browser, rather than the one selected in the dropdown menu. Recent changes to timezone and/or daylight saving rules may not be reflected until you perform an update of the instance that Matillion ETL is running on. Update instructions can be found here.

Example: In the above image, the client wants to schedule data backups at 3:15am CEST. Since Matillion ETL factors in daylight savings hours automatically, CET is chosen from the dropdown timezone list to represent CEST. In the 'Test' window, Matillion ETL shows this time and also the UTC time that it has been calculated from. However, the browser window currently being used is in British Summer Time (GMT+1) and so we see a time local to this browser in the 'Next Run' field at the top of the window. All 3 times (GMT+1, CEST and UTC) are the same, but relative to different zones. 

Jobs can be scheduled to run on a weekly basis or indeed each day if all days are selected in Days Of Week. Similarly, numbers of days (separated by commas) can be entered into Days Of Month to run on a monthly basis. Use * to mean every day. Use L to indicate the last day of the month. You can also use an expression such as 15W to indicate the closest weekday to the 15th.

Note: Scheduled tasks cannot occur during the maintenance window of the EC2 cluster that Matillion ETL is running on. Inputting a time during the maintenance window will yield an error message when the schedule is tested.



The rest of the information can be selected from dropdown menus that read from the current Matillion ETL instance.
  • Version: Select the version of the project you wish to run at the scheduled time.
  • Job: Select the Orchestration job you wish to run at the scheduled time.
  • Environment: Select the environment you wish to run against at the scheduled time.

You may test the schedule by pressing the Test Schedule button. If successful, it will indicate the next time the schedule will run. The test will not actually schedule anything but instead just checks the information in the form. Jobs are not scheduled until you confirm by pressing OK.

When a schedule runs, it runs a particular Orchestration Job found within a particular Version with a particular Environment, therefore please select each of these items before saving with the OK button.

You will see the tasks run in the Tasks panel


Further information

The scheduling system is based on Quartz, a powerful scheduling system with some advanced features that are covered in more detail here.

Hint: If you choose a time-zone with daylight savings you make get unexpected results for schedules up to 3 hours after midnight on the day the clocks are adjusted.


Next Steps

Normally when a job is running automatically its desirable to have some notification as to when  its completed. The optimal way to configure this is using the SNS Component in your orchestration.