Matillion ETL for Redshift features a scheduler that will launch orchestration jobs automatically at a pre-defined regular time interval. Schedules are set up against a project and you can have multiple schedules set up. The scheduler is straight forward to set and based on Quartz technology
Creating a ScheduleThe scheduler is available from the Project → Manage Schedules menu.
Scheduled jobs can be added and removed using the + and - buttons and each can be enabled/disabled with a checkbox beside each schedule.
After giving your job a name, the time of execution can be entered by giving Hours and Minutes. For example, 4:15pm would be 16 hours and 15 minutes.
A job can be run multiple times by entering multiple hours or minutes separated by commas. The wildcard character, *, can be used to signify every hour or every minute in their respective fields.
All job times are given as GMT (+0) and must be offset to local time using the Timezone dropdown menu. Recent changes to timezone and/or daylight saving rules may not be reflected until you perform an update of the instance that Matillion ETL for Redshift is running on. Update instructions can be found here.
Jobs can be scheduled to run on a weekly basis or indeed each day if all days are selected in Days Of Week. Similarly, numbers of days (separated by commas) can be entered into Days Of Month to run on a monthly basis. Use * to mean every day. Use L to indicate the last day of the month. You can also use an expression such as 15W to indicate the closest weekday to the 15th.
Note: Scheduled tasks cannot occur during the maintenance window of the Redshift cluster that Matillion ETL for Redshift is running on. Inputting a time during the maintenance window will yield an error message when the schedule is tested.
The rest of the information can be selected from dropdown menus that read from the current Matillion ETL for Redshift instance.
- Version: Select the version of the project you wish to run at the scheduled time.
- Job: Select the Orchestration job you wish to run at the scheduled time.
- Environment: Select the environment you wish to run against at the scheduled time.
You may test the schedule by pressing the Test Schedule button. If successful, it will indicate the next time the schedule will run. The test will not actually schedule anything but instead just checks the information in the form. Jobs are not scheduled until you confirm by pressing OK.
When a schedule runs, it runs a particular Orchestration Job found within a particular Version with a particular Environment, therefore please select each of these items before saving with the OK button.
You will see the tasks run in the Tasks panel
The scheduling system is based on Quartz, a powerful scheduling system with some advanced features that are covered in more detail here.
Hint: If you choose a time-zone with daylight savings you make get unexpected results for schedules up to 3 hours after midnight on the day the clocks are adjusted.