API v1 - Running Jobs

Overview

You can use the Matillion ETL API to run any orchestration job within a Matillion ETL instance. This enables you to run jobs programmatically, via a REST API GUI, or via or a command-line interface, as described in Matillion ETL API - v1.

This article describes how to run an orchestration job from the API.

The best-practice method for running jobs as part of a pipeline is to use a messaging queue such as:

• Azure Queue Storage. Read Azure Queue Message Configuration for details.
• GCP Pub/Sub. Read Manage Pub-Sub Configuration for details.
• AWS Simple Queue Service. Read Manage SQS Configuration for details.

This endpoint requires a POST method API call, as variables for the job to be queued will have to be sent in the body of the call in JSON format.

Prerequisites

To run a job via the API, gather the following information:

• The address of your Matillion ETL instance
• The Group name
• The Project name
• The Version name
• The Job name
• Login credentials of a user who has the API Role within the Matillion ETL instance

The API "run" URL

The API URL must fully describe the job to be run, as follows:

http://{instanceAddress}/rest/v1/group/name/{groupName}/project/name/{projectName}/version/name/{versionName}/job/name/{jobName}/run?environmentName={environmentName}

POST body

Any variables required by the job must be sent in the body of the API call in JSON format. Two types of variable can be sent: Job Variables and Grid Variables.

The JSON required to pass job (scalar) and grid variables has the following format:

{
"scalarVariables": {
"{variable-name-1}": "{value-1}", "{variable-name-2}": "{value-2}"
},
"gridVariables": {
"{grid-variable-name}": [
[
"{column-value-1}",
"{column-value-2}",
"{column-value-3}"
]
]
}
}

Where:

• {variable-name} is the name of a job variable.
• {value} is the value that will be assigned to the job variable when the job executes.
• {grid-variable-name} is the name of a grid variable.
• {column-value} is a value that will be assigned to a grid variable column when the job executes. Multiple values can be passed, separated by commas, and they will be assigned to columns in the order that the columns have been defined in the grid variable. So the first value listed in the JSON post body goes into the grid variable's first column, the second listed value goes into the second column, etc.

You can pass any number of job variables in the scalarVariables statement, separating each "{variable-name}": "{value}" pair with a comma. All job variables must be pre-defined in Matillion ETL before you can pass values via the API. If you attempt to pass a value to a variable which doesn't exist, the API call will execute and schedule the job, but the job will ignore the passed value.
The grid variable and all of its columns must be pre-defined in Matillion ETL before you can pass values via the API. If you attempt to pass values to a grid variable which doesn't exist, the API call will execute and schedule the job, but the job will ignore the passed values. However, if the variable does exist then you must pass exactly one value for every column in the grid. If you pass too many or too few values, the job will fail to execute.

Server response

On completion of the API call, Matillion ETL will return a response of the following form:

{
"success": true,
"msg": "Successfully queued job apiTest",
"id": 33060
}

The response doesn't indicate that the job has completed, just that it has been queued to run. You can verify that the job has run and completed successfully (or otherwise) by looking at the Tasks Panel or the Task History in the Matillion ETL instance. You can also obtain this information programmatically via the API using the Tasks API via the task id from the server response.