Matillion ETL API - v1

Introduction

For a hierarchical map of all v1 API endpoints, please see the API v1 Map documentation.

Matillion ETL instances come with a vast selection of API options that can be accessed by adding '/rest/v1' to the end of your instance location. For example:

127.0.0.1/rest/v1

Accessing endpoints through your browser will yield metadata that will help you navigate the API. Below is a list of all top-level endpoints and their descriptions.

/ EndpointService A list of all endpoints and associated metadata.
/apiprofile APIProfileProviderService APIs concerning API profiles set up in the Matillion instance.
/audit AuditService For exporting Audit Logs from the Matillion instance.
/credential CredentialsBaseService APIs concerning the Credentials (AWS and GCP) set up in the Matillion instance.
/group GroupProviderService APIs concerning entire groups within the Matillion instance. This includes accessing data on, importing and exporting entire Matillion ETL projects.
/driver DriverProviderService APIs concerning the Database Drivers set up in the Matillion ETL instance.
/notice NoticeProviderService APIs concerning the Notice panel in the Matillion ETL client.
/oauth OAuthProviderService APIs concerning the 3rd Party OAuths set up in Matillion ETL for use with the data staging ('Query') orchestration components.
/permission PermissionBaseService APIs for viewing, importing and exporting user permissions.
/queue QueueProviderService APIs concerning the SQS ConfigurationSQS Configuration set up in the Matillion ETL instance.
/task TaskByIdService APIs concerning the status and execution of tasks in the Matillion ETL instance.
/userconfig UserConfigurationProviderService View users and import/export their configurations.
 

Exploring the v1 API

The v1 API is a large, hierarchical structure that can be explored with little/no prior knowledge of the system. Using a GET method on any endpoint will usually return a list of possible named resources associated with that endpoint. For example:

<instance address>/rest/v1/group/

Will return a list of available groups defined within the Matillion ETL instance.
A particular group can then be specified by the suffix '/name/<name>. For example, if we want to access the 'Test' group then we would use:

<instance address>/rest/v1/group/name/Test

Note that resource names are case-sensitive and must be URL encoded where appropriate (for example, when the resource name contains a space).

Whenever you reach a named resource endpoint, the API will expose API metadata for that resource, including PATH, GET and POST method options available. In the example above, the metadata would show POST options for '/password' and '/project'. Thus we could then use:

<instance address>/rest/v1/group/name/Test/project

To attain a list of projects within that group that have been defined within the Matillion ETL instance. This process can be repeated to 'dig down' into lower levels of the API, gaining access to increasingly specific resources such as individual jobs and tasks.

To illustrate this further, below is a diagram of the 'group' endpoint showing possible PATH, GET and POST options.
 

For example, to delete a specific schedule 'DatabaseBackup' using the API we could use:

POST <instance address>/rest/v1/group/name/Test/project/name/Example/schedule/name/DatabaseBackup/delete

Note that there are often multiple ways of achieving the same task within the API as resources and methods branch out and overlap.

 
 

Example - Exporting a Job

Note: Jobs imported/exported through the API are incompatible with the in-client import/export feature. If a job is exported through the API it must be imported through the API and vice-versa.

We want to export a project called 'APIExample' from our Matillion ETL project. We can explore the API as outlined in the above section to navigate to the desired project. We have thusly navigated through the Test group and found a list of jobs.

/rest/v1/group/name/Test/project/name/Test/version/name/default/job Which returns

[
 Test
 Live
 APIExport
]

We'll be exporting the APIExport project. To do so, simply name the project at the end of you API call and use the export endpoint. An example below is shown in Postman but you are free to use the API tool you find most appropriate.

The output can then be saved as a JSON, just as though you had exported the project from the Matillion ETL GUI.

Note that project exports using v1 API do not include passwords. To export projects,


Example - Importing a Job

Now that you have an exported job (see previous example), let's use the API to import that job into a Matillion ETL instance. Note that, when importing, there is no 'merge' option. If a resource of the same name already exists, you must delete the existing resource before importing the new.

Using Postman, we have set the method to POST and navigated again to the appropriate endpoint:

/rest/v1/group/name/Test/project/

Note that this time we are not naming a specific project as we hope to import one that does not yet exist. Since we are importing, the endpoint becomes...

/rest/v1/group/name/Test/project/import

In Postman, we must use a POST method API call and attach our project (in JSON form, as exported) in the body as a JSON file, shown below.

This API call will create this project in the Matillion ETL instance. Note that we have changed the project name to APIImported in the JSON, else we could get an error for importing a job whose name already exists (APIExample). This project can now be switched to in the Matillion ETL instance.

 


Example: Importing/Exporting using CURL

Note: Jobs imported/exported through the API are incompatible with the in-client import/export feature. If a job is exported through the API it must be imported through the API and vice-versa.

With reference to the above examples, the equivalent can also be achieved using CURL commands. Below we briefly outline the export and import commands for a single Matillion ETL job with the following example parameters:

Instance IP: 89.15.66.11
Group Name: Demonstration Projects
Project Name: My Project
Version Name: default
Job Name: dim_airport_setup
New Job Filename: dim_airport_setup_new.json


This will return a JSON for the dim_airport_setup job:


Job Export

curl -o dim_airport_setup.json -X GET -u api-user:api-user --insecure "http://89.15.66.11/rest/v1/group/name/Demonstration%20Projects/project/name/My%20Project/version/name/default/job/name/dim_airport_setup/export"


The job file can then be edited and renamed to dim_airport_setup_new.json (so as to avoid a name conflict with the existing job) before being imported back into Matillion ETL:

 

Job Import

curl -X POST -u api-user:api-user --insecure "http://89.15.66.11/rest/v1/group/name/Demonstration%20Projects/project/name/forename%20surname/version/name/default/job/import" -H "Content-Type: application/json" --data-binary @dim_airport_setup_new.json

Note: The '--insecure' flag is required in this case because an unsecured 'http' address is being used.



Example: Exporting/Importing an API profile.


Export an API Profile

The following snippets can export and import a profile called MyProfile and with following credentials:
  • api-user: Username for the instance (with access to the API).
  • api-password: Password for the above username.

Export an API profile
curl -X GET -o MyAPIProfile.json -u api-user:api-password https://serverip/rest/v1/apiprofile/name/MyProfile/export
 

Import an API Profile

curl -X POST -u api-user:api-password "https://server-ip/rest/v1/apiprofile/import" -H "Content-Type: application/json" --data-binary @MyAPIProfile.json
 

Deleting a resource.

 

Resources can be deleted through the v1 API by one of two methods; either by using a DELETE method with the desired resource endpoint or a POST method and appending the endpoint with /delete. For example:

 

DELETE method:

curl -X DELETE -u api-user:api-password https://server-ip/rest/v1/group/name/Test/project/name/Test/version/name/default/job/name/CopyData/delete
 

POST method:

curl -X POST -u api-user:api-password https://server-ip/rest/v1/group/name/Test/project/name/Test/version/name/default/job/name/CopyData/delete


All resources can be deleted in the same manner. The examples below show the same methods but for deleting a schedule.

DELETE method:

curl -X DELETE -u api-user:api-password https://server-ip/rest/v1/group/name/Test/project/name/Test/schedule/name/ExampleSched
 

POST method:

curl -X POST -u api-user:api-password https://server-ip/rest/v1/group/name/Test/project/name/Test/schedule/name/ExampleSched/delete

A successful deletion is usually indicated by the API returning a message similar to:

{
    "success": true,
    "msg": "Successfully deleted schedule: ExampleSched",
    "id": 0
}
 

v0 API

For more general information regarding the Matillion ETL API, including notes on the v0 API, please consult the Matillion ETL API documentation.