For some customers the best practice topology is to have multiple instances of Matillion ETL (e.g. for Development, Test and Live) and then manage the migration using source control management system combined with a build server or script environment. This allows Matillion ETL to support DevOps scenarios where all configuration is held centrally.
Below, we outline a project called “Marketplace Data Warehouse” in a group called “Matillion”. The job is called “Load Marketplace Data” and exists in the version “BW_21-7-2016”. The following steps will guide you through a typical application of SCM to Matillion ETL.
Note: The following steps are generic and can be adapted for just about any source control management system. For the sake of example, we outline the below with respect to GIT, however videos for other SCM systems are given at the bottom of the page.
Part 1 - Export a project and commit to source control
Exporting all the data for a given version is simple. Identify the version to export, and ensure all the jobs are included in the result with export=true. (The URL is quoted to protect the & characters being interpreted by bash.)
curl -o <filename> -X GET -u <user>:<password> -H "Content-Type: application/json" "https://server-name-or-ip/rest/v0/projects?groupName=<groupname>&projectName=<Project Name>&versionName=<Version Name>&export=true"
curl -o MatillionMarketplace.json -X GET -u <user>:<password> -H "Content-Type: application/json" "http://10.12.1.28/rest/v0/projects?groupName=Matillion&projectName=Marketplace%20Data%20Warehouse&versionName=BW_21-7-2016&export=true"
If you want to output formatted Json you can use the jq command. e.g.
curl -X GET -u <user>:<password> -H "Content-Type: application/json" "http://10.12.1.28/rest/v0/projects?groupName=Matillion&projectName=Marketplace%20Data%20Warehouse&versionName=BW_21-7-2016&export=true" | jq ‘.’ > MatillionMarketplace.json
Part 2 - To commit to GIT
To begin with GIT, create a workspace:
mkdir MatillionWorkspaceChange into the directory:
cd MatillionWorkspaceInitialise the git repository:
git initCreate a branch to match the version in Matillion ETL:
git checkout -b BW_21-7-2016Add the files retrieved from the API:
git add *Commit the files:
git commit -m "Matillion Marketplace initial commit"Push changes to the repository:
git remote add origin https://email@example.com/matillion/matillion-etl-scm.git git push origin --all
Part 3 - Checkout a project from source control and push to a new Matillion ETL instance
Continuing from part 2, here we will be checking-out the Matillion ETL project version we checked-in earlier.
Check out the project from SCM. For example:
git clone https://firstname.lastname@example.org/matillion/matillion-etl-scm.gitChange into the project:
cd matillion-etl-scm/Ensure we are looking at the correct branch (or version):
git branchTo push the checked-out JSON data into a new server:
curl -X POST http://server-name-or-ip/rest/v0/projects -H "Content-Type: application/json" --data-binary @JsonFile.jsone.g.
curl -X -u <user>:<password> POST http://10.12.2.229/rest/v0/projects -H "Content-Type: application/json" --data-binary @MatillionMarketplace.jsonThe return value shows that the operation was a success. An invalid ID is returned from this call, since this operation generates many new ID’s (for the project, environment, version, jobs…) - you can see the current project structure by doing a GET request to the same URL but without the export=true option.