Is there an intention to have only 1 JSON format regardless of we export the Job manually through Matillion's UI or we export the Job from Matillion's API?
The current solution is inconvenient since we can't deploy automatically (via API) to Test and Prod environments whatever has been commited in our Version Control System. We have to export the same Job from Dev first but that Job might be already different than the one manually exported. This is just an example, because there are other inconveniences.
Regards, Tiago Silva
9 Community Answers
Paul Johnson —
Currently the JSON schema is different for jobs exported via the UI and those exported programmatically.
This should not present a problem unless you are exporting jobs manually and then attempting to deploy those same jobs via API.
In this case you should, for consistency, have a process which exports the job and imports the job into another environment without having a user involved in some part.
What triggers a deployment from one environment to another is a commit in the Version Control System. Continuous Integration.
Let's take as an example a new Matillion Job. When I have created a Matillion Job for the first time, I want to deploy it to Test environment. How do I deploy it to Test environment? I commit it in our Version Control System and our Continuous Integration framework will deploy it to Test environment automatically. In order to commit it in our Version Control System I have to export it from Matillion using UI.
One problem here is that our Continuous Integration platform can't deploy to Test environment the Job I have just commited because it was exported from UI and it will deploy it via API. The Continuous Integration framework has to export the same Job via API first and deploy it afterwards.
This situation is an issue already. Between the time I exported the Job manually (which is the Job that I actually want to deploy to Test environment) until the time I commit it in our Version Control System it might have been altered. Example: let's say another colleague opened the Job in Matillion and is adding a new Calculator component. Our Continuous Integration framework will export a Job that is being altered and will deploy it to Test environment.
Yes, I have seen it and we are actually doing kind of the same thing. However, as I wrote before, what triggers a deployment from DEV to TEST is a commit in the Version Control System. We are continuously integrating. When we commit into our Version Control System we commit the JSON we got via UI, and that JSON can't be deployed to TEST. That's as simple as that. There are workarounds, of course. We have implemented one that works but this is not optimal at all.
Is there any intention to make these 2 formats (via UI and via API) the same in the future? Or is there any intention to make it possible to export a JSON from UI that is API-compatible? This could be a simple check box when we export the Job in the UI. This would facilitate a lot.
Yes, it is on our roadmap to make the UI export compatible with API. In the meantime, our recommendation is to use an API call to export necessary jobs/assets and then commit to your version control system (VCS) — which you already do.
Matillion works differently to other tools in that any change you make in the interface is immediately saved. There is no save button. You will need some trigger to signal that you have completed your changes and are ready to promote changes to your VCS.
If you use any collaboration tools like Jira, marking a ticket as closed may act as a trigger to kick-off a script that could use our api to export the job and add it to your VCS. Do you use any task-management tools which could help in this scenario?
Our trigger to signal that we have completed our changes and the Job is ready to be promoted to TEST is to export the Job via UI and commit it in our VCS. We do this at Job level. Then we have full control over each Job and can see who did what and when.
We do use JIRA but marking a ticket as closed (or any other state) wouldn't work. First of all a ticket can originate several Matillion Jobs and sometimes we want to deploy only part of them. Second we want to deploy to TEST many times (small increments of functionality) before closing the ticket. Etc etc etc.
This could be solved with a check-box in your UI so that we could export an API-compatible Job from your UI. Then we could commit Jobs frequently and as soon as the Job is a "Minimum Viable Product"...
The three metadata exchange formats differ from each other deliberately in fact, because of the different purposes for which they are intended.
UI-based import/export – for interactive use
v0 API – for copying entire projects at once
v1 API – much more fine-grained control, and with support for more object types than the v0 API
It’s possible that we could add a compatibility checkbox to the UI-based export, although there may be technical reasons that would prevent that. Thankyou for the suggestion, and please monitor our release notes for possible future developments in this area.
Code promotion from one environment to another should be handled with in the ETL tool, exporting code from tool to json file & importing it into another environment is not a good solution. It pose a lot of challenges.