Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

JSON format - UI vs API

Hi,

Is there an intention to have only 1 JSON format regardless of we export the Job manually through Matillion's UI or we export the Job from Matillion's API?

The current solution is inconvenient since we can't deploy automatically (via API) to Test and Prod environments whatever has been commited in our Version Control System. We have to export the same Job from Dev first but that Job might be already different than the one manually exported. This is just an example, because there are other inconveniences.

Regards,
Tiago Silva

9 Community Answers

Matillion Agent  

Paul Johnson —

Hi Tiago,
Currently the JSON schema is different for jobs exported via the UI and those exported programmatically.
This should not present a problem unless you are exporting jobs manually and then attempting to deploy those same jobs via API.

In this case you should, for consistency, have a process which exports the job and imports the job into another environment without having a user involved in some part.

Regards,
Paul


Tiago Silva —

Hi Paul,

What triggers a deployment from one environment to another is a commit in the Version Control System. Continuous Integration.

Let's take as an example a new Matillion Job. When I have created a Matillion Job for the first time, I want to deploy it to Test environment. How do I deploy it to Test environment? I commit it in our Version Control System and our Continuous Integration framework will deploy it to Test environment automatically.
In order to commit it in our Version Control System I have to export it from Matillion using UI.

One problem here is that our Continuous Integration platform can't deploy to Test environment the Job I have just commited because it was exported from UI and it will deploy it via API.
The Continuous Integration framework has to export the same Job via API first and deploy it afterwards.

This situation is an issue already.
Between the time I exported the Job manually (which is the Job that I actually want to deploy to Test environment) until the time I commit it in our Version Control System it might have been altered. Example: let's say another colleague opened the Job in Matillion and is adding a new Calculator component. Our Continuous Integration framework will export a Job that is being altered and will deploy it to Test environment.

/TS


Matillion Agent  

Paul Johnson —

Hi Tiago,

This article explains how you can export a job and commit to your source control without exporting it via the UI

Regards,
Paul


Tiago Silva —

Hi,

Yes, I have seen it and we are actually doing kind of the same thing.
However, as I wrote before, what triggers a deployment from DEV to TEST is a commit in the Version Control System. We are continuously integrating.
When we commit into our Version Control System we commit the JSON we got via UI, and that JSON can't be deployed to TEST. That's as simple as that. There are workarounds, of course. We have implemented one that works but this is not optimal at all.

Is there any intention to make these 2 formats (via UI and via API) the same in the future?
Or is there any intention to make it possible to export a JSON from UI that is API-compatible? This could be a simple check box when we export the Job in the UI. This would facilitate a lot.

/TS


Matillion Agent  

Kalyan Arangam —

Hi Tiago,

Yes, it is on our roadmap to make the UI export compatible with API. In the meantime, our recommendation is to use an API call to export necessary jobs/assets and then commit to your version control system (VCS) — which you already do.

Matillion works differently to other tools in that any change you make in the interface is immediately saved. There is no save button. You will need some trigger to signal that you have completed your changes and are ready to promote changes to your VCS.

If you use any collaboration tools like Jira, marking a ticket as closed may act as a trigger to kick-off a script that could use our api to export the job and add it to your VCS. Do you use any task-management tools which could help in this scenario?

Best
Kalyan


Tiago Silva —

Hi,

Our trigger to signal that we have completed our changes and the Job is ready to be promoted to TEST is to export the Job via UI and commit it in our VCS.
We do this at Job level. Then we have full control over each Job and can see who did what and when.

We do use JIRA but marking a ticket as closed (or any other state) wouldn't work. First of all a ticket can originate several Matillion Jobs and sometimes we want to deploy only part of them. Second we want to deploy to TEST many times (small increments of functionality) before closing the ticket. Etc etc etc.

This could be solved with a check-box in your UI so that we could export an API-compatible Job from your UI. Then we could commit Jobs frequently and as soon as the Job is a "Minimum Viable Product"...

/TS


Matillion Agent  

Ian Funnell —

Hi Tiago,

The three metadata exchange formats differ from each other deliberately in fact, because of the different purposes for which they are intended.

  • UI-based import/export – for interactive use
  • v0 API – for copying entire projects at once
  • v1 API – much more fine-grained control, and with support for more object types than the v0 API

It’s possible that we could add a compatibility checkbox to the UI-based export, although there may be technical reasons that would prevent that. Thankyou for the suggestion, and please monitor our release notes for possible future developments in this area.

Best regards,
Ian


Mohan Mane —

Code promotion from one environment to another should be handled with in the ETL tool, exporting code from tool to json file & importing it into another environment is not a good solution. It pose a lot of challenges.


Matillion Agent  

Ian Funnell —

Hi Mohan,

Matillion provides a number of mechanisms for code promotion from one environment to another, from within the UI.

Broadly they are:

The metadata exchange formats we were discussing earlier in this thread enable you to extend the capabilities to include an external source code management product like Git, Mercurial, SourceSafe etc.

Best regards,
Ian

Post Your Community Answer

To add an answer please login