Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Source Control integration

In working through our deployment pipeline plans, we want to use multiple Matillion instances, and use an automated pipeline (in Jenkins) pull projects from one instance (dev), commit them to source control, and on approval, push them to the another instance (prod).

This follows the basic flow discussed in the linked blog posts.

However, I'm seeing a few problem points which individually aren't blockers, but in combination are causing me pain.

1. Cannot import an existing object (project, version or job)
2. Cannot delete the 'default' version of a project.
3. Deleting a project deletes the Group above it if there are no other projects in the group.

My solution to 1 was to export at the project/version level, as that would keep me in the same group/project, and allow for Matillion 'version' to track the branch name in git.

But then 2 blocks me from ever pushing updates back to 'master', because I have to delete the project to replace the 'default' version via API. and because the JSON output is *different* if I export at the project level than the project/version level, I can't use the version aware export to recreate the project.

Thoughts?

refs:
https://www.matillion.com/blog/integrate-source-code-control-systems-matillion-etl/
https://www.matillion.com/blog/matillion-etl-deployment-options-instances/

5 Community Answers

Matillion Agent  

Dan D'Orazio —

Hi Mike -

Apologies for the trouble you are having.

There is a support article here which does help to provide a more specific example and use-case for SCM Integration. It may prove useful to you, even though it doesn’t specifically call out some of the pain points you mention.

To that end, I hope the following can be of some assistance.

In response to the combination of items 1 and 2
Option 1
You may consider creating another version in Matillion to act as ‘master’. This would allow you to perform your exports and imports at the Project/Version level, while retaining the ability to delete whichever version is necessary as part of your deployment process. The ‘default’ version, in any deployed environment, would no longer be used.

Option 2
Increment your versions, as opposed to replacing existing versions. This has the added benefit of not requiring that versions be deleted / replaced. Though, depending on your pipeline plans, this may still be a requirement if you wish to retain the ability to replace previously deployed versions.

A caveat to working with versions. If you are using Schedules in Matillion, those are associated with a given version. So, you would need one additional step to re-associate Schedules from previous versions with new or replaced versions. There is an API endpoint for this purpose, which allows you to update schedules that would be invalidated from a version delete, or, to associate existing schedules with the newest version. I sent an example of how this could be accomplished in an e-mail, if using versions is the preferred method.

In response to item 3
The Group should be recreated, given the same name as before, on re-import of the project. I’ve tested this using the v1 endpoint below.
‘rest/v1/group/name/{}/project/import’

Since Project Exports contain all the Versions and all the Schedules (and versions wouldn’t necessarily need to be used), it does avoid invalidation of Schedules.

Let us know if we can be any additional assistance. I’m happy to setup a quick call if it would be beneficial.

Best -
Dan


Mike Robinson —

Dan,

I'm not able to recreate your item 3 response. If the group does not already exist I get:

url: https://10.180.9.200/rest/v1/group/name/TestGroup/project/import
response body:
{
"name" : "Projects",
"success" : false,
"statusList" : [ {
"success" : false,
"name" : "TestProject",
"exception" : "com.matillion.bi.emerald.server.rest.v1.exception.RestAPIException",
"msg" : "User [mrobinson] failed to retrieve the Project Group with Name [TestGroup]. No Project Group with name [TestGroup]. "
} ]
}


Matillion Agent  

Dan D'Orazio —

Hi Mike -

Interesting. Are you able to share, in an attachment perhaps, the entire statement you are sending to the API, minus the credentials?

Best -
Dan


Mike Robinson —

For the immediate time being, I've simply wrapped my project import script with an import at the group level using a JSON template based on a freshly created project (templating out the the group and project names).

So my project import steps are:
- send a templated minimal json blob to: /rest/v1/group/import with group as my intended group, and project called 'init'
- send the actual project export json to /rest/v1/group/name/<groupname>/project/import
- call /rest/v1/group/name/<groupname>/project/name/init/delete

It's not elegant or ideal, but it does the trick.

for reference, we're using Version: 1.36.5 (build 246) redshift.


Matillion Agent  

Dan D'Orazio —

Hi Mike -

I agree. Having to stub out a project for this purpose isn’t ideal. My test was using 1.37.6 (build 301) for Redshift. Our release notes don’t mention new functionality for this directly, but it is possible that the work related to our Data Migration Tool may have had an impact. If / when you decide to bring down that update, it might be worth checking it out.

One last thing worth mentioning, perhaps. Git integration is on our Roadmap, so it’s understood that the current mechanics around SCM Integration are sub-optimal.

Best -
Dan

Post Your Community Answer

To add an answer please login