Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

V1 API Import Project

The v1 documentation [1] includes details about exporting a project, but there is no documentation relating to importing a project via the v1 API.

Please can you add that capability or document it, if it already exists.


[1] https://redshiftsupport.matillion.com/customer/portal/articles/2920263-api-v1-map

10 Community Answers

Matillion Agent  

Laura Malins —

Hi John

You can import a project. The syntax you’d need is

http://myserver/rest/v1/group/name/project/import

I’ll ensure the documentation reflects this.

Thanks
Laura


John Auld —

I have exported a project using the v1 api and I get an error when I try to import the project using the v1 api.

The file created by the export looks like valid json.


Export uri - the vars names are shown here, not the values.

http://${fqhn}/rest/v1/group/name/${groupName}/project/name/${projectName}/environment/name/${projectEnvironment}/export

Import uri - using post method and the json file saved from the command above

http://${fqhn}/rest/v1/group/name/project/name/import


Matillion Version: 1.32.6 (build 114)
AMI Version: 1.32.6


Importing Matillion project: datawarehouse using import file: C:\temp\matillion\datawarehouse_project_v1.json
Import URI: http://internal-Dev-Services-Matillion-three-803398690.eu-west-1.elb.amazonaws.com/rest/v1/group/name/project/name/import
Error: unable to import project
The remote server returned an error: (500) Internal Server Error.
#
Full Stack trace
#
{"success":false,"msg":"HTTP 404 Not Found.","id":-1}


John Auld —

Ignore the import URI above - that was testing something else. The correct one is:

Importing Matillion project: datawarehouse using import file: C:\temp\matillion\datawarehouse_project_v1.json
Import URI: http://internal-Dev-Services-Matillion-three-803398690.eu-west-1.elb.amazonaws.com/rest/v1/group/name/project/import
Error: unable to import project
The remote server returned an error: (500) Internal Server Error.
#
Full Stack trace
#
{"success":false,"msg":"HTTP 404 Not Found.","id":-1}


John Auld —

Powershell used to import


$projectFilePath = 'C:\temp\matillion\datawarehouse_project_v1.json'

[string] $matillionUri = "http://internal-Dev-Services-Matillion-three-803398690.eu-west-1.elb.amazonaws.com"

# basic auth
[string] $user = 'ec2-user'
[string] $passwd = 'something_secret_not_this'

[string] $pair = "${passwd}:${passwd}"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
[string] $basicAuthValue = "Basic $base64"
$headers = @{}
$headers.Add("Authorization", $basicAuthValue)
$headers.Add("Content-Type", "application/json")
$headers.Add("charset", "utf-8")

try
{
#[string] $uri = "$matillionUri/rest/v0/projects"
#
# v1 http://<instance address>/rest/v1/group/name/project/import
$uri = "$matillionUri/rest/v1/group/name/project/import"

Write-Host "Importing Matillion project: $projectName using import file: $projectFilePath"
Write-Host " Import URI: $uri"

# Fix for encoding issues
# Save file to ASCII and reload to strip out incompatible characters (e.g. UTF8 with BOM)
$json = Get-Content -Path $projectFilePath -Raw
$json | Out-File -Encoding ASCII -FilePath $projectFilePath
$json = Get-Content -Path $projectFilePath -Raw

$response = Invoke-RestMethod -Method POST -Headers $headers -Uri $uri -Body $json
}
catch
{
Write-Host "Error: unable to import project"
Write-Host $_.Exception.Message
Write-Host "#"
Write-Host "Full Stack trace"
Write-Host "#"
Write-Host $_
}


Matillion Agent  

Laura Malins —

Hi John

I think this is the same issue we discussed in case 10127? The version of Matillion you’re using doesn’t fully support v1 of the API. This was only fully released in version 1.31.

Thanks
Laura


John Auld —

The version is: 1.32.6 (build 114), which is listed above.

I have noticed that exporting with v0 produces a json file containing details of the project.

Lines 1-10 of a v0 export

{
"environment" : "redshift",
"version" : "1.32.6",
"groups" : [ {
"id" : 2856,
"projects" : [ {
"id" : 2859,
"name" : "datawarehouse",
"description" : null,
"versions" : {

The file also contains a line with projectGroup" : "ucas"

With a v1 export, the string datawarehouse is not found in the json and the json has a different schema. The projectGroup variable is not present either and searching for "ucas" shows that there is no equivalent variable.

Given that the API call does not contain the project group or name and the v1 format json does not do so either, it does not seem possible to import from a v1 format export.

Lines 1-10 of a v1 export

{
"objects" : [ {
"name" : "default",
"description" : "Default Version",
"locked" : false,
"jobs" : {
"objects" : [ {
"jobObject" : {
"JobType" : ".OrchestrationJob",
"id" : 427,


Matillion Agent  

Kalyan Arangam —

hi John,

Looks like you might have found a shortcoming with the V1 api. One cannot create a new (empty) Project Group using the API.
Matillion deletes a Project Group if the last project under it is deleted. This may be the reason we never created an endpoint in our V1 api to create a project group (oversight).
I have created a ticket to investigate and address this shortcoming in future (EMD-6645)

In the short term, there are two options.

  1. Create the target Project Group manually using the Create Project interface. Then import the project into it.
  2. If your source matillion server has a single Project under a Project-Group then export at the group level and import at the group level. Here’s the syntax for that.

Example of exporting a Project group named Demonstration Projects_. The JSON will not contain the Project Group name.
curl -X GET -o mtln_project
grp.json -u api-user:password “http://10.12.22.193/rest/v1/group/name/Demonstration%20Projects/export”

Importing into a new server.
curl -X POST -u api-user:password “http://31.243.49.22/rest/v1/group/import” -H “Content-Type: application/json” —data-binary @mtln_project_grp.json

Hope that makes helps.

Best
Kalyan


Mike Robinson —

$ curl -s -X GET -u "user:pass" --insecure "https://matillion.host/rest/v1/group/name/Test/export"
{"success":false,"msg":"Session [a_API] failed to retrieve the Project with ID [25] from the Project Group with ID [22]. User does not have access to project.","id":-1}[ 181203

Doesn't look like I'm able (with an admin account) to export at the group level, thus no importing elsewhere.


Mike Robinson —

Nevermind, seems to work on a Version: 1.36.5 (build 246) instance (for reference, failing instance is: Version: 1.35.9 (build 206))


Matillion Agent  

David Lipowitz —

Hi Mike,

Glad it’s working. Please let us know if you need anything else on this.

Best Regards,
Dave

Post Your Community Answer

To add an answer please login