Orchestration Components

Triggering Job Dependencies

This is what I have done already. I have simply created two separate buckets in S3, and there I will have csv files which contain data. So once a file...

adding additional columns to redshift table

Hi, I am attempting to orchestrate a daily load of summarised marketing information of the previous days' data using the API component and RSDs. I am passing to the RSD...

Python Editor

Hi does anyone edit their python code in anything other than the Python compononent. Are there any clever ways to debug the python script. Also a suggestion for the development...

Passing connection options for API authorization

Hi, I'm having trouble with passing a environment variable into an API component as a connection option. In the connection options I am using parameter and value pair as: profilesettings:...

Google BigQuery Unnesting Arrays

Does Matillion support unnesting Arrays in the GBQ component? select id ,array_item from dataset.table as a ,unnest(a.array) as array_item

S3 Unload - Transfer data to an other S3 environment

Hi, how can I load data from our S3 to an other S3 environment? Now I have tried to use S3 Unload component, but how can I set the Access...

S3 Unload and S3 path

Hi! How can I create a s3 path like s3://bucket/yyyy-mm-dd/filename using S3 Unload component? The path is for example s3://bucket/2018-04-23/filename and the date should change everyday. Now I have written...

move files in S3

Is it possible to use S3 put object to move files from one folder/bucket to another ? Currently I am using a python script to move my files.

File Iterator &Filter Regex

I have multiple file getting uploaded to the same folder in s3 . I would like to process all files prefixed with "Client_1". I have a variable and I'm using...

Passing variables into Bash script

Hi, I have Loop Iteration followed with Bash script, based on iterations i have to pass the value into bash script. Exp: For First iteration the value 1 should be...