Orchestration Components

Python to Redshift

Hi, I am trying to write Pandas dataframe object to Redshift table via Python component. Currently we are passing dataframe data to Matillion Grid variable which siginificantly slows down the...

SQL Script orchestration Component

We have a job with multiple orchestration and transformation components. This job also includes SQL Script Orchestration component. We are seeing that when this SQL script component is running, it...

Mongo DB Query component and self-signed certificates

Is there a way to configure the MongoDB Query component to accept a self-signed certificate for SSL? I have a scenario where the MongoDB instance I'm querying uses SSL, but...

python Script - context how to get the properties of a grid variable

Let's say I have a grid variable called GV in which I defined 3 columns ColA, ColB, ColC. For each column I have 4 default rows. ColA ColB ColC ABC...

Database Query fails with incorrect parameters

I Have a Database Query component which takes in a number of parameters. Every now and then, the component fails and returns the error "Parameters contain errors". Upon inspection however,...

Setting Delimiter for Intermediate S3 Files

When I use the RDS Query component, is there a way to specify how columns should be delimited in the S3 file it creates as it transfers the data to...

Check if Job is running

Hello, we have to ETL jobs: "main" runs at night and "realtime" runs every 15 minutes. To avoid waste data, both jobs must not run at the same time. Therefore...

Loop until condition is met

For example, I want to make an API call every one minute, until the data in the API response meets a certain condition. How can I set up an iterator...

Stopping Database Query against Postgrs

I want to control length of a transaction against a Postgres server (a slave), to make sure not to disturb this production server. I try to set socketTimeout to 2...

Google Spreadsheets Metadata

Is there a way to force Redshift to refresh the meta data associated with a Google Spreadsheet? I've changed the column structure of the spreadsheet and Matillion does not recognize...