First of all, Matillion is wonderful ETL tool and very comprehensive with user friendly features. I am new to Matillion and still exploring various options provided by Matillion to load data from RDS mySQL to Redshift. Recently, I am analyzing how to handle data load in case any failure occurs while job is running and few of the rows are already processed. Does it commit after all loads or any part commit happens. What are the approaches here to handle scheduled job failures. How to set up alarm mechanism to support team in case any job failure .
1 Community Answers
Paul Johnson —
The default behavior and best practice is for the load components is to drop and recreate the staging tables each time they are run, this means if a job was re-run from the beginning after failing part way through then you would never end up with duplicate records in the staging table.
For a detailed guide on how to set up error handling in Matillion please take a look at the following articles: