Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Is there way to deal with schema change when doing Matillion CDC?

For example, if a new column added. How is the best way to handle these cases?

2 Community Answers

Rajish Shakya —

Just to give some context - we are planning to use Matillion CDC but our worries are that if there is any schema changes in any of the source tables our Matillion ETL will fail. So we are wondering how to handle situation like adding column, removing column, adding a table, removing a table, changing column name in the source tables etc using Matillion CDC.


Matillion Agent  

Arawan Gajajiva —

Hi Rajish -

If the schema of a source table used in CDC changes, you need to apply those schema changes manually to the target table in your Cloud Datawarehouse before applying the schema change in the source. If the schema changes are done on the source before applying to the target, the CDC process will fail.
Regarding adding a table, when setting up the CDC Task, if you have specified a pattern to define your source tables and the new table matches that pattern, it should automatically get picked up. If you have explicitly defined your source tables or the new table does not match the pattern you have defined in the CDC Task, then you can add a new CDC Task for the new table.

Hope this helps!
Arawan

Post Your Community Answer

To add an answer please login