Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Need to delete rendundant rows from redshift

Hi
I am replicating rds mysql database tables on to redshift. everything is working fine. Even updated rows I am able to track using last_updated_time stamp and modifiying in redshift using a transformation job.

But I am not able to figure out how to track deleted rows in mysql tables and delete them in redshift as well. Please help on this.

Thanks in advance.

1 Community Answers

Matillion Agent  

Kalyan Arangam —

hi Hari,

Please look into the Detect changes component. It provides an Indicator field with various statuses for each row by comparing incoming data with existing data in Redshift. If a row does not exist in the source data (MySQL) and exists in target, the row get a ‘D’(delete) in the indicator column. See more about it here -

https://redshiftsupport.matillion.com/customer/portal/articles/2329042?b_id=8915

To be fair, It depends on how your data is organised and how a delete is handled in your source system. How would you otherwise recognise that a row has been deleted?

Many systems tend to have some sort of flag or a trigger that may track deleted rows by writing them to a separate table. Are there any such mechanisms in place in your source system?

Best
Kalyan

Post Your Community Answer

To add an answer please login