Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Setting Delimiter for Intermediate S3 Files

When I use the RDS Query component, is there a way to specify how columns should be delimited in the S3 file it creates as it transfers the data to Redshift?

Some of the fields in my source data include carriage returns and commas that throw off COPY in Redshift as it reads data from S3 because the carriage returns/commas lead it to interpret records as having ended abruptly or as missing fields.

2 Community Answers

Matillion Agent  

Kalyan Arangam —

Hi Matt,

Unfortunately, we do not have options to control that behaviour. I have raised a ticket with our dev team to review this (EMD-7817) and provide options in future. In the meantime, the only alternative is to replace newlines with alternate strings and replace them in the redshift database in a transformation job.

Best
Kalyan


Matt Burr —

Thanks, Kalyan. And I appreciate your reaching out to the dev team.

Post Your Community Answer

To add an answer please login