When I use the RDS Query component, is there a way to specify how columns should be delimited in the S3 file it creates as it transfers the data to Redshift?
Some of the fields in my source data include carriage returns and commas that throw off COPY in Redshift as it reads data from S3 because the carriage returns/commas lead it to interpret records as having ended abruptly or as missing fields.
2 Community Answers
Kalyan Arangam —
Unfortunately, we do not have options to control that behaviour. I have raised a ticket with our dev team to review this (EMD-7817) and provide options in future. In the meantime, the only alternative is to replace newlines with alternate strings and replace them in the redshift database in a transformation job.