Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Converting Redshift table data to Parquet

Hi,

I'm already having some Redshift tables, and I"m in need to unload those data back to S3 once I have done all the ETL transformations.

So I wanted to unload it as a parquet format, instead of normal delimited files.

How can I proceed with this? Is there any existing component to do this?

Thanks.

4 Community Answers

Matillion Agent  

Laura Malins —

Hi Kulasangar

Matillion can’t currently unload from Redshift to Parquet as this isn’t supported by Redshift.

Instead you could look at creating Redshift Spectrum tables with your data which can then be stored in Parquet format?

Thanks
Laura


Kulasangar Gowrisangar —

Hi Laura, thank you so much for your assistance. I'll look into that and get back to you.

To make myself clear, it isn't possible to unload delimited files as parquet files into the s3 bucket (ie: by converting delimited files into parquet format)?

Thanks.


Matillion Agent  

Ian Funnell —

Hi Kulasangar,

Correct, currently Matillion can not output data directly into Parquet format: we only support output to gzipped CSV.

We are considering adding Parquet output to Matillion, so please monitor the release notes in case we do add that functionality to Matillion itself in future.

Best regards,
Ian


Kulasangar Gowrisangar —

Thank you so much Ian for the clarification.

Regards.

Post Your Community Answer

To add an answer please login