Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Data migration from Teradata to Redshift using Database query component taking long time

Hi Team,

I am migrating Terdata table Data (approx 200 million records) to Redshift using Database query component , I have setup 17 currency as I am on Cluster ds2.xlarge 17 node.This load job is running for more than 3 hours now, can any one suggest what could be the bottle neck and how to fix this.

regards
Ravi

3 Community Answers

Matillion Agent  

Kalyan Arangam —

Hi Ravi,

Teradata seems to support a connection option TYPE=FASTEXPORT which may help speedup the data transfer process.
Please read more about this here – https://developer.teradata.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#BGBFBBEG

However, certain types like BLOB and CLOB are not supported. So do not use this if your sources have these columns in them.

A quick scan also shows another parameter MAX_MESSAGE_BODY that you can set to increase your connection throughput.

Please have a go at these and see if it helps. I would also advise looking into Teradata forums for more information on these .

Best
Kalyan


Ravi Kumar —

Thanks Kalyan,

I will try to test with these option and update you.

Can you please suggest as to how much currency is best practices when we are using Database query component , like in my case my Cluster configuration is ds2.xlarge 17 node.




Matillion Agent  

Laura Malins —

Hi Ravi,

The concurrency on the Database Query component relates to the number of files created in S3. This should be a multiple of the number of nodes you have in Redshift so it sounds like you have this right with 17.

Thanks
Laura

Post Your Community Answer

To add an answer please login