Data migration from Teradata to Redshift using Database query component taking long time
I am migrating Terdata table Data (approx 200 million records) to Redshift using Database query component , I have setup 17 currency as I am on Cluster ds2.xlarge 17 node.This load job is running for more than 3 hours now, can any one suggest what could be the bottle neck and how to fix this.
3 Community Answers
Kalyan Arangam —
Teradata seems to support a connection option TYPE=FASTEXPORT which may help speedup the data transfer process.
Please read more about this here – https://developer.teradata.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#BGBFBBEG
However, certain types like BLOB and CLOB are not supported. So do not use this if your sources have these columns in them.
A quick scan also shows another parameter MAX_MESSAGE_BODY that you can set to increase your connection throughput.
Please have a go at these and see if it helps. I would also advise looking into Teradata forums for more information on these .
The concurrency on the Database Query component relates to the number of files created in S3. This should be a multiple of the number of nodes you have in Redshift so it sounds like you have this right with 17.