I created an orchestration that creates two .csv files. In this process I created a python script to rename the .csv files to include the current date 'YYYYMMDD' in the file name. Now I need to load these .csv files up to an sftp site. I tried to use the SFTP Put Object but the file name changes each day. What options do I have to move the files in the s3 Bucket to an SFTP site.
3 Community Answers
Kalyan Arangam —
I would recommend using a S3 GET component to transfer files. SFTP PUT is to be deprecated in a later release and is there for backward compatibility.
As for the changing filename, you may use a job-level variable to capture the date you calculated in python (‘YYYYMMDD’ ). Then use that variable in ‘S3 Path’ property of the S3 GET component.
For example, assuming your job-variable is called v_today, the s3 path could be as follows -
Thank you for the quick response! This was really helpful!
I now can't get the SFTP connection to Authenticate. Any suggestions? I have double checked the URL and the user name and password. I am able to connect to the SFTP outside of Matillion but am not able to get it to authenticate in the S3 Get Object.
Does it work if you are hard-coding instead of using variables? or it just doesnt authenticate with your sftp server?
Are you able to share a screenshot of your settings and the error message?
Please note that information posted on this thread may be visible to others. You may send a response via email to email@example.com and it will create a private ticket visible only to you.
Hope that makes sense :)