Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

S3 to SFTP

I created an orchestration that creates two .csv files. In this process I created a python script to rename the .csv files to include the current date 'YYYYMMDD' in the file name. Now I need to load these .csv files up to an sftp site. I tried to use the SFTP Put Object but the file name changes each day. What options do I have to move the files in the s3 Bucket to an SFTP site.

3 Community Answers

Matillion Agent  

Kalyan Arangam —

Hi Curtis,

I would recommend using a S3 GET component to transfer files. SFTP PUT is to be deprecated in a later release and is there for backward compatibility.

As for the changing filename, you may use a job-level variable to capture the date you calculated in python (‘YYYYMMDD’ ). Then use that variable in ‘S3 Path’ property of the S3 GET component.

For example, assuming your job-variable is called v_today, the s3 path could be as follows -

s3://my_bucket/my_folder/filename_${v_today}.csv

Hope that helps.

Best
Kalyan


Curtis Sparks —

Kalyan,

Thank you for the quick response! This was really helpful!

I now can't get the SFTP connection to Authenticate. Any suggestions? I have double checked the URL and the user name and password. I am able to connect to the SFTP outside of Matillion but am not able to get it to authenticate in the S3 Get Object.


Matillion Agent  

Kalyan Arangam —

Hi Curtis,

Does it work if you are hard-coding instead of using variables? or it just doesnt authenticate with your sftp server?

Are you able to share a screenshot of your settings and the error message?

Please note that information posted on this thread may be visible to others. You may send a response via email to support@matillion.com and it will create a private ticket visible only to you.
Hope that makes sense :)

Best
Kalyan

Post Your Community Answer

To add an answer please login