I am trying to use the bash script component (in conjunction with a file iterator) to move a file from one directory of an sftp site to another. Is there a way to do this using some command before the 'mv'. I know with AWS, CLI tools can be used like 'aws s3 mv', but this would be operating on an SFTP site, not s3.
If there a prefix, like 'sftp mv' or something that allows me to access and operate on the file system of an SFTP server.
12 Community Answers
Laura Malins —
You’re doing the right thing. However there’s no mv command for sftp. Instead you can use rename – this works just like mv. So you want to run:
There are a few things going on here. The first is authentication and the second is syntax (the latter is why sftp threw the usage message you forwarded).
For authentication, you’ll need to get with the sftp host’s sysadmin to get the proper authentication technique for your environment. Typically authentication is handled using an identity file (much like ssh) so logging into an sftp server often looks like this:
sftp -i /path/to/identity/file.pem username@host
Once logged into sftp, you can issue the rename command using the following:
rename /dir1/file.txt /dir2/file.txt
Everything described so far assumes a human is typing commands in an interactive shell. For the Shell Script component in Matillion, you’ll want to pipe commands into the sftp command something like this:
Regarding the Warning: Identity file [omitted] not accessible: Permission denied message: yes, the .pem file needs to be on the Matillion host in EC2. One wrinkle is that it needs to be visible to the tomcat user as that account runs the Matillion service (including the Bash Script component). A sensible place for this would be the /usr/share/tomcat8/.ssh directory but anywhere the tomcat user can see it is fine.
Regarding the Host key verification failed message, this is cropping up because the remote sftp server hasn’t been verified by the user (again, in this case tomcat). You’ll need to do this manually just the one time with the following commands once the .pem is available:
It will then prompt you to confirm the remote host is valid by asking:
Are you sure you want to continue connecting (yes/no)?
Type yes and this will add an entry in tomcat’s .ssh/known_hosts file. Then exit out of the sftp server as well as the tomcat linux command line you entered with the sudo command above. You should be good going forward after that.
Should there be more than one sftp server you want to connect to, you’ll have to run an sftp command like the one above for each remote host you want to configure.
Hope that all makes sense and please let me know if you need more details. Glad to jump on a quick screen share if that’s more expedient.
This approach is the industry standard as it is the most secure and will enable sftp to work without a password at the command line. That said, you will need to add the Matillion host’s tomcat user’s public key to the remote sftp server’s authorized_keys file. This is fairly simple but requires that you have access to the remote host via ssh. If you have that, then I can walk you (or your sysadmin) through those configuration steps.
All that said, and if nothing above will work, then the most common approach to passing sftp passwords on the command line is to use the sshpass utility. This utility however doesn’t come prepackaged and needs to be compiled from it’s source code. I can walk you through that as well, but I strongly recommend using the public-key authentication method discussed above.
Please let me know your thoughts and how you’d like to proceed.