Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Bash Script component

I am trying to use the bash script component (in conjunction with a file iterator) to move a file from one directory of an sftp site to another. Is there a way to do this using some command before the 'mv'. I know with AWS, CLI tools can be used like 'aws s3 mv', but this would be operating on an SFTP site, not s3.

If there a prefix, like 'sftp mv' or something that allows me to access and operate on the file system of an SFTP server.

Thank,

Alex

12 Community Answers

Matillion Agent  

Laura Malins —

Hi Alex

You’re doing the right thing. However there’s no mv command for sftp. Instead you can use rename – this works just like mv. So you want to run:

sftp rename

Thanks
Laura


Alex Pollatos —

Thanks for the lightning fast reply Laura!

With regard to the syntax, would it be something like

sftp rename sftp://<username>@<IP>:<port>/<full path>/ sftp://<username>@<IP>:<port>/<new full path>/

like in the File Iterator

or would it be something closer to what one would put into a Terminal Shell.

Also—are credentials just taken care of or do i need to include relevant passwords?

Thanks again,

Alex


Alex Pollatos —

In doing what I described above, i receive this output:

usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
[-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
[-o ssh_option] [-P port] [-R num_requests] [-S program]
[-s subsystem | sftp_server] host
sftp [user@]host[:file ...]
sftp [user@]host[:dir[/]]
sftp -b batchfile [user@]host


Matillion Agent  

David Lipowitz —

HI Alex,

I’m looking into this. Will get back with you shortly.

Best Regards,
Dave


Matillion Agent  

David Lipowitz —

Hi Alex,

There are a few things going on here. The first is authentication and the second is syntax (the latter is why sftp threw the usage message you forwarded).

For authentication, you’ll need to get with the sftp host’s sysadmin to get the proper authentication technique for your environment. Typically authentication is handled using an identity file (much like ssh) so logging into an sftp server often looks like this:

    sftp -i /path/to/identity/file.pem username@host 

Once logged into sftp, you can issue the rename command using the following:

    rename /dir1/file.txt /dir2/file.txt

Everything described so far assumes a human is typing commands in an interactive shell. For the Shell Script component in Matillion, you’ll want to pipe commands into the sftp command something like this:

    echo "rename /dir1/file.txt /dir2/file.txt" | sftp -i /path/to/identity/file.pem username@host

Further, you can variabilize the file name using a matillion variable:

    echo "rename /dir1/${filename} /dir2/${filename}" | sftp -i /path/to/identity/file.pem username@host

I think that gets you to the functionality we discussed yesterday. I hope that helps and please let us know if you have any more questions.

Best Regards,
Dave


Alex Pollatos —

Thanks so much, very comprehensive answer,

One follow-up: should the identity pem file be stored locally on the ec2 instance I am running matillion on, and then use that path?

I am currently trying this:

sftp -i /<path to file stored on matillion ec2>/<key name>.pem.txt <user>@<IP>

and receiving the following error:

Warning: Identity file [omitted] not accessible: Permission denied.
Host key verification failed.
Couldn't read packet: Connection reset by peer

Cheers,

Alex


Alex Pollatos —

From within the Matillion ec2 instance, in a shell, I am able to SFTP into my server and issue commands, but not in the Bash Script component.

I think it has to do with the path to the identity file being reference in the Bash Component


Matillion Agent  

David Lipowitz —

Hi Alex,

Glad that information was helpful.

Regarding the Warning: Identity file [omitted] not accessible: Permission denied message: yes, the .pem file needs to be on the Matillion host in EC2. One wrinkle is that it needs to be visible to the tomcat user as that account runs the Matillion service (including the Bash Script component). A sensible place for this would be the /usr/share/tomcat8/.ssh directory but anywhere the tomcat user can see it is fine.

Regarding the Host key verification failed message, this is cropping up because the remote sftp server hasn’t been verified by the user (again, in this case tomcat). You’ll need to do this manually just the one time with the following commands once the .pem is available:

    sudo -u tomcat bash
    sftp -i /path/to/identity/file.pem username@host

It will then prompt you to confirm the remote host is valid by asking:

    Are you sure you want to continue connecting (yes/no)?

Type yes and this will add an entry in tomcat’s .ssh/known_hosts file. Then exit out of the sftp server as well as the tomcat linux command line you entered with the sudo command above. You should be good going forward after that.

Should there be more than one sftp server you want to connect to, you’ll have to run an sftp command like the one above for each remote host you want to configure.

Hope that all makes sense and please let me know if you need more details. Glad to jump on a quick screen share if that’s more expedient.

Best Regards,
Dave


Alex Pollatos —

That worked, except for one small detail.

The SFTP site's user has a password that needs to be input. Can that be done within the bash script using another pipeline or somewhere inside the sftp -i... command.

Thanks for sticking with this,

Alex


Matillion Agent  

David Lipowitz —

Hi Alex,

I’ve done some research on this and it appears almost as if securely passing a password to an sftp server is being made deliberately difficult!

If the identity file approach isn’t viable, perhaps you can set up public key authentication using the instructions found here: ssh Without Password in Few Simple Steps

This approach is the industry standard as it is the most secure and will enable sftp to work without a password at the command line. That said, you will need to add the Matillion host’s tomcat user’s public key to the remote sftp server’s authorized_keys file. This is fairly simple but requires that you have access to the remote host via ssh. If you have that, then I can walk you (or your sysadmin) through those configuration steps.

All that said, and if nothing above will work, then the most common approach to passing sftp passwords on the command line is to use the sshpass utility. This utility however doesn’t come prepackaged and needs to be compiled from it’s source code. I can walk you through that as well, but I strongly recommend using the public-key authentication method discussed above.

Please let me know your thoughts and how you’d like to proceed.

Best Regards,
Dave


Alex Pollatos —

I have ssh access to the SFTP site, so adding the Tomcat user's public key would be ideal in that case.

How would I go about doing so?

Thanks


Matillion Agent  

David Lipowitz —

Hi Alex,

There’s some pretty good instructions here: ssh Without Password in Few Simple Steps

More than happy to walk you through it over a screen share if you like. If so, please let me know what times might work.

Best Regards,
Dave

Post Your Community Answer

To add an answer please login