Transfer files between an STFP server and cloud storage services
The Secure File Transfer Protocol (SFTP) is a common method used to transfer files in an encrypted format. However, setting up data feeds between an SFTP server and a cloud storage account can be more challenging since the SFTP protocol does not support connections directly to services like Dropbox, Box, Google Drive, OneDrive, or Amazon S3.
In this tutorial, we’ll walk through how to transfer files between an SFTP server and a cloud storage account, including how to control this data feed programmatically.
To begin with, let’s set up the connections to an SFTP server and a cloud storage account. In this case, we’ll create a data feed between SFTP and a folder in Dropbox, and so these are the connections we’ll need.
For a quick way to do this, you can step through one the following templates:
You can also set up connections to the SFTP server and your cloud storage account using the Flex.io app. In the Connections area, click the New Connection button and select SFTP. From there, you can change the name and alias, and add the SFTP server information:
When you set up your connections, grab the alias for each one to use in your code. Here are the aliases we’ll use in our code examples below:
- SFTP connection alias:
- Dropbox connection alias:
Now that we have the connections set up, we can access their contents using the alias for each one. To start, let’s generate a list of files from the root folder on the SFTP server.
We can also list files in a sub-folder or filter the list based on name, type, and other criteria.
To set up a basic data feed, let’s start by transfering all files from a sub-directory on the SFTP server to a folder in Dropbox.
We can specify a set of files to transfer with an array that lists the files to include.
We can also transfer files based on their type using a wildcard (in this case, all
.copy step, you can also create a new destination folder. For example, let’s copy files into a timestamped folder in Dropbox.
Finally, by adding one or more
.filter steps to the pipe, you can set up filter criteria to transfer files based on file size, date, or other criteria.
There are many additional ways you can customize your data feeds. For example, you can move files into different folders based on their name, type or other criteria, zip and unzip files, convert file formats, and process file content.
Here are some additional templates that might be helpful: