Transfer files between an STFP server and cloud storage services

Introduction

The Secure File Transfer Protocol (SFTP) is a common method used to transfer files in an encrypted format. However, setting up data feeds between an SFTP server and a cloud storage account can be more challenging since the SFTP protocol does not support connections directly to services like Dropbox, Box, Google Drive, OneDrive, or Amazon S3.

In this tutorial, we’ll walk through how to transfer files between an SFTP server and a cloud storage account, including how to control this data feed programmatically.

Setup

To use the Flex.io API, you’ll need to have a Flex.io account and an API key, which you can grab from the Dashboard. Also, you’ll need to have the Flex.io JavaScript SDK installed:

Add connections

To begin with, let’s set up the connections to an SFTP server and a cloud storage account. In this case, we’ll create a data feed between SFTP and a folder in Dropbox, and so these are the connections we’ll need.

For a quick way to do this, you can step through one the following templates:

You can also set up connections to the SFTP server and your cloud storage account using the Flex.io app. In the Connections area, click the New Connection button and select SFTP. From there, you can change the name and alias, and add the SFTP server information:

New SFTP Connection

When you set up your connections, grab the alias for each one to use in your code. Here are the aliases we’ll use in our code examples below:

  • SFTP connection alias: tutorial-sftp
  • Dropbox connection alias: tutorial-dropbox

List the SFTP files

Now that we have the connections set up, we can access their contents using the alias for each one. To start, let’s generate a list of files from the root folder on the SFTP server.

We can also list files in a sub-folder or filter the list based on name, type, and other criteria.

Transfer files

To set up a basic data feed, let’s start by transfering all files from a sub-directory on the SFTP server to a folder in Dropbox.

Copy files based on name

We can specify a set of files to transfer with an array that lists the files to include.

Copy files based on file type

We can also transfer files based on their type using a wildcard (in this case, all .csv files).

Create a new destination folder

Using the .copy step, you can also create a new destination folder. For example, let’s copy files into a timestamped folder in Dropbox.

Filter files based on size and date

Finally, by adding one or more .filter steps to the pipe, you can set up filter criteria to transfer files based on file size, date, or other criteria.

Going further

There are many additional ways you can customize your data feeds. For example, you can move files into different folders based on their name, type or other criteria, zip and unzip files, convert file formats, and process file content.

Here are some additional templates that might be helpful: