Transfer files between cloud storage services

Introduction

This tutorial will walk through how to set up a data feed to transfer files between cloud storage accounts, like Dropbox, Box, Google Drive, OneDrive or an Amazon S3 bucket.

Flex.io gives you the ability to control your data feed programmatically. For example, you can set up filter criteria to limit the files you transfer based on name, type, size or other criteria, zip and unzip files, validate content or add other processing steps right out of the box.

To build this data feed, we’ll do the following:

  1. Create a connection for two different cloud storage accounts
  2. Access the list of files from your source folder
  3. Build a pipe to copy files that match certain criteria

Setup

To use the Flex.io API, you need to sign up for a Flex.io account. Once you’ve signed up, go to the Dashboard to find your API key.

We’ll be building this data feed with the Flex.io JavaScript SDK, and so you’ll need to have this installed as well:

Connect to cloud storage accounts

The first step is to set up the connections to your cloud storage accounts. In this case, we’ll connect to Dropbox and Google Drive.

A quick way to do this is to step through one of the following templates:

You can also set up connections to your cloud storage accounts using the Flex.io app. In the Connections area, click the New Connection button and select Dropbox. From there, you can change the name and alias, and authenticate the connection to your account:

New Dropbox Connection

When you set up your Dropbox and Google Drive connections, grab the alias for each one to use in your code. Here are the aliases we’ll use in our code examples below:

  • Dropbox connection alias: tutorial-dropbox
  • Google Drive connection alias: tutorial-googledrive

Get a list of files

Now that we have the connections set up, we can access their contents using the alias for each one. First, let’s get a list of files from Dropbox to see what we’re working with.

We can also generate a combined file list from both connections.

Transfer files

To start, let’s simply copy all files from a Dropbox folder to Google Drive.

Copy files based on name

We can limit the files we transfer with an array that lists the specific files to include.

Copy files based on file type

We can also transfer all files with a particular file extension by adding a wildcard (in this case, all .csv files).

Create a new destination folder

Using the .copy step, you can also create a new destination folder. For example, let’s copy files into a timestamped folder in Google Drive we’ll call backup-${process.time.unix}.

Filter files based on size and date

Finally, by adding one or more .filter steps to the pipe, you can set up filter criteria to transfer files based on file size or date.

Going further

We’ve only scratched the surface of what you can do with these data feeds. In addition to moving files, you can also add steps to zip and unzip files, as well as process and transform their content. Here are some additional templates that might be helpful: