Transfer files between an STFP server and cloud storage services
In this tutorial, you’ll learn how to create a serverless function that copies files from SFTP to a cloud storage service.
You’ll learn how to:
- Connect to SFTP and a cloud storage service
- Get a list of files to copy
- Select the files that are copied based on filetype
- Copy the files from SFTP to the cloud storage service
To complete this tutorial, you’ll need:
- A Flex.io account and API key
- An SFTP service you’d like to copy files from
- A cloud storage account you’d like to copy the files to. Here are some of the cloud storage services available in Flex.io: Dropbox, Box, Google Drive and Amazon S3.
First, you’ll need to connect to a SFTP service and then the cloud storage service you’ll be copying to.
- To connect to these services, navigate to the Connection page and add your Connections.
- For the SFTP service you’ll copy from, give the connection the alias
example-sftp-to-cloud-storage-source. This will be used as part of the path in the code that performs a transfer, similar to a network drive mapping.
- For the cloud storage service you’ll copy to, give the connection the alias
Next, get a list of files you want to copy. To do this:
Create a new pipe in Flex.io
In the pipe, you’ll see an initial Execute Task; make sure the language option is set to
Python, and then overwrite the example “Hello, World” code with the following:
Once you’ve got a basic list, change the path to the folder that has the files you’d like to copy. For example, if you want to copy files in a folder called
/temp/files, you’d change the path to:
In the previous step of this tutorial, the output list was limited to 5 items. Depending on the folder you listed, you may have also noticed that the list includes directories as well as files.
You’ll likely want to change the filter criteria to only include files, and probably other criteria, such as file extension or filename. For now, update the directory listing to only include text files.
- Update your execute step with the following:
- Run the pipe, and you should see something like:
Finally, copy the files that you’ve listed.
- First, add another Execute Task. Make sure the language option is set to
Pythonand then update the execute code with the following placeholder logic that simply echoes the output of the previous step:
Run the pipe and, and you should see the filtered list of items in the Output.
Now, update your new Execute Task with code to copy the items in the list to the cloud storage service:
This copy logic could be included in the Execute Task that lists the files. However, by breaking out the copy and list logic, you gain significant flexibility in changing either the list or copy functionality without affecting the other.
For example, you may want to copy files based on a list you supply when you call the pipe from an API endpoint, or based on a file that you read in, or some combination. In that case, you’d simply need to update the list logic to use that list and simply pass on the new list to the copy.
Other options in the copy functionality may include applying some type of transformation or conversion. Or you may want to add an additional task after the copy to send a notification that the files have been copied or report on an error. Chaining together distinct execute steps tends to offer better reusability.