Connections

Overview

A data feed is only as good as its input and output.

In Flex.io, any input or output that requires stored authentication credentials is considered a “Connection”, including cloud storage accounts, databases, SaaS apps and API calls. In the Flex.io Web App, the Connections tab is used to securely manage connections that can be used in your pipe definitions:

Connections List

You may create new connections at any time either by adding them in the Flex.io web application or adding them programmatically, within your pipe definition. This enables you to choose where to store your credentials, depending on your requirements.

Once you have a connection set up, you can then reference its path in your pipe definition with its EID or alias.

Using Connections

After adding or defining a Connection, you’ll typically use it in conjunction with either a request or read task (for inputs) or a write task (for outputs).

To reference the Connection you may use its EID or and alias. By default, every connection you create has its own unique EID string, such as g98ffn156zsj. However, EIDs aren’t necessarily memorable. So, in addition, you can create a human-readable alias to use in your connection definition. For example, you might call your connection to Dropbox something like my-dropbox.

Each account has its own namespace, so all you need to remember is this alias when referencing a connection. Learn more about creating aliases in the sections below on adding a connection through the web app or programmatically.

OK, let’s walk through some typical examples of using connections:

Adding Inputs without Using Connections

Let’s start with the most basic case: a freely available URL from the web. You may want to access an API or URL that doesn’t require authentication. While you could set up a connection for this path, you can also simply access it directly, like this:

.request('https://www.foo.com/bar.csv')

Similarly, to access a private URL, you could choose to add your authentication information directly into your request string, like this:

.request('https://www.foo.com/bar.csv?token=123456789')

Using Connections as Inputs

However, to avoid passing credentials in your definition, you can first add a Connection and then reference it via its EID or an alias you provide.

For example, you could read from a Dropbox connection using its EID or alias, respectively:

.read('/g98ffn156zsj/myfolder/*')

.read('/my-dropbox/myfolder/*')

Or, you could request data from a Twilio API connection using its EID or alias, respectively:

.request('/ht5ud5r6z7ii')

.request('/my-twilio')

Using Connections as Outputs

Similarly, if you want to push data to a Connection you’ll typically use a write task. For example:

For example, you can write to a Dropbox connection using its EID or alias, respectively:

.write('/g98ffn156zsj/myfolder/foo.csv')

.write('/my-dropbox/myfolder/foo.csv')

Moving Data Between Connections

To create a pipe that utilizes connections for both an input and output, you can simply string them together into a pipe definition. For example, if you want to copy a Twilio table to your Dropbox folder using aliases, your definition might look something like this:

For further information on connection paths, see our guide on creating and using pipes. Also, technical specifications can be found in the API reference.

Adding Connections

There are two ways to set up a connection in Flex.io. You may add connections using the Flex.io Web App or you can choose to define a connection programmatically

Adding Connections via Web App

To add a connection from the web app, sign in to Flex.io and navigate to the Connections tab. Then, click the “New Connection” button at the top right of the screen to bring up a connection chooser dialog:

Connections Chooser Dialog

This dialog will help you set up pre-built connections to cloud storage, databases and Saas apps. You may also connect to any API using the Custom API connector option.

Different services have different means of authentication, therefore each connection set up will vary slightly depending on the type of service. Below we’ll walk through the most common connection authentication methods including Oauth, Basic Authentication, Bearer Tokens, Database Authentication and Custom Authentication.

Oauth

Oauth is a browser-based authentication mechanism used by many services, such as Dropbox, GitHub and Google Sheets.

Connections: Oauth

An Oauth dialog has the following options:

When finished, select the “Create Connection” button to save your connection.

Basic Authentication

Basic Authentication is used when your service requires a login and password combination, like accessing Twilio call logs:

Connections: Basic Authentication

A Basic Authentication dialog has the following options:

When finished, select the “Create Connection” button to save your connection.

Bearer Tokens

Many services provide secure access via tokens, such as Amazon Web Services:

Connections: Bearer Tokens

A Bearer Token dialog has the following options:

When finished, select the “Create Connection” button to save your connection.

Database Authentication

It’s often useful to be able to read or write to a cloud-based database, like Postgres or MySQL. To do so, you’ll need to give Flex.io access to your database through your firewall as well as provide authentication information via the database dialog:

Connections: Database Authentication

A Database Authentiation dialog has the following options:

When finished, select the “Create Connection” button to save your connection.

Custom API Authentication

For any API connection for which Flex.io does not provide a specific connector, you can create a custom connection using the services raw API calls. To do this, select “Custom API” from the Connector Chooser dialog:

Connections: Custom API Authentication

A Custom API dialog has the following options:

When finished, select the “Create Connection” button to save your connection.

Adding Connections using Templates

In addition to manually adding Connections via the Flex.io web app as discussed above, you may also set up connections by using a template. Templates provide a step-by-step approach to configuring a data feed.

For example, you could use this simple template that sets up a feed to copy files from Google Drive to Dropbox.

Connections from Templates

After signing in, you will see a step-by-step process, including seting up connections and selecting files.

Connections from Templates

Upon completing the template, you’ll have a pipe definition to deploy, like this:

Defining Connections Programmatically

In addition to setting up and storing connection information in-app, you can also define connections in your code and connect when you run your pipe. These connections can be permanent connection definitions that appear in your connection list, or temporary local connections that are defined and used in the pipe.

Setting up a connection in code enables you to define all the resources needed by a pipe so that the logic is self-contained. In addition, it enables you to store authentication information elsewhere and then pass it to Flex.io when it’s needed.

As an example of a programmatic definition, the following pipe connects to Amazon S3 and then takes files from specified S3 paths and emails them as attachments. In this case, an example key and secret are shown, but these could be passed separately in the call to the API pipe.

When this example runs, the pipe creates a local connection to AWS S3 with the alias my-cloud-storage; then the pipe sends an email with two attachments from this connection, which are specified as a path with the local connection alias /my-cloud-storage and the path of the file within the connection /my-file1.txt.

You can see the full API specification for the Connection task here. However, below we’ll walk through the most common Connection authentication methods, including Oauth, Basic Authentication, Bearer Tokens, Database Authentication and Custom Authentication.

Oauth

In general, authentication via Oauth requires user activation, via a browser, to create a Connection. Flex.io provides a browser-based Oauth authentication in the Web App. However, if you have an access token that you’ve obtained by performing your own authentication, you can pass this along in your connection definition:

For example:

Basic Authentication

Some services use basic authentication parameters. For example, if you’re creating a connection to Twilio, you would use:

Database Authentication

In addition to login and password, databases require host and port information. A Postgres connection definition might look like this:

Managing Connections

Once a Connection has been added, you can update or delete it from the Flex.io Web App or via the API.

In the web app, click on the Connections tab to see the list of connections:

Connections List

The connection list shows each connection added to your Flex.io account. In the left-navigation, each connection summary shows the connection name, the alias or EID and the path, if relevant.

If a connection has been authenticated and valid, the connection summary shows a green checkmark icon. If the connection has been added, but has either not been authenticated or is no longer functional, the commection summary shows a red ‘x’ icon.

You can also get the list of all your connections via the API along with a status as follows:

See the API reference for details on connection properties such as status codes

Editing or Updating a Connection

You may edit or update a connection from the Web App or directly using the API:

Editing Connections from the Web App

Editing or updating a connection is similar to adding a connection.

Simply navigate to the connection you wish to edit, click on the connection summary and the connection detail will appear. Edit your connection information and, if appropriate, test or reauthenticate the connection and then click “Save Changes” to update.

Deleting a Connection

You may delete a connection from the Web App or directly using the API:

Deleting Connections from the Web App

If you’d like to remove a connection from your Connections, select the connection summary you’d like to delete. Then, at the right of the connection summary, click on the down arrow to open up a dropdown menu. Select the “Delete” button. A dialog will pop up asking you to confirm your deletion. Click “Delete Connection” to remove the connection from your list or “Cancel” to return to the connection list.

Viewing a Connection’s Directory or File List

Once you’ve created a connection, you can view lists of the files and directories in them

For example, you can view directories of these types of connections from the ‘Storage’ tab. Click on the connection summary to show a drill-down list of files associated with the connection.

Connections View Directory

In addition, you can access the directory of these connections programmatically using the list task. For example, to see the root folder of a dropbox connection with an alias of my-dropbox you would use the following pipe: