Pipes

Overview

Flex.io is an API for building and deploying data feeds. Your data feed logic is defined using a collection of steps, called pipes. The pipe definition is written using either the REST API or the Javascript SDK

What is a Pipe?

A pipe is a collection of tasks that run sequentially.

These tasks can read and write from cloud storage, databases and APIs using connections as well as make API requests. Tasks such as list, create, read, write, delete, and request are used to read and write to the resources defined by the connections.

Tasks can also process data, including converting file formats, transforming data and applying functions on a collection of files. Tasks such as convert are used to convert the content from one format to another. A task such as execute are used to run custom Python or Javascript code on content being passed through the pipe.

How Do Pipes Work?

When a pipe is run, a process is created with a copy of the pipe tasks, which then execute sequentially. Input from the caller is passed to the first task, which operates on the input and returns an output; then the output of that task is passed to the next task and so on, until the tasks are finished and the process returns a final output.

Pipes are referenced with a unique identifer, called an eid, which is assigned to the pipe when it is created. They can also be referenced with a logical identifier, called an alias, which a user can give to them.

Once a pipe is defined and saved, it can be scheduled to run in the background or triggered by a webhook or an email. See the deployment guide for further information on these options.

What Can Pipes Do?

Data delivery by way of pipes can vary widely depending on their application. For example, pipes can:

For live pipe examples, you can step through a template, walk through a tutorial, view some sample code snippets or see live demonstrations on GitHub

Creating Pipe Definitions

In Flex.io, there are two ways to save and run a pipe defition, through the Flex.io Web App or programmatically using the API.

The web appplication provides a simple way to create connections and build a pipe definition quickly. After saving the pipe in the web application, you can then call it from its API endpoint or schedule it to run in the background.

You may also choose to build your pipe definitions programmatically and store that definition with the rest of your code. The pipe definitions can then be run inline, called via their endpoint or scheduled.

See below for the mechanics of building and deploying pipe definitions using these two options.

Building Pipes via Web App

In this section, we’ll show you how to build a pipe via the web application. To use Flex.io web application, simply sign in to Flex.io.

Using the Pipe List

Once you’ve signed in to the Flex.io web application, navigate to the “Pipes” tab on the left and you’ll see your list of pipes:

Getting Started: Pipe List

Creating a New Pipe

To create a new, blank pipe, click the ‘New Pipe’ button at the top right of the screen. This will take you to the pipe builder where you can define and test your pipe defition and, ultimately, save your pipe to the pipe list.

Using Existing Pipes

The pipe list shows you all of your pipes along with other detail, including:

Using the Pipe Builder

The pipe builder is a dialog that enables you to create and test pipe definitions. It also provides a one-stop-shop for pipe utilities, such as scheduling and deployment snippets.

To use the pipe builder to define a pipe, either click the ‘New Pipe’ button from the Pipe list, click on a pipe in the pipe list, or select ‘edit’ from the action menu from any pipe in the pipe list.

Getting Started: Pipe Builder New

The pipe builder has two sections. The top section displays the pipe properties and the bottom section contains the pipe configuration.

Pipe Properties

The pipe properties section provides you a place to edit information about your pipe, including:

Pipe Configuration

The pipe configuration section enables you to define and test your pipe, including:

Build a Pipe

To provide a simple example, we’ll build a pipe from the pipe builder as discussed in the previous section. This pipe will simply takes a CSV file via a URL and converts it into JSON format.

In the pipe properties section, give your pipe a Name, like “CSV to JSON Endpoint”. Also give your pipe an Alias like “json-endpoint”.

Then, in the pipe configuration section, paste in the following code snippet:

Your builder area should now look something like this:

Getting Started: Pipe Builder with Definition

Run the Pipe

The builder uses the Flex.io Javascript SDK syntax to define pipes. In this case we’re using a request task to read a URL of sample contact names and then a convert task to convert the CSV to JSON.

To run the pipe, click the “Run” button. Now, after running, you should see a response in JSON like this (note: for comparison, see the original CSV file here):

[
{
"id": "1",
"first_name": "Kimbra",
"last_name": "Pettigrew",
"email": "kpettigrew0@dedecms.com",
"gender": "Female",
"ip_address": "253.123.30.10"
},
{
"id": "2",
"first_name": "Cornall",
"last_name": "Rowlett",
"email": "crowlett1@sitemeter.com",
"gender": "Male",
"ip_address": "223.20.241.246"
},
{...}
]

Save and Deploy the Pipe

You’ve now successfully defined a functioning pipe.

Now you just need to save the definition and call it. To save, click the “Save” button in the upper right of the builder. This will save your pipe to the Pipe List.

You can deploy your pipe in a myriad of ways, but let’s simply call this via cURL.

Next to your API Endpoint, click on the “How do I deploy this pipe?” link. One of the options will be a cURL command that will look something like this (but will include your own API key):

curl -X POST 'https://api.flex.io/v1/me/pipes/json-endpoint/run' -H 'Authorization: Bearer {YOUR-API-KEY}'

Fire this off in cURL and, as expected, you’ll see the same JSON as you saw from within the builder.

Once a pipe is defined and saved, it can be scheduled to run in the background or triggered by a webhook or an email. For further information, see the deployment guide

Building Pipes Programmatically

Creating Pipes using the REST API

In the REST API, pipes are created by making an HTTP POST request to the /pipes API endpoint. When pipes are created, it’s common (but not required) to include basic information about the pipe, including the pipe name, pipe alias, and pipe task.

Now, let’s take a step-by-step approach to build up a pipe with two tasks.

Create a Basic Pipe

For example, to create a pipe using the REST API, make the following request, where {token} is your API key, {pipe-name} is the name of the pipe you would like to create, {pipe-alias} is the identifier you would like to use to reference the pipe, and {task} is the JSON object for the task:

Create a Pipe with Single Task

Pipes are usually created with a task to run, which is set in the pipe task parameter.

For example, to create a pipe with a request task, make the following request:

Create a Pipe with Multiple Tasks

Most pipes chain together multiple requests, so that the input of a given task is the output of the preceding task. To chain together multiple requests, a special sequence task is used, and the tasks to chain together are specified in the items property of the sequence task.

Here’s an example that requests a CSV file from a URL and converts it:

Running Pipes

Pipes can be run with from an API call. When pipes are run from the API, request parameters are sent to the pipe, the pipe executes the tasks, and then the pipe output is returned to the caller.

Running a pipe from the REST API

To run a pipe with the REST API, create either an HTTP GET request or an HTTP POST request to the pipe run API endpoint.

In the case of an HTTP GET request, use the following where {id} is the eid or alias for your pipe, token is your API key, and any parameters you want to pass to the pipe are specified as part of the URL as key and value pairs as usual:

In the case of an HTTP POST request, use the following where {id} is the eid or alias for your pipe, token is your API key, and any parameters you want to pass to the pipe are specified as a JSON payload in the data payload of the request:

Running a pipe from the Javascript SDK

To run a pipe from the Javascript SDK, call the run() function on the pipe object.

See the Javascript SDK info on run for more info.

Get a List of Pipes

Pipes can be listed with an HTTP GET request to the /pipes API endpoint:

Update Pipe Information

Pipes can be updated with an HTTP POST request to the /pipes/{id} API endpoint:

Using Pipe Parameters

Passing Parameters to a Pipe

When pipes are run from the REST API endpoint, parameters can be passed to the pipe either through QUERY parameters in the URL in a GET request or in the data payload of a POST request.

In the following GET request, the key/value pair in the query string is passed to the pipe:

In the following POST request, the data payload is passed to the pipe:

Getting the Output of a Pipe

When pipes are run calling the pipe run API endpoint, the API request blocks until the pipe process is finished, and the results are returned in the response.