Pipes

Overview

Flex.io provides a graphical housing for stitching together serverless functions along with out-of-the-box helper tasks that take the pain out of OAuth, notifications, scheduling, local storage, library dependencies and other ‘glue’ code.

A Flex.io serverless function is defined by a collection of streaming tasks, called pipes. Once defined, pipes can be scheduled or triggered to run with an API request, webhook or by sending an email.

What is a Pipe?

A pipe is a collection of tasks that run sequentially.

The most commonly used task is the Execute Task, which runs custom Python or Node.js code. The Execute task can also can read and write from data sources like cloud storage, databases and APIs using Connections and connection-related bindings(e.g., accessing an Oauth refresh token). Additional helper Tasks can be configured to issue requests, convert file formats and send email notifications.

How Do Pipes Work?

When a pipe is run, a process is created that runs your Pipe Tasks sequentially. Input from the caller is passed to the first task, which operates on the input and returns an output; then the output of that task is passed to the next task and so on, until the tasks are finished and the process returns a final output.

Pipes are referenced with a unique identifer, called an eid, which is assigned to the pipe when it is created. They can also be referenced with a logical identifier, called an alias, which a user can give to them.

Once a pipe is defined and saved, it can be scheduled to run in the background or triggered by a webhook or an email. See the deployment guide for further information on these options.

What Can Pipes Do?

Pipes are simply a vehicle to quickly productionalize a function or group of functions. Flex.io pipes are particularly useful for a class of utilities that require a connection to cloud services, but can vary widely depending on application. For example, pipes can:

For live pipe examples, you can walk through a tutorial, view some common Execute task code snippets or see live demonstrations on GitHub

Creating Pipe Definitions

A pipe definition is the collection of tasks that run sequentially. The Flex.io Pipe Builder provides a simple way to add connectivity to external data sources and destinations and build pipe definitions.

When a pipe is saved, it is stored in the Pipe List. When a pipe is “OFF”, it can be tested, edited and prototyped. When a pipe is “ON” it can be deployed externally, such as being called via API endpoint or being scheduled it to run in the background.

See below for the mechanics of building and deploying pipe definitions. To build your first pipe, please see the Getting Started Guide.

Using the Pipe List

Once you’ve signed in to the Flex.io web application, navigate to the “Pipes” tab and you’ll see your list of pipes:

Pipes: Pipe List

Creating a New Pipe

To create a new pipe, click the ‘New Pipe’ button at the top right of the screen. This will take you to the Pipe Builder where you can define and test your pipe defition.

After creating a new pipe, it will be saved to your Pipe List.

Using Existing Pipes

The pipe list shows you all of your pipes along with other detail, including:

You may sort your list by using the controls next to the Pipe List headers.

Using the Pipe Builder

The pipe builder enables you to create and test pipe definitions. It provides a one-stop-shop to add functions and snap on other helper tasks to make your process production-ready.

You can access the Pipe Builder by creating a new pipe or editing an existing pipe.

Pipes: Pipe Builder New

The pipe builder has five sections, which all work in concert:

Further, the Pipe Builder includes other Properties and Controls for administrative options like setting a name or alias as well as, of course, activating your pipe.

Using Input

At the top of the Pipe Builder is the Input section. This is an optional section used to prototype your pipe with sample post parameters prior to deploying a pipe using parameters.

You may toggle the section open and closed using the arrow at the right.

Pipes: Params Input Section

The Input section enables you to customize the way you send the form data along with keys and values.

For an example of how this works, do the following:

  1. Create a New Pipe.

  2. In the Execute Task, replace the existing “hello world” code with the following Python script:

  1. Run the pipe by pressing the “TEST” button and by default the script will return the five most recent posts from Hacker News, unless you change the parameter.

  2. To test a different parameter, use the Input section to add a Key of count and a value of 3.

  3. Run the pipe again by pressing the “TEST” button and the script will now only return the three most recent posts.


The Input section is meant for testing and prototyping your params. For deploying a pipe using parameters, see the deployment guide.

Using Tasks

The second section of the Pipe Builder is for adding Tasks. The Tasks section enables you to create your pipe definition by chaining together different steps that run sequentially.

Pipes: Tasks Section

Tasks options include the following:

Adding a New Task

In a blank pipe with no existing Tasks, you may add a Task by clicking on any of the Task boxes.

In a pipe with existing Tasks, you may add a Task by clicking on the “+” button above or below the existing Task.

Pipes: Adding Tasks

After clicking on the “+” button, you may click on a Task box to open up a new Task in your Pipe. The Pipe will run Tasks sequentially, from first Task to last.

Deleting a Task

To remove a Task, hover over the Task you wish to delete and a “Trash” icon will appear on the left:

Pipes: Adding Tasks

Click the “Trash” icon and your Task will be removed from the pipe definition.

Using Output

The third section of the Pipe Builder is the Output. The output section enables you to see the result of your pipe.

To see the result of your pipe, click the “TEST” button at the top right of the Pipe Builder.

Pipes: Adding Tasks

If the result of the final Task is a stream of text, this will appear in the Output section. However, if the final step in your pipe is some other type, like, for example, an Email task, the output will remain empty, since the result of the pipe was not text, but a sent email.

Using Deployment

The fourth section of the Pipe Builder provides Pipe Deployment options, including ways to schedule it, call it via API or run on-demand.

Pipes: Adding Tasks

Deployment options include:

See the Deployment Guide for specifics on how to use and configure each of these deployment options.

Using the Definition Editor

The Definition Editor is tucked off into the sidebar on your left. Once opened, it provides a YAML or JSON view of your pipe, as well as the UI for a web front-end, if applicable.

Here is a YAML definition:

Pipes: Definition Editor YAML

Here is a JSON defition:

Pipes: Definition Editor JSON

The definition editor is particularly useful for sharing pipe definitions, as you can simply copy the YAML from one pipe and paste it into a new pipe.

To try this, do the following:

  1. Create a New Pipe.

  2. Open the Definition Editor by clicking on the icon (< >).

  3. Once the Defition Editor panel is open, make sure the YAML button is chosen, then select everything in the definition and replace it with the following code:

This is a simple “hello world” example with the pipe name, description and alias pre-filled. As you paste in the new YAML, the pipe will update with the new pipe.

For further pipe examples you can copy and paste, see the sample pipes section.

Using Controls and Properties

Pipe Controls are action items to use when building pipes like the “TEST” button and Pipe Staus toggle. Pipe Properties include the setting the Pipe name, description and alias.

Pipe Controls

This section discusses the key controls in the Pipe Builder, which can be found in the header of the page:

Pipes: Controls

The two buttons that control prototyping and activating your pipe are as follows:

Pipe Properties

The pipe properties section provides you a place to edit information about your pipe. To do so click on the edit (pencil) icon in the header next to the Pipe name. When you do, the Pipe properties will open:

Pipes: Controls

Pipe properties enables you to add or edit the following:

Sample Pipes

Below are some example pipes that show off various facets of the application. The YAML snippets below can be copied and then pasted into the Definition Editor to add the pipe to your Pipe List.

Analyze Data with Nodejs and Python

This pipe analyzes job postings on GitHub Jobs chaining together different execute tasks in different languages (Nodejs with Lodash and Python with Pandas). Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List.

Server-side Input Validation

This pipe validates input with a server-side logic that hides data used to validate it (with Python Cerberus). Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List.

Email Results of a Python Function

This pipe gets the top 5 stories from the Firebase Hacker News Feed and delivers via email. Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List. After adding the pipe, update the Email Task with your email address.

Visualize Data from an API Feed

This pipe visualizes data from an API feed (using Python Matplotlib). Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List.

Save Data to Local Storage

This pipe aggregates, de-duplicates and saves recent stories from Hacker News to local storage. Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List.