Flex.io provides a graphical housing for stitching together serverless functions along with out-of-the-box helper tasks that take the pain out of OAuth, notifications, scheduling, local storage, library dependencies and other ‘glue’ code.
A Flex.io serverless function is defined by a collection of streaming tasks, called pipes. Once defined, pipes can be scheduled or triggered to run with an API request, webhook or by sending an email.
A pipe is a collection of tasks that run sequentially.
The most commonly used task is the Execute Task, which runs custom Python or Node.js code. The Execute task can also can read and write from data sources like cloud storage, databases and APIs using Connections and connection-related bindings(e.g., accessing an Oauth refresh token). Additional helper Tasks can be configured to issue requests, convert file formats and send email notifications.
When a pipe is run, a process is created that runs your Pipe Tasks sequentially. Input from the caller is passed to the first task, which operates on the input and returns an output; then the output of that task is passed to the next task and so on, until the tasks are finished and the process returns a final output.
Pipes are referenced with a unique identifer, called an
eid, which is assigned to the pipe when it is created. They can also be referenced with a logical identifier, called an
alias, which a user can give to them.
Once a pipe is defined and saved, it can be scheduled to run in the background or triggered by a webhook or an email. See the deployment guide for further information on these options.
Pipes are simply a vehicle to quickly productionalize a function or group of functions. Flex.io pipes are particularly useful for a class of utilities that require a connection to cloud services, but can vary widely depending on application. For example, pipes can:
- Pass OAuth service tokens as an API (e.g., access the GitHub API to access private repo files).
- Reading and writing to cloud storage (e.g., copy files from Dropbox and send as an email attachment).
- Chaining together multiple fuctions in multiple languages (e.g., chain together a node.js lodash function with a python pandas function)
- Pre-rendering and caching vizualizations (e.g., use matplotlib to render images server side and call via API endpoint)
- Process data to produce a result (e.g., take call logs from Twilio and create a pivot table).
- Receive paramterss as input and return results according to that input (e.g., Use a web form to accept user search criteria and return the filtered result).
- Perform an action on single files or multiple files (e.g., select a bucket of images on AWS S3 and convert them to thumbnails).
A pipe definition is the collection of tasks that run sequentially. The Flex.io Pipe Builder provides a simple way to add connectivity to external data sources and destinations and build pipe definitions.
When a pipe is saved, it is stored in the Pipe List. When a pipe is “OFF”, it can be tested, edited and prototyped. When a pipe is “ON” it can be deployed externally, such as being called via API endpoint or being scheduled it to run in the background.
See below for the mechanics of building and deploying pipe definitions. To build your first pipe, please see the Getting Started Guide.
Once you’ve signed in to the Flex.io web application, navigate to the “Pipes” tab and you’ll see your list of pipes:
To create a new pipe, click the ‘New Pipe’ button at the top right of the screen. This will take you to the Pipe Builder where you can define and test your pipe defition.
After creating a new pipe, it will be saved to your Pipe List.
The pipe list shows you all of your pipes along with other detail, including:
- Pipe name. This is the name you give to your pipe. You may edit this in your Pipe Properties.
- Pipe description. This is a description of your pipe. You may edit this in your Pipe Properties.
- Executions. This displays the total number of times your pipe was run (both testing and deployed), for the previous 30 days.
- Deployment. These icons provide a quick view of the deployment options that have been selected for the pipe, including API endpoint, Scheduling and Flex.io Web Interface. Icons that are grayed out are turned off; icons that are blue are turned on.
- Status. The toggle at the right side of a pipe shows whether a pipe is ON of OFF. If a pipe is OFF, you may edit and prototype it. If it is ON, you may deploy it.
Action menu. The arrow at the far right of the pipe is a drop-down menu that enables you to do one of the following:
- Edit. This option takes you to the Pipe Builder to edit and test your pipe definition.
- Duplicate. This option will create a second, duplicate copy of your pipe and add it to the list.
Delete. This option enables you to remove the pipe from your account. Prior to deletion, you’ll need to confirm this step.
You may sort your list by using the controls next to the Pipe List headers.
The pipe builder enables you to create and test pipe definitions. It provides a one-stop-shop to add functions and snap on other helper tasks to make your process production-ready.
The pipe builder has five sections, which all work in concert:
- Input. The input section enables you to test your pipe with post parameter, if applicable.
- Tasks. The tasks section enables you to create your pipe definition by chaining together different Tasks.
- Output. The output section enables you to see the result of your pipe, after clicking the TEST button.
- Deployment. The deployment section provides you with options for scheduling your pipe, such as calling it via API endpoint or creating a web front-end that can be run on-demand.
- Definition Editor. The definition editor provides a JSON and YAML view of your pipe definition. The definition editor is particularly useful for sharing pipe definitions by copying and pasting the YAML; see sample pipes for examples.
Further, the Pipe Builder includes other Properties and Controls for administrative options like setting a name or alias as well as, of course, activating your pipe.
At the top of the Pipe Builder is the Input section. This is an optional section used to prototype your pipe with sample post parameters prior to deploying a pipe using parameters.
You may toggle the section open and closed using the arrow at the right.
The Input section enables you to customize the way you send the form data along with keys and values.
For an example of how this works, do the following:
Create a New Pipe.
In the Execute Task, replace the existing “hello world” code with the following Python script:
Run the pipe by pressing the “TEST” button and by default the script will return the five most recent posts from Hacker News, unless you change the parameter.
To test a different parameter, use the Input section to add a Key of
countand a value of
Run the pipe again by pressing the “TEST” button and the script will now only return the three most recent posts.
The Input section is meant for testing and prototyping your params. For deploying a pipe using parameters, see the deployment guide.
The second section of the Pipe Builder is for adding Tasks. The Tasks section enables you to create your pipe definition by chaining together different steps that run sequentially.
Tasks options include the following:
- Execute. The Execute Task is used to run Python or Node.js scripts. Scripts can be run inline or can be linked to remotely, including from GitHub.
- OAuth. The OAuth Task is a wizard that helps you set up or utilize an existing Connection, which uses OAuth. After saving, it creates an Execute Task with a script for making authenticated API calls to the service defined in the Connection
- Read. The Read Task is used to set up or utilize an existing Connection Storage Type Connection and choose the file you’d like to read. The output of the Read Task is written to stdout to be consumed by the following Task.
- Write. The Write Task is used to set up or utilize an existing Connection Storage Type Connection and choose the destination to which you’d like to write files. The input of the Write Task is the stdout from the previous task in your Pipe.
- Request. The Request Task is used to create a HTTP request. The HTTP response from the HTTP request in this task is written to stdout to be consumed by your next task.
- Convert. The Convert Task is a simple junction box that helps convert from one format to another. The input of the Convert Task is the stdout from the previous task in your Pipe and the output of the Convert Task is written to stdout to be consumed by the following Task.
- Email. The Email Task is used to set up an email notification.
- Connect. The Connect Task binds a Connection to a variable reference for use in the Pipe. This enables the Pipe to be portable by referencing the Connection without have to include the Connection’s
In a blank pipe with no existing Tasks, you may add a Task by clicking on any of the Task boxes.
In a pipe with existing Tasks, you may add a Task by clicking on the “+” button above or below the existing Task.
After clicking on the “+” button, you may click on a Task box to open up a new Task in your Pipe. The Pipe will run Tasks sequentially, from first Task to last.
To remove a Task, hover over the Task you wish to delete and a “Trash” icon will appear on the left:
Click the “Trash” icon and your Task will be removed from the pipe definition.
The third section of the Pipe Builder is the Output. The output section enables you to see the result of your pipe.
To see the result of your pipe, click the “TEST” button at the top right of the Pipe Builder.
If the result of the final Task is a stream of text, this will appear in the Output section. However, if the final step in your pipe is some other type, like, for example, an Email task, the output will remain empty, since the result of the pipe was not text, but a sent email.
The fourth section of the Pipe Builder provides Pipe Deployment options, including ways to schedule it, call it via API or run on-demand.
Deployment options include:
- Run on a schedule. This option enables you to run your pipe on a fixed schedule (e.g., cron)
- Run using an API endpoint. This option enables you to create an API endpoint, used along with your API key. API endpoints an be used in front-end or back-end code and can be used as a GET or POST webhook.
- Run using a Flex.io Web Interface. This option provides an on-demand view of the pipe, often for a non-technical end user. The web view enables a developer to hide the wiring of their pipe and only expose actions useful for an end-user (such on-demand running, authentication options and query parameters).
See the Deployment Guide for specifics on how to use and configure each of these deployment options.
The Definition Editor is tucked off into the sidebar on your left. Once opened, it provides a YAML or JSON view of your pipe, as well as the UI for a web front-end, if applicable.
Here is a YAML definition:
Here is a JSON defition:
The definition editor is particularly useful for sharing pipe definitions, as you can simply copy the YAML from one pipe and paste it into a new pipe.
To try this, do the following:
Create a New Pipe.
Open the Definition Editor by clicking on the icon (< >).
Once the Defition Editor panel is open, make sure the YAML button is chosen, then select everything in the definition and replace it with the following code:
This is a simple “hello world” example with the pipe name, description and alias pre-filled. As you paste in the new YAML, the pipe will update with the new pipe.
For further pipe examples you can copy and paste, see the sample pipes section.
Pipe Controls are action items to use when building pipes like the “TEST” button and Pipe Staus toggle. Pipe Properties include the setting the Pipe name, description and alias.
This section discusses the key controls in the Pipe Builder, which can be found in the header of the page:
The two buttons that control prototyping and activating your pipe are as follows:
- Run button. You can run your pipe as you build and protoype it by clicking the “TEST” button. As long as the pipe has not been activated, you can run it manually in this way.
- Activation Status toggle. The “ON/OFF” toggle provides the status of the pipe as either active or inactive. When a pipe is inactive, you can manually run it, while building and testing it. Once you wish to deploy it, such as scheduling or using it via API endpoint, you must turn your pipe “ON” to activate it. When active, you will no longer be able to edit the pipe. However, you may deactivate the pipe at any time to edit it.
The pipe properties section provides you a place to edit information about your pipe. To do so click on the edit (pencil) icon in the header next to the Pipe name. When you do, the Pipe properties will open:
Pipe properties enables you to add or edit the following:
- Name. You can provide a name for your pipe here.
- Description. You can enter a long form description of your pipe here.
- API endpoint and alias. You can access the API endpoint for your pipe here as well as in the Deployment section. If you copy your endpoint without adding an alias, it will include a unique
eidstring. However, instead of using the
eid, you may enter a more human-readable alias and use that to reference your pipe instead. Aliases should to be between 3 and 80 characters long.
This pipe analyzes job postings on GitHub Jobs chaining together different execute tasks in different languages (Nodejs with Lodash and Python with Pandas). Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List.
This pipe validates input with a server-side logic that hides data used to validate it (with Python Cerberus). Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List.
This pipe gets the top 5 stories from the Firebase Hacker News Feed and delivers via email. Copy the YAML below and paste it into the Definition Editor to add the pipe to your Pipe List. After adding the pipe, update the Email Task with your email address.