Quick Start

Quick Start

Welcome

Welcome to Flex.io! Flex.io is an API for building smart, flexible data feeds with a few lines of code. These data feeds are defined and run using Flex.io pipes.

In this quick start guide, we’ll show you how to:

Got more questions? Check out our FAQ or ask us for help; we’d love to hear from you. So, without further ado, let’s get started!

What is a pipe?

A pipe is a collection of tasks that run sequentially. These tasks can read and write from cloud storage, databases and APIs, which are set up using connections. Tasks can also perform a variety of other operations, including listing contents in cloud storage, making API requests, converting file format and running custom Python or Javascript code on content being passed through the pipe.

When a pipe is run, the pipe input is passed to the first task, which operates on the input and returns an output; then the output of that task is passed to the next task and so on, until the tasks are finished and the pipe returns a final output.

Here is an example pipe using the Flex.io Javascript SDK:

Click the ‘Run’ button in the snippet above to see the output. Here’s a breakdown of this pipe into its individual components:

When the pipe is created, the request and python tasks are added to the pipe. When run() is called on the pipe, the tasks in the pipe object are sent to Flex.io, and the tasks are executed sequentially:

Now, let’s get you building your own.

Get your API key

First, in order to use the API, you’ll need to get an API Key. Go to the sign up dialog and register. Once you’ve signed up, you’ll see a welcome dialog that has your API key.

You can also get your API Key from your dashboard page in the Flex.io application. To do this, sign in into Flex.io, which will take you to the dashboard page. On the dashboard page, you’ll find your API key under the section labelled “API Keys”.

Make an API request

You can make an API request from several places:

Flex.io provides a variety of ways to access the API, including HTTP request, command line syntax, SDK and the Flex.io web app console. The examples in this guide will focus primarily on two of the more common implementations: CURL for the command line and the Javascript SDK most other uses.

Note that for each of the scripts below, you will replace the following with your own credentials:

API request: Command line

To call the API from the command line, simply issue an HTTP request from a command line and include your API key in the “Authorization” HTTP header. Here is an example using CURL:

curl -X GET 'https://api.flex.io/v1/me/pipes' --header "Authorization: Bearer {token}"

API request: Client-side code

To call the API from client-side code, you can simply issue an HTTP request to the API, like this:

$(function() {
$.ajax({
type: 'GET',
url: 'https://api.flex.io/v1/me/pipes',
beforeSend: function(xhr) {
xhr.setRequestHeader('Authorization', 'Bearer {token}')
},
dataType: 'json',
success: function() {
}
}})

However, in most cases, you’ll use the Javascript SDK to create rich pipes in your code. To use the Javascript SDK in your client-side code, note that you must first include a link to it as follows:

<script src="https://cdn.jsdelivr.net/npm/flexio-sdk-js@latest"></script>

The code below shows how to get a list of all pipes in your account from the client-side using the SDK:

<!doctype html>
<body>
</body>
</html>

<script src="https://cdn.jsdelivr.net/npm/flexio-sdk-js@latest"></script>
<script>
(function(window, document, Flexio, undefined) {
Flexio.setup('{token}');
Flexio.pipes.list(function(err, response) {
console.log(response.text)
})
})(window, document, Flexio)
<script>

API request: Server-side code

To call the API from server-side code, you can simply issue an HTTP request to the API; here’s how this would look using Python Requests:

import requests
url = 'https://api.flex.io/v1/me/pipes'
headers = {'Authorization': 'Bearer {token}'}
r = requests.get(url, headers=headers)
print r.json

If you’re using Node.js, you can use the Flex.io Javascript SDK to interact with the Flex.io API. To do this, first install the SDK using the npm package manager at your command prompt:

npm install flexio-sdk-js

For more information about installing the SDK with npm, see the Javascript SDK documention.

Then, in Node.js, you can get a list of pipes as follows with the SDK:

var Flexio = require('flexio-sdk-js')
Flexio.setup('{token}');
Flexio.pipes.list(function(err, response) {
console.log(response.text)
})

Run your first pipe

Now let’s run our first pipe. We’ll begin by using a sample pipe provided when you signed up for your account. To run a pipe, you simply reference it by its identifier, either its automatically assigned entity identifier (eid) or its alias (alias), which is a logical name you give to the pipe that can be used in place of the identifier in API calls.

Get a list of your pipe identifiers

To find an eid or pipe alias we can use, let’s find out which pipes are currently in your account. For the scripts below, you will replace the following with your own credentials:


curl -X GET 'https://api.flex.io/v1/me/pipes' --header "Authorization: Bearer {token}"

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipes.list(function(err, response) {
console.log(response.text)
})

After running, you should get back a list of the pipes already in your account, including two sample pipes:

[
{
"eid": "l69wq7csx6t6",
"eid_type": "PIP",
"eid_status": "A",
"alias": "contact-refinement",
},
{
"eid": "h0cdcmy0g0xf",
"eid_type": "PIP",
"eid_status": "A",
"alias": "github-commits",
}
]

The eid the assigned identifier; alias is the alias of the pipe.

Run one of your pipes using its identifier

To run a pipe, copy either the eid or alias from a pipe in your list. For the scripts below, you will replace the following with your own credentials:


curl -X POST 'https://api.flex.io/v1/me/pipes/{pipe-alias}/run' --header "Authorization: Bearer {token}"

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipes.run('{pipe-alias}', function(err, response) {
console.log(response.text)
})

When the pipe runs, you’ll see the result of the pipe operation echoed to stdout in the command line or the console.

Create your first pipe

Now that we’ve run a pipe, let’s create a new one. In this section, you’ll do the following:

Note that for the scripts below, you will replace the following with your own credentials:

Creata a new pipe with one task

This is a simple pipe that uses a request task to get a CSV file from a GitHub URL.


curl -X POST 'https://api.flex.io/v1/me/pipes'
--header "Authorization: Bearer {token}"
--header "Content-Type: application/json"
--data '
{
"name": "{pipe-name}",
"alias": "{pipe-alias}",
"description": "{pipe-description}",
"task": {
"op": "request",
"params": {
"url": "https://raw.githubusercontent.com/flexiodata/data/master/mockaroo/names-and-ip-addresses.csv"
}
}
}
'

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipe()
.request('https://raw.githubusercontent.com/flexiodata/data/master/mockaroo/names-and-ip-addresses.csv')
.save({ alias: '{pipe-alias}' })
})

When the pipe runs, you’ll see the result of the pipe operation echoed to stdout in the command line or the console like this:

id,first_name,last_name,email,gender,ip_address
1,Kimbra,Pettigrew,kpettigrew0@dedecms.com,Female,253.123.30.10
2,Cornall,Rowlett,crowlett1@sitemeter.com,Male,223.20.241.246
3,May,Robelow,mrobelow2@is.gd,Female,200.221.39.69
4,Sibyl,Vanyashkin,svanyashkin3@unc.edu,Female,103.201.21.61
5,Nikolaus,Bernette,nbernette4@parallels.com,Male,192.249.74.49
6,Tiffie,Leyes,tleyes5@multiply.com,Female,152.35.251.190
7,Julianne,Dalinder,jdalinder6@dedecms.com,Female,26.237.107.12
8,Grantham,Gammett,ggammett7@skype.com,Male,87.137.168.80
9,Fairfax,Hancock,fhancock8@homestead.com,Male,171.235.144.3
10,Leopold,Buss,lbuss9@dedecms.com,Male,50.81.190.48
etc.

Confirm the new pipe has been added to your account

By running this code, you’ve added a new pipe to your account that requests a CSV file from a GitHub URL. To confirm the pipe was created, re-run the pipe list API call from the section above to return a list of all pipes from our account.

For clarity, let’s say you created a pipe alias called gollum-csves-from-githubses in the above script. You would now return a list with your original sample pipes as well as the new pipe:

[
{
"eid": "l69wq7csx6t6",
"eid_type": "PIP",
"eid_status": "A",
"alias": "gollum-contact-refinement",
},
{
"eid": "h0cdcmy0g0xf",
"eid_type": "PIP",
"eid_status": "A",
"alias": "gollum-github-commits",
}
{
"eid": "x7trcmy0h9qr",
"eid_type": "PIP",
"eid_status": "A",
"alias": "gollum-csves-from-githubses",
}
]

Create a new pipe with two tasks

In the previous example, we used a single request task to get a file. In this example, we’ll add a second convert task to convert the file we requested from CSV to JSON format. When we want to run multiple tasks, we use a sequence task to group the tasks to run in sequence. The initial input for the sequence task is passed to the first task in the sequence, then the output of the first task in the sequence will be passed onto the next task in the sequence, and so on, until the last task in the sequence. The output of the last task in the sequence is returned as the output of the sequence task. Here’s an example showing the sequence task wrapping a request and a convert task:


curl -X POST 'https://api.flex.io/v1/me/pipes'
--header "Authorization: Bearer {token}"
--header "Content-Type: application/json"
--data '
{
"name": "{pipe-name}",
"alias": "{pipe-alias}",
"description": "{pipe-description}",
"task": {
"op": "sequence"
"params": {
"items": [
{
"op": "request",
"params": {
"url": "https://raw.githubusercontent.com/flexiodata/data/master/mockaroo/names-and-ip-addresses.csv"
}
},
{
"op": "convert",
"params": {
"to": "csv"
"from": "json"
}
}
]
}
}
}
'

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipe()
.request('https://raw.githubusercontent.com/flexiodata/data/master/mockaroo/names-and-ip-addresses.csv')
.convert('csv', 'json')
.save({ alias: '{pipe-alias}' })
})

When the pipe runs, you’ll see the result of the pipe operation echoed to stdout in the command line or the console like this:

[
{
"id": "1",
"first_name": "Kimbra",
"last_name": "Pettigrew",
"email": "kpettigrew0@dedecms.com",
"gender": "Female",
"ip_address": "253.123.30.10"
},
{
"id": "2",
"first_name": "Cornall",
"last_name": "Rowlett",
"email": "crowlett1@sitemeter.com",
"gender": "Male",
"ip_address": "223.20.241.246"
},
{
"id": "3",
"first_name": "May",
"last_name": "Robelow",
"email": "mrobelow2@is.gd",
"gender": "Female",
"ip_address": "200.221.39.69"
},
{
"id": "4",
"first_name": "Sibyl",
"last_name": "Vanyashkin",
"email": "svanyashkin3@unc.edu",
"gender": "Female",
"ip_address": "103.201.21.61"
},
etc.

Adding a second task is an example of chaining pipe tasks together to compactly define logic thats reads and transforms data. For more examples like this, see the Examples section below.

Create an inline pipe

In certain cases, you may want to define and run code inline on the client-side without having to worry about names and identifiers. For example, you may wish to create a fully-client side upload form that runs when a user clicks a button.

To do this, use the SDK. Using the two-task pipe above, here’s a complete example for how you define and run a pipe inline in client-side code:

<!doctype html>
<body>
</body>
</html>

<script src="https://cdn.jsdelivr.net/npm/flexio-sdk-js@latest"></script>
<script>
(function(window, document, Flexio, undefined) {
Flexio.setup('{token}');
Flexio.pipe()
.request('https://raw.githubusercontent.com/flexiodata/data/master/mockaroo/names-and-ip-addresses.csv')
.convert('csv', 'json')
.run(function(err, response) {
console.log(response.text)
});
})(window, document, Flexio)
<script>

Read and write to cloud storage

To read and write from cloud storage, you’ll need to authenticate with your storage provider inside the Flex.io web application.

Flex.io Cloud Storage

To create a connection to your cloud storage, do the following:

  1. Log into the Flex.io web application
  2. In the menu on the left, select the Storage tab.
  3. To create a new storage connection, select the New button in the upper-right
  4. In the dialog that appears, select your cloud storage service
  5. Enter your credentials and authenticate with the provider and/or test your connection
  6. For ease of referencing in your code, add an alias such as my-cloud-storage
  7. Click the Create Storage button to add the storage service to your account

Once you’ve added storage in Flex.io, you can now read and write from files in that connection using the API.

Read from a file in your cloud storage

Once you’ve added your cloud storage, you’re ready to create a pipe that reads files from it. For the scripts below, you will replace the following with your own credentials:


curl -X POST 'https://api.flex.io/v1/me/pipes'
--header "Authorization: Bearer {token}"
--header "Content-Type: application/json"
--data '
{
"name": "{pipe-name}",
"alias": "{pipe-alias}",
"description": "{pipe-description}",
"task": {
"op": "read",
"params": {
"path": "/{storage-alias}/{path}"
}
}
}
'

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipe()
.read('/{storage-alias}/{path}')
.save({ alias: '{pipe-alias}' })
})

For clarity, if your API key is dpbxyygskrrjnntwhkwp, pipe-name is Read fileses from dropbox, pipe-alias is gollum-read-from-storage, desription is We reads our precious fileses, storage-alias is gollum-dropbox and path is myprecious/ring.txt, your script would look like this:


curl -X POST 'https://api.flex.io/v1/me/pipes'
--header "Authorization: Bearer dpbxyygskrrjnntwhkwp"
--header "Content-Type: application/json"
--data '
{
"name": "Read fileses from dropbox",
"alias": "gollum-read-from-storage",
"description": "We reads our precious fileses",
"task": {
"op": "read",
"params": {
"path": "/gollum-dropbox/myprecious/ring.txt"
}
}
}
'

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('dpbxyygskrrjnntwhkwp');
Flexio.pipe()
.read('/gollum-dropbox/myprecious/ring.txt')
.save({ alias: 'gollum-read-from-storage' })
})

Write to a file in your cloud storage

Writing to cloud storage is exactly the same methodology as reading from storage (see above), except that you use the write task instead of the read task. For your {path}, you can simply choose an output folder instead of specifying the file name (e.g., my-folder/output).


curl -X POST 'https://api.flex.io/v1/me/pipes'
--header "Authorization: Bearer {token}"
--header "Content-Type: application/json"
--data '
{
"name": "{pipe-name}",
"alias": "{pipe-alias}",
"description": "{pipe-description}",
"task": {
"op": "write",
"params": {
"path": "/{storage-alias}/{path}"
}
}
}
'

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipe()
.write('/{storage-alias}/{path}')
.save({ alias: '{pipe-alias}' })
})

Move a file from one cloud storage location to another

Now, let’s move a file from one cloud storage location to another. First, make sure you have two cloud storage accounts in the Flex.io app. Then, we’ll simply add the read and write tasks sequentially:


curl -X POST 'https://api.flex.io/v1/me/pipes'
--header "Authorization: Bearer {token}"
--header "Content-Type: application/json"
--data '
{
"name": "{pipe-name}",
"alias": "{pipe-alias}",
"description": "{pipe-description}",
"task": {
"op": "sequence",
"params": {
"items": [
{
"op": "read",
"params": {
"path": "/{storage-alias-1}/{path}"
}
},
{
"op": "write",
"params": {
"path": "/{storage-alias-2}/{path}"
}
}
]
}
}
}
'

var Flexio = require('flexio-sdk-js') // not needed in client-side code
Flexio.setup('{token}');
Flexio.pipe()
.read('/{storage-alias-1}/{path}')
.write('/{storage-alias-2}/{path}')
.save({ alias: '{pipe-alias}' })
})

Examples

Hello world

Use echo to output text:

List the files in cloud storage

Use list to read the directories and files in cloud storage:

Read from cloud storage

Use read to read from a file in cloud storage:

Write to cloud storage

Use write to write to a file in cloud storage:

Convert a file format

Use convert to convert files from one format to another; here’s how to convert a CSV file to JSON:

Render a webpage as image

Use render to render a webpage as a png image:

Execute Javascript

Use javascript to execute javascript:

Execute Python

Use python to execute python:

Execute Python with Pandas

Use request to read a CSV and python to create a pivot table using the pandas library:

Execute Python with Matplotlib

Use request to read data from an API, convert to convert from JSON to a csv format and python to create a chart using the Matplotlib library:

Additional Info

FAQ

What can I do with Flexio?

Here are some examples:

Who uses Flex.io?

We’ve built Flex.io for developers to easily create serverless data feeds, in a programmatic way without needing to set up the normal data infrastructure like servers and storage. If this appeals to you, please let us know how we can help. If you see something you think you’d like to do but don’t have an explicit template or tutorial, just give us a shout and we’ll be happy to assist!

How does Flex.io work?

Flex.io helps developers quickly create serverless data feeds using programmatic pipes. Pipes are created and run in the Flex.io cloud service using the Flex.io API or the Flex.io Javascript SDK, which is built on top of the API.

Pipes are a collection of tasks that run sequentially. These tasks can read and write from cloud storage, databases and APIs, which are set up using connections. Tasks can also perform a variety of other operations, including listing contents in cloud storage, making API requests, converting file format and running custom Python or Javascript code on content being passed through the pipe.

When a pipe is run, the pipe input is passed to the first task, which operates on the input and returns an output; then the output of that task is passed to the next task and so on, until the tasks are finished and the pipe returns a final output.

Can you give me an example of a Flex.io pipe?

When the pipe is created, the request and python tasks are added to the pipe. When run() is called on the pipe, the tasks in the pipe object are sent to Flex.io, and the tasks are executed sequentially: the request task issues an HTTP request to the specified URL, and the CSV is read and passed onto the next step; the python step takes this CSV from the output of the request step, summarizes it using Python’s Pandas pivot function, and then passes the final pipe output to the run() callback function in the response.text variable.

How much does Flex.io cost?

Here’s our current pricing.

How do I get started?

You’re in the right place! Start reading this quick start guide from the top.

Where can I find out more?

To see numerous code examples and run them, take a look at the Flex.io templates:

To see how to build specific solutions with full-fledged demos, checkout out the Flex.io documentation:

To find out more about the product and the pricing, take a look at the following:

Who are you guys anyway?

Thanks for asking! Here’s our team.

Help

Need help? Click on the message icon in the lower-right of this page or the application and type away. Or email us at support@flex.io. We’d love to help you!

Contact Us

Want to talk? Give us a shout!

Talk to Us

hello@flex.io
support@flex.io

Visit Us

Flex.io
1 North State Street
Suite 1500
Chicago, IL 60602

Documentation

Take a look at these to find out more about the SDK and API:

Tutorials

Check out our tutorials to find out how to solve specific problems.