Welcome

Thanks for stopping by! This getting started guide will get you building and running pipes with Flex.io. It’ll also introduce you to the command line and API. From there, we’ll point you to the docs and our support team to help you on your way.


Flex.io enables you to build and run cloud-based pipes.

Simply put, a pipe does the following:

Once you’ve built a pipe, you can run it manually from within the application, from the command line or from an API. Pipes can also be scheduled from the web app and you may also run pipes that have been shared with you by others.

In this getting started guide, we’ll show you how to:

When you’re ready to start building pipes, here are some handy reference guides:

Along the way, we’re here to help, so please contact us if you have questions, feedback or ideas.

Let’s jump in!

Running a pipe

Within the Web App, you can run a pipe by simply clicking on the Run Pipe button at the top-right of the pipe builder.

Run Pipe Button

Let’s try it.

  1. Sign into Flex.io: https://www.flex.io/app/signin
  2. Upon sign-in, you’ll see the Project List; click on ‘Sample Project’ to open the pipes for that project.
  3. You’ll now see two sample pipes, 'Contact Refinement’ and 'GitHub Commits’. Click on 'GitHub Commits’ to open the pipe builder.
  4. Click the Run Pipe button at the top-right.

When the pipe is running, you’ll see a series of status bars showing the progress. Then, when the pipe is finished, you’ll see the result below the pipe steps. You can click on any step to see the command and corresponding output for that given step. Further, you may modify the pipe by editing, removing or adding pipes steps.

Building a pipe

You can build a pipe in the Web App using the Pipe Builder. A longer walkthrough can be found in the Web App Documentation. Further, a video detailing the process can be found on YouTube here.

For this tutorial, we’ll start with a blank pipe and then add an input and a few other steps.

Create a New Pipe

Let’s start by creating a new pipe.

  1. Sign in to Flex.io and navigate to the Sample Project, as discussed above.

     

    New Pipe Button

     

  2. Click on the New Pipe Button at the top right of the page and a New Pipe dialog will appear.

     

    New Pipe Dialog

     

  3. Enter the following:

    • Name: Getting Started
    • Alias: <username>-getting-started (note: <username> is the username you created when you signed up; so if your username is “paul”, you would have “paul-getting-started”)
    • Description: This is a getting started pipe

     

  4. Click the Create Pipe button and you will be taken to the pipe builder Overview tab. The Pipe Overview is where you can set up your input and ouput and see a summary of your pipe steps. If you click to the Pipe Builder tab you can view, add and edit processing steps.

     

    New Pipe Builder

     

Build a Pipe

Now, let’s add some pipe steps. We’ll add input, convert, merge, select, rename, and sort steps to demonstrate. Normally, you can set up your input and output in the overview tab. However, for this example, we’ll take a command-driven approach for all steps in the pipe.

  1. On to top-navigation bar, click the Pipe Builder tab and you will be taken to the pipe step view. Click the “Add a Step” button and an untitled step will appear, ready for your command. Then, in the command bar, paste in the following input command:

     

    input file: https://api.github.com/repos/flexiodata/docs-web-app/commits => docs-web-app-commits, https://api.github.com/repos/flexiodata/docs-getting-started/commits => docs-getting-started-commits, https://api.github.com/repos/flexiodata/docs-api/commits => docs-api-commits
    

     

     

    Input Command

     

  2. Click the Save Changes button. This will save your step, but not run your pipe.

     

    Input Command

     

  3. Click the Run button at the top right. This will run the input command and provide you a preview of the api json. As you build the pipe, you can choose when to refresh your previews by re-running.

     

    Input Command

     

  4. Click on the + button at the left to add another step. Paste the following convert command into the Command Bar to convert the unstructured text into a table and click Save Changes:

     

    convert to: table
    

     

  5. If you choose to click the Run button, you will see a table in the preview.

  6. Click on the + button to add another step. Paste the following merge command into the Command Bar to combine the separate tables from the previous step into one table and click Save Changes:

     

    merge
    

     

  7. If you choose to click the Run button, you will see a combined table in the preview.

  8. Click on the + button to add another step. Paste the following select command into the Command Bar to reduce the number of fields down to a limited set and select Save Changes:

     

    select col: commit.author.date, commit.author.name, commit.author.email, commit.committer.name, commit.committer.email, commit.message
    

     

  9. If you choose to click the Run button, you will see a preview populated with only the selected fields.

  10. Click on the + button to add another step. Paste the following rename command into the Command Bar to rename the columns and select Save Changes:

     

    rename col: commit.author.date=>author_date, commit.author.name=>author_name, commit.author.email=>author_email, commit.committer.name=>committer_name, commit.committer.email=>committer_email, commit.message=>commit_message
    

     

  11. If you choose to click the Run button, you will see a preview populated with the renamed fields.

  12. Click on the + button to add another step. Paste the following sort command into the Command Bar to sort the table and click Save Changes:

     

    sort col: author_date desc
    

     

  13. Click the Run button to refresh your pipe. The table in the preview will now populate with the sorted results.

At this point, you can keep adding steps to build up the pipe logic; you can also edit a step by changing the command and pressing the Save Changes button. To remove a step, hover over the tile next to the step and an ‘X’ will appear, then click to delete. Inputs and outputs may also be added directly in the pipe overview tab.

Please see the full pipe documentation for further details, along with all command and function syntax.

Using the command line

The Flex.io Command Line Interface (CLI) is a tool for managing and running your Flex.io services from the command line. With just one compact utility, you can integrate Flex.io services into your scripts and command-line environment, enabling direct integration with other systems and software.

Once you’ve installed and configured the CLI, you’ll be able to see a list of your pipes and run them from the client. You can even feed data from the command line into the pipe for processing, then get back the results.

Install the CLI

Several different versions of the command line are available; see here for a complete list of options. For demonstration purposes here, we’ll use the js client, which you can easily install with npm:

  1. Make sure you’ve downloaded and installed Node.js
  2. At the command prompt, enter the following command:

     

    $ npm install -g flexio
    
Configure the CLI

After installation, you’ll next need to configure the CLI with an API key. Create an API Key as follows:

  1. Sign into Flex.io
  2. Once you’ve signed into Flex.io, click on your user profile in the upper-right and click on ‘Account’ from the dropdown menu.
  3. In the Account area, select the 'API’ tab.
  4. In the API area, click on the Create API Key button to generate a new API key

     

    API Key

  5. Once you’ve generated the API key, go back to the command line and enter the following:

     

    $ flexio configure
    

     

  6. Next you’ll be prompted to enter your API key:

     

    $ flexio configure
    Configuring profile 'default'
    Please enter your API key: <paste your api key here>
    

     

Listing and Running your Pipes

Now that you’ve installed and configured the command line, you’re ready to list and run your pipes:

  1. To list your pipes, enter the following at the command prompt:

     

    $ flexio pipes list
    

     

  2. Once you’ve listed the pipes, browse the list. You’ll see the identifiers (eids) for your pipes along with their aliases and descriptions. For example, in your list of pipes, you should see the two sample pipes from your 'Sample Project’ repository with the descriptions “Review commits across multiple…” and “Select a subset of a list of contacts”. You should also see the pipe we built in the “Building a pipe” section with “Getting Started” name, alias, and description.

     

  3. In general, to run a pipe, enter a command like 'flexio pipes run <pipe-alias>’ at the command prompt, where <pipe-alias> is the alias for the pipe, which can be set when creating or editing a pipe.

     

    $ flexio pipes run <pipe-alias>
    

     

  4. Let’s run the 'Getting Started’ sample pipe, assuming the alias is 'paul-getting-started’:

     

    $ flexio pipes run paul-getting-started
    

     

  5. If we want to get back the data from the output, we add the “–output data” parameter to the command:

     

    $ flexio pipes run <pipe-alias> --output data
    

     

  6. Let’s get back the output for the 'Getting Started’ sample pipe, assuming the alias is 'paul-getting-started’:

     

    $ flexio pipes run paul-getting-started --output data
    -or-
    $ flexio pipes run paul-getting-started --output data > output.txt
    

     

  7. With the command line, it’s also possible to send input files or stdin to a pipe.

     

    $ flexio pipes run <pipe-alias> file.txt
    -or-
    $ flexio pipes run <pipe-alias> file.txt *.csv
    -or-
    $ echo Hello, World | flexio pipes run <pipe-alias>
    

     

Further details on the CLI and SDKs can be found in the API Reference Documentation.

Using the API

The Flex.io API allows users to list and run pipes, configure and manage the pipe process as well as read the pipe process output. Users can set inputs and read outputs so that files can be easily processed with the pipe logic at runtime.

The Flex.io API is organized around REST: pipes, processes, and other resources are accessed with URL endpoints and manipulated using GET, POST, and DELETE HTTP verbs. Results are returned as JSON. Errors are returned using HTTP error codes.

Accessing an API Key

To use the API, you’ll need an API key:

  1. Sign into Flex.io
  2. Once you’ve signed into Flex.io, click on your user profile in the upper-right and click on ‘Account’ from the dropdown menu.
  3. In the Account area, select the 'API’ tab.
  4. In the API area, click on the Create API Key button to generate a new API key

API Key

Examples

Once you have an API key, you can list your pipes by doing the following:

  1. Create an HTTP GET request on “https://www.flex.io/api/v1/pipes”
  2. Set the Authorization header to your API key: “Authorization: Bearer :token”.

For example, in CURL, this would look like:

curl -X GET 'https://www.flex.io/api/v1/pipes'
-H "Authorization: Bearer zxcvzxcvzxcvzxcv"

You can also run your pipes by doing the following:

  1. Create an HTTP POST request on “https://www.flex.io/api/v1/pipes/<pipe-alias>/run”
  2. Set the Authorization header to your API key: “Authorization: Bearer :token”

For example, in CURL, this would look like:

curl -X POST 'https://www.flex.io/api/v1/pipes/<pipe-alias>/run'
-H "Authorization: Bearer zxcvzxcvzxcvzxcv"

To run the “Getting Started” sample pipe, enter:

curl -X POST 'https://www.flex.io/api/v1/pipes/paul-getting-started/run'
-H "Authorization: Bearer zxcvzxcvzxcvzxcv"

Further details on the API can be found in the API Reference Documentation.

Getting further help

Flex.io is currently in beta, so while we whip everything into tip top shape, you’ll no doubt encounter some bugs and missing features.

But! Good news. We’re more than happy to lend a hand, either via email, phone or webcast.

Please shoot us a note at support@flex.io and we’ll hop to it! Thanks!

 

 

 

 

 

©2017 All Rights Reserved. Flex.io™ is a trademark of Gold Prairie LLC.