Process a CSV file using Javascript

Create a pipe that reads a CSV file and processes each row using Javascript.

Overview

This template shows how to process and transform the data in a CSV file with a Javascript function. You can use this to create a pipe for batch processing CSV files, or to add a data processing step to a serverless data feed.

Code template

How it works

When Flexio.pipe() is called, a local pipe object is created and returned. This pipe object contains information about the pipe, including an array of tasks that will be executed sequentially when the pipe is run.

In this case, a pipe is created and three tasks are added to the pipe for execution. The first task, request makes an HTTP request to get a CSV that will get passed on to subsequent tasks. The second task, convert converts this CSV to a table so that the individual rows and columns can be processed. The third task, javascript executes javascript code defined in the function(context) function, iterating over each row and writing the converted row to the output with context.output.write(JSON.stringify(row).toLowerCase()+"\n").

When run() is called on the pipe, the tasks in the pipe object are sent to Flex.io, and the tasks are executed sequentially, with the results of each step being passed onto the next step: the CSV is read from the specified URL, formatted into a table, and read row-by-row, converting each row to lowercase and outputting it. After the tasks are executed, the final output is returned to the caller, and is echoed to the user interface in a callback function.

Extend it

Example 1

Create a new column in a CSV from other columns; remove unwanted columns from a CSV.

Example 2

Calculate statistics on a CSV.