Convert and Clean a CSV file using Python

With the API, you can quickly create a data feed that reads and converts a CSV file and performs a cleaning process on each row with Python.


This template shows how to process and transform the data in a CSV file with a Python function. You can use this to create a pipe for batch processing CSV files, or to add a data processing step to a serverless data feed.

Code template

How it works

When Flexio.pipe() is called, a local pipe object is created and returned. This pipe object contains information about the pipe, including an array of tasks that will be executed sequentially when the pipe is run.

In this case, a pipe is created and three tasks are added to the pipe for execution. The first task, request makes an HTTP request to get a CSV that will get passed on to subsequent tasks. The second task, convert converts this CSV to a table so that the individual rows and columns can be processed. The third task, python executes python code defined in the flexio_handler(context) function, iterating over each row and writing the converted row to the output with context.output.write(json.dumps(row).lower() + "\\n").

When run() is called on the pipe, the tasks in the pipe object are sent to, and the tasks are executed sequentially, with the results of each step being passed onto the next step: the CSV is read from the specified URL, formatted into a table, and read row-by-row, converting each row to lowercase and outputting it. After the tasks are executed, the final output is returned to the caller, and is echoed to the user interface in a callback function.

Extend it

Example 1

Create a new column in a CSV from other columns; remove unwanted columns from a CSV.

Example 2

Calculate statistics on a CSV.