Convert and Clean a CSV file using Python
With the Flex.io API, you can quickly create a data feed that reads and converts a CSV file and performs a cleaning process on each row with Python.
This template shows how to process and transform the data in a CSV file with a Python function. You can use this to create a pipe for batch processing CSV files, or to add a data processing step to a serverless data feed.
Flexio.pipe() is called, a local pipe object is created and returned. This pipe object contains information about the pipe, including an array of tasks that will be executed sequentially when the pipe is run.
In this case, a pipe is created and three tasks are added to the pipe for execution. The first task,
request makes an HTTP request to get a CSV that will get passed on to subsequent tasks. The second task,
convert converts this CSV to a table so that the individual rows and columns can be processed. The third task,
python executes python code defined in the
flexio_handler(context) function, iterating over each row and writing the converted row to the output with
context.output.write(json.dumps(row).lower() + "\\n").
run() is called on the pipe, the tasks in the pipe object are sent to Flex.io, and the tasks are executed sequentially, with the results of each step being passed onto the next step: the CSV is read from the specified URL, formatted into a table, and read row-by-row, converting each row to lowercase and outputting it. After the tasks are executed, the final output is returned to the caller, and is echoed to the user interface in a callback function.
Create a new column in a CSV from other columns; remove unwanted columns from a CSV.
Calculate statistics on a CSV.