Filtering Data in Python (Example of Functional Programming Approach)

(Note: Please see my latest posts at my new blog!)

Let’s say you are doing a cancer study and have a list of patients of various ages in a tab-delimited file. You want to limit the study to patients who are 60 years or older. One way you could do this is use a for loop and process the data one row at a time and remove any patients below your threshold. Another way is to insert the data into a SQL database and use a WHERE clause to filter the data and then extract it back out. (That’s pretty desperate, but I know it happens!)

One simple way to do this in Python is to use the filter function. Let’s say you pull the data from the file into a series of tuples.

file = open("Patients.csv", 'r')
patients = [line.rstrip().split('\t') for line in file]

Now suppose age is the 3rd column in the data. You need to create a small function to determine whether a tuple meets the criteria:

def f(x): return int(x[2]) >= 60

Then you use the filter function and apply that function to the data.

matches = filter(f, patients)

This is a contrived example, but I hope it illustrates a beginning of how you might use functional programming and that it gives you a flavor for how this can be a powerful approach.