Python script for parsing and using CSV data

Question

I am given a CSV file of stations, with data like this:

station_id,date,temperature_c 68,2000.375,10.500 68,2000.542,5.400 68,2000.958,23.000 68,2001.125,20.400 68,2001.292,13.300 68,2001.375,10.400 68,2001.958,21.800 68,2002.208,15.500 

and so on for many different station_ids.

Then I want to create a Python program that (1) gives the minimum reading (the third column) and (2) the station with the maximum "travel distance" with its readings. Thus if a station has 3 readings of -5,0,8 then that would mean a travel distance of 13. This can take an optional date range. Here is what I did.

#!/usr/bin/python from collections import defaultdict import csv import random import sys # In order to track each station's statistics, we'll create a Station class # to hold the data on a per-station basis. class Station: def __init__(self): self.readings = [] self.minimum = 99999999.0 self.travel = 0 # travel holds the change in temperature reading-by-reading def get_travel(self): return self.travel def set_travel(self, n): self.travel += abs(n) # getter & setter for station minimums def get_minimum(self): return self.minimum def set_minimum(self, n): self.minimum = n # infrastructure for future code expansion def get_readings(self): return self.readings def set_readings(self, date, temp): self.readings.append({ "date" : date, "temp" : temp}) """ Reporter class handles a list of Stations """ class Reporter: def __init__(self): # stations dict with entries for holding the specified stats. self.stations = defaultdict(Station) self.global_minimum = { "station" : "default", "date" : 1, "temp" : 9999999 } self.longest_travel = { "station" : "default", "range" : 0 } """ Determines which station recorded the coldest temperature args: CSV file returns: dict with data """ def minimum_temperature(self, filename): with open(filename, 'r') as datafile: try: csv_reader = csv.reader(datafile) next(csv_reader) # reading line-by-line since CSV could be a big file for row in csv_reader: station, date, temp = row # save the station's readings self.stations[station].set_readings(date, temp) temp = float(temp) if (temp < self.stations[station].get_minimum()): self.stations[station].set_minimum(temp) if(temp < self.global_minimum["temp"]): self.global_minimum = { "station" : station, "temp" : temp, "date" : date } # The specs state that in the event that a tie occurs simply return # one pair at random. if (temp == self.global_minimum["temp"]): if (random.randint(1,100) % 2 == 0): self.global_minimum = { "station" : station, "date" : date, "temp" : temp } except csv.Error as e: sys.exit('file {}, line {}: {}'.format(filename, reader.line_num, e)) return self.global_minimum """ Determines which station "traveled" the most args: CSV file, begin date (optional), end date (optional) returns: dict with data """ def max_travel(self,filename,begin=1.0,end=9999.9): with open(filename, 'r') as datafile: try: csv_reader = csv.reader(datafile) next(csv_reader) # reading line-by-line since CSV could be a big file for row in csv_reader: station, date, temp = row # save for future expansion self.stations[station].set_readings(date, temp) date = float(date) if date > begin and date < end: temp = float(temp) self.stations[station].set_travel(temp) travel = self.stations[station].get_travel() if ( travel > self.longest_travel["range"]): self.longest_travel = { "station" : station, "range" : travel } except csv.Error as e: sys.exit('file {}, line {}: {}'.format(filename, reader.line_num, e)) return self.longest_travel if __name__ == "__main__": csv_file = sys.argv[1] # fetch lowest temperature reporter = Reporter() global_minimum = reporter.minimum_temperature(csv_file) print("station {} had the global minimum on {}".format(global_minimum["station"], global_minimum["date"])) # fetch maximum travel overall longest_travel = reporter.max_travel(csv_file) print("station {} had the greatest travel at {}".format(longest_travel["station"], longest_travel["range"])) # now try a date range reporter2 = Reporter() begin = 2001.0 end = 2006.0 longest_travel = reporter2.max_travel(csv_file,begin,end) print("for {} to {}, station {} had the greatest travel at {}".format(begin, end, longest_travel["station"], longest_travel["range"])) 

I'm particularly interested in speeding it up and memory usage but also how to Pythonically deal with with getters/setters.

Answer 1

Getters and Setters

If you are to use this pattern, the @property decorator will accomplish this in a more pythonic way:

class Station: def __init__(self): self.readings = [] self._minimum = 99999999.0 self._travel = 0 @property def minimum(self, value): return self._minimum @minimum.setter def minimum(self, value): self._minimum = value @property def travel(self, value): return self._travel @travel.setter def travel(self, value): self._travel = abs(value) def add_reading(self, date, temp): self.readings.append({ "date" : date, "temp" : temp}) # access the elements this way station = Station() print(station.travel) 0 station.travel += 1 print(station.travel) 1 

However, the setter pattern doesn't really make sense for your readings, because you aren't really setting, you're appending. So keep a function that appends a reading to the readings list, it's more explicit this way. I wouldn't expect:

station.readings = ('date', 'reading') 

to do anything other than set the attribute, but it actually has a surprising side-effect.

Spacing

Your indentation should be four spaces, not two.

Answer 2

• Probably a typo in your except clauses; I suspect it's supposed to say csv_reader.line_num.
• What is going on with the "travel"? What you've actually written is "sum of absolute values", but I assume what you actually want is either "max minus min" or "sum of day-to-day deltas".
• My opinion is that getters/setters/properties are inherently un-pythonic. More importantly, they're opaque. They look like attributes but they're functions that can have side-effects. Don't hide the possibility of side-effects from the audience; do just use an actual attribute wwh
• I like the DictReader provided by the csv module. It's not everything we might ask for, but it'll save us accidentally handling malformed data.
• You're doing a lot of work to handle the file in a stream, but you're also reading the file multiple times and you're keeping all the data in memory once it's read.
  • This could get buggy. For example, in your main, you re-use reporter for the second pass, which means it will contain every datapoint in duplicate.
  • You should pick one: Either get everything you want in a single pass, or read all the data into an appropriate structure and run queries on that. Since you specify speed and memory usage as concerns, I'll assume you want the first choice (but I suspect that's the wrong decision).
• You probably don't need a Reporter class at all. It's not helping you manage any state (that you want to keep) across functions, so just declare your functions in the top namespace.