Storing Data For Consumption in Python

Question

I have a program which needs some constant data, in JSON-like format. However, this data only needs to be consumed by my Python program, and by making it Python code, I can include types like datetime.datetime and enums, as well as custom types and any other type I wish to use. It would also improve performance, as parsing JSON is skipped in this. Would it be a bad idea to provide my data as Python source code, using Python data types and import it when I need to use it? Are there downsides to this approach?

I am talking about storing the data in variables initiated when the program is run. In particular, I want to use enums; of course these can be replaced by strings, but I feel that the likelyhood for bugs and other mistakes is higher when using strings. I'd also like to be able to use objects with methods on them. This is data that is meant to be written by humans and read by programs.

Answer 1

One downsite is mainly the initial conversion effort, together with the potential maintenance effort, which needs to be compared to the effort you will have for parsing the JSON-like format (especially when it is not JSON, but only "JSON-like", as you wrote).

This means, it depends on how much data you have to process, how complex it is, how the original source looks like and how often you expect changes / maintenance requirements.

• If you have a half dozen fields and numbers, and the data changes rarely, just go ahead, use some Python code and don't overthink this.

• However, if the amount of data is a lot bigger, you have some input files for the data in some strictly defined format and you expect regular changes, converting the data manually to Python code can become quickly tedious. Hence, this becomes a trade-off, where at some point the initial effort for implementing a parser may become less than the manual maintenance effort over time.

• You could also use the parser to generate Python code with data initialization code - but that would only be a performance optimization and maybe not worth the effort.

Hence, my recommendation here is:

• Start with the most simple solution first which works, maybe manual creation of the Python code.

• Write a parser when you think the effort is worth it

• Don't worry about performance for processing the data until it starts to bother you.

There is another thing to keep in mind: who is actually responsible for maintaining the data?

• is it the Python programmer who has no trouble to maintain the data directly in source code?

• or is it some business person who has no idea what Python is, but can work with text files or spreadsheet data?

If it is the latter, you better let the business person maintain the data in some spreadsheet and parse the data from there.