Reading a config file in python

Question

We're wanting to read data in from a config file and provide it to other python modules in a helpful way. We have found a number of ways to accomplish this that range from the module being easy to understand but verbose to use, to the module being a little more complicated but really nice to use.

Below I will present three different methods to read config file data. What are your opinions? Which method would you go with, or would you do something completely different?

The example config file

[section1] # The trailing slash of this url should be removed. # (In our actual code, we do a little more work to normalize the urls in the config file) url = http://example.com/ api_key = 12345 [section2] # These items are commented out to see how the config loader will handle defaults. # One of these should have a default, the other should throw a useful error # if an attempt is made to access it when it is not set. # with_default = # without_default = 

Method 1

Pros: Very simple to read and understand.

Cons: I have to do things like remove the trailing slash of the url manually outside this module. Also, I am unable to throw very useful errors for the end-user because I'm letting ConfigParser handle errors for me. It is also a little verbose to use this module.

from configparser import ConfigParser _defaults = { 'section1': { 'url': 'http://example.org', }, 'section2' : { 'with_default' : 'default-value', }, } mainconf = ConfigParser() mainconf.read_dict(_defaults) mainconf.read('./example.config') 

Usage

# Dummy server request function request = print from method1 import mainconf normalize_url = lambda url: url.rstrip('/') request(normalize_url(mainconf['section1']['url'])+'/v1/endpoint', { 'api-key': mainconf['section1']['api_key'], 'data': mainconf['section2']['with_default'], }) # This should throw an error. mainconf['section2']['without_default'] 

Method 2

Pros: I can now provide more helpful errors. I can also access some (but not all) attributes simply by doing things like method2.url, which is much cleaner.

Cons: Inconsistency because only some attributes can be accessed through method2.url while other require using get().

from configparser import ConfigParser _normalize_url = lambda url: url.rstrip('/') _defaults = { 'section1': { 'url': 'http://example.org', }, 'section2' : { 'with_default' : 'default-value', }, } _mainconf = ConfigParser() _mainconf.read_dict(_defaults) _mainconf.read('./example.config') def get(section, item): try: ret = _mainconf[section][item] except KeyError: raise LookupError(f'{item} in section {section} not found in config file') from None if section == 'section1' and item == 'url': ret = _normalize_url(ret) return ret # Provide items on a module level that has defaults. # If they don't have defaults, you must go through get() so a good error would # be thrown if the item is not found. url = get('section1', 'url') with_default = get('section2', 'with_default') 

Usage

# Dummy server request function request = print import method2 request(method2.url+'/v1/endpoint', { 'api-key': method2.get('section1', 'api_key'), 'data': method2.with_default, }) # This should throw a helpful error. method2.get('section2', 'without_default') 

Method 3

Pros: All attributes can be accessed simply by doing mainconf.url, nice! It also throws good errors for missing items.

Cons: The code is a little bit more complected to make this work. Also, it might feel a little too magical to override __getattr__.

from configparser import ConfigParser _normalize_url = lambda url: url.rstrip('/') _attr_to_conf = { 'url': ('section1', 'url'), 'api_key': ('section1', 'api_key'), 'with_default': ('section2', 'with_default'), 'without_default': ('section2', 'without_default'), } _defaults = { 'section1': { 'url': 'http://example.org', }, 'section2' : { 'with_default' : 'default-value', }, } _mainconf = ConfigParser() _mainconf.read_dict(_defaults) _mainconf.read('./example.config') class MainConf: def __getattr__(self, attr): section, item = _attr_to_conf[attr] try: ret = _mainconf[section][item] except KeyError: raise AttributeError(f"Can't get attribute '{attr}' on 'MainConf' because the config file does not contain '{item}' in the section '{section}'") from None if attr == 'url': ret = _normalize_url(ret) return ret mainconf = MainConf() 

Usage

# Dummy server request function request = print from method3 import mainconf request(mainconf.url+'/v1/endpoint', { 'api-key': mainconf.api_key, 'data': mainconf.with_default, }) # This should throw a helpful error. mainconf.without_default 

Answer 1

Definitely go with Method 1:

• There is no custom code to support. ConfigParser is part of the standard Python library; any programmer can easily look up its documentation to see how it works, if they weren't already familiar with it.

• URL normalization has no business being part of the configuration file parser. Including such normalization would violate the Single Responsibility Principle as well as the Principle of Least Surprise. What next — a rule that automatically converts http: URLs to https:?

  In fact, I'm not convinced that URL normalization is what you want. If you want to construct URLs by appending a path, use urllib.parse.urljoin(). Both of these calls would produce the same result:

  from urllib.parse import urljoin urljoin('http://example.com/', '/v1/endpoint') urljoin('http://example.com', '/v1/endpoint')