Configuration file with list of key-value pairs in python

I have a python script that analyzes a set of error messages and checks for each message if it matches a certain pattern (regular expression) in order to group these messages. For example "file x does not exist" and "file y does not exist" would match "file .* does not exist" and be accounted as two occurrences of "file not found" category.

As the number of patterns and categories is growing, I'd like to put these couples "regular expression/display string" in a configuration file, basically a dictionary serialization of some sort.

I would like this file to be editable by hand, so I'm discarding any form of binary serialization, and also I'd rather not resort to xml serialization to avoid problems with characters to escape (& <> and so on...).

Do you have any idea of what could be a good way of accomplishing this?

Update: thanks to Daren Thomas and Federico Ramponi, but I cannot have an external python file with possibly arbitrary code.


Asked by: Lydia491 | Posted: 01-10-2021






Answer 1

I sometimes just write a python module (i.e. file) called config.py or something with following contents:

config = {
    'name': 'hello',
    'see?': 'world'
}

this can then be 'read' like so:

from config import config
config['name']
config['see?']

easy.

Answered by: David285 | Posted: 02-11-2021



Answer 2

You have two decent options:

  1. Python standard config file format using ConfigParser
  2. YAML using a library like PyYAML

The standard Python configuration files look like INI files with [sections] and key : value or key = value pairs. The advantages to this format are:

  • No third-party libraries necessary
  • Simple, familiar file format.

YAML is different in that it is designed to be a human friendly data serialization format rather than specifically designed for configuration. It is very readable and gives you a couple different ways to represent the same data. For your problem, you could create a YAML file that looks like this:

file .* does not exist : file not found
user .* not found : authorization error

Or like this:

{ file .* does not exist: file not found,
  user .* not found: authorization error }

Using PyYAML couldn't be simpler:

import yaml

errors = yaml.load(open('my.yaml'))

At this point errors is a Python dictionary with the expected format. YAML is capable of representing more than dictionaries: if you prefer a list of pairs, use this format:

-
  - file .* does not exist 
  - file not found
-
  - user .* not found
  - authorization error

Or

[ [file .* does not exist, file not found],
  [user .* not found, authorization error]]

Which will produce a list of lists when yaml.load is called.

One advantage of YAML is that you could use it to export your existing, hard-coded data out to a file to create the initial version, rather than cut/paste plus a bunch of find/replace to get the data into the right format.

The YAML format will take a little more time to get familiar with, but using PyYAML is even simpler than using ConfigParser with the advantage is that you have more options regarding how your data is represented using YAML.

Either one sounds like it will fit your current needs, ConfigParser will be easier to start with while YAML gives you more flexibilty in the future, if your needs expand.

Best of luck!

Answered by: Dainton317 | Posted: 02-11-2021



Answer 3

I've heard that ConfigObj is easier to work with than ConfigParser. It is used by a lot of big projects, IPython, Trac, Turbogears, etc...

From their introduction:

ConfigObj is a simple but powerful config file reader and writer: an ini file round tripper. Its main feature is that it is very easy to use, with a straightforward programmer's interface and a simple syntax for config files. It has lots of other features though :

  • Nested sections (subsections), to any level
  • List values
  • Multiple line values
  • String interpolation (substitution)
  • Integrated with a powerful validation system
    • including automatic type checking/conversion
    • repeated sections
    • and allowing default values
  • When writing out config files, ConfigObj preserves all comments and the order of members and sections
  • Many useful methods and options for working with configuration files (like the 'reload' method)
  • Full Unicode support

Answered by: Patrick934 | Posted: 02-11-2021



Answer 4

I think you want the ConfigParser module in the standard library. It reads and writes INI style files. The examples and documentation in the standard documentation I've linked to are very comprehensive.

Answered by: Maddie262 | Posted: 02-11-2021



Answer 5

If you are the only one that has access to the configuration file, you can use a simple, low-level solution. Keep the "dictionary" in a text file as a list of tuples (regexp, message) exactly as if it was a python expression:

[
("file .* does not exist", "file not found"),
("user .* not authorized", "authorization error")
]
In your code, load it, then eval it, and compile the regexps in the result:
f = open("messages.py")
messages = eval(f.read()) # caution: you must be sure of what's in that file
f.close()
messages = [(re.compile(r), m) for (r,m) in messages]
and you end up with a list of tuples (compiled_regexp, message).

Answered by: Chelsea806 | Posted: 02-11-2021



Answer 6

I typically do as Daren suggested, just make your config file a Python script:

patterns = {
    'file .* does not exist': 'file not found',
    'user .* not found': 'authorization error',
}

Then you can use it as:

import config

for pattern in config.patterns:
    if re.search(pattern, log_message):
        print config.patterns[pattern]

This is what Django does with their settings file, by the way.

Answered by: Edward171 | Posted: 02-11-2021



Similar questions

python - django application configuration

I'm dying to get started Django but I'm really struggling with the initial setup. I have Python/MySql/Apache2.2/mod_python installed. Now I'm trying to create a proper directory structure and then update Django and Apache settings.py/httpd docs respectively. Specifically the location tag in the latter. Django and Python are based on simplicity but this is one huge oversight from the Django folks to not provide more guidanc...


python - Py2Exe - "The application configuration is incorrect."

I've compiled my Python program using Py2Exe, and on the client's computer we've satisfied all the dependencies using dependency walker, but we still get "The application configuration is incorrect. Reinstalling the application may correct the problem." I'm also using wxPython. The client does not have administrator access. Any ideas?


python - What is the best approach to implement configuration app with Django?

I need to program kind of configuration registry for Django-based application. Requirements: Most likely param_name : param_value structure Editable via admin interface Has to work with syncdb. How to deal with a situation in which other apps depend on configuration model and the model itself has not been initialized yet in DB? Let's say I would like to hav...


django - Is it ever polite to put code in a python configuration file?

One of my favorite features about python is that you can write configuration files in python that are very simple to read and understand. If you put a few boundaries on yourself, you can be pretty confident that non-pythonistas will know exactly what you mean and will be perfectly capable of reconfiguring your program. My question is, what exactly are those boundaries? My own personal heuristic was ...


python - SQLAlchemy - Mapper configuration and declarative base

I am writing a multimedia archive database backend and I want to use joined table inheritance. I am using Python with SQLAlchemy with the declarative extension. The table holding the media record is as follows: _Base = declarative_base() class Record(_Base): __tablename__ = 'records' item_id = Column(String(M_ITEM_ID), ForeignKey('items.id')) storage_id = Column(String(M_STORAGE_ID), ForeignKe...


oop - reading a configuration information only once in Python

I'm using the ConfigParser to read the configuration information stored in a file. I'm able to read the content and use it across other modules in the project. I'm not sure if the configuration file is read every time I call config.get(parameters). How can I make sure that the configuration information is read only once and rest of the time its read from the cache.


command line arguments - Python - configuration options, how to input/handle?

When your application takes a few (~ 5) configuration parameters, and the application is going to be used by non-technology users (i.e. KISS), how do you usually handle reading configuration options, and then passing around the parameters between objects/functions (multiple modules)? Options examples: input and output directories/file names...


configuration - Where to store secret keys and password in Python

I have a small Python program, which uses a Google Maps API secret key. I'm getting ready to check-in my code, and I don't want to include the secret key in SVN. In the canonical PHP app you put secret keys, database passwords, and other app specific config in LocalSettings.php. Is there a similar file/location which Python programmers expect to find and modify?


What is a good configuration file library for c thats not xml (preferably has python bindings)?

Closed. This question does not meet Stack Overflow guid...


python - How to setup mod_python configuration variables?

I'm running a Python server with mod_python, and I've run into some issues with configuration variables. This is actually two questions rolled into one, because I think they are highly related: I need a way to configure variables that will be available in Python while running. I currently just have a module that sets some name-value pairs that I import into other modules, but I was reading up on






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top