Python distutils and replacing strings in code

I often find a need to put paths in my code in order to find data or in some cases tool-specific modules. I've so far always used autotools because of this--it's just so easy to call sed to replace a few strings at build time. However, I'd like to find a more Pythonic way of doing this, i.e. use distutils or some other blessed way of building/installing. I've never managed to find anything relating to this in distutils documentation though so how do other people solve this problem?

Asked by: Miranda772 | Posted: 28-01-2022

Answer 1

For modules paths, a common practice is putting them in .pth files, as documented here. The site module provides a space for Site-specific configuration hooks, you can use it to tailor your environment.

Answered by: Vanessa658 | Posted: 01-03-2022

Answer 2

Well, with distutils (in the standard library) you have "package data". This is data that lives inside the package itself. Explained here how to do it. This is clearly not ideal, as you will have to use some kind of __file__ hacks to look up the location of the data at runtime.

So then comes setuptools (not in the standard library), which additionally has ways of looking up the location of that data at runtime. Explained here how to do it. But again that has it's own set of problems, for example, it may have trouble finding the data files on an uninstalled raw package.

There are also additional third party tools. The one I have used is kiwi.environ. It offers data directories, and runtime lookup, but I wouldn't recommend it for general use, as it is geared towards PyGTK development and Glade file location.

I would imagine there are other third party tools around, and others will elaborate.

Answered by: Jack554 | Posted: 01-03-2022

Answer 3

Currently, the best way to bundle data with code is going the setuptools way and use pkg_resources:

from pkg_resources import resource_filename, resource_stream
stream = resource_stream("PACKAGE", "path/to/data_f.ile")

This has the advantage of also working with Python eggs. It has the (IMHO) disadvantage that you need to put your data files into your code directory, which is accepted practice (one of the very, very few practices I disagree with).

As for Linux distros, I can (reasonably) assure you that your program will run without any problems (and patches) on any modern Debian-derived system if you use pkg_resources. I don't know about Fedora/openSUSE, but I would assume that it works as well.

It works on Windows, but it does currently not work with py2exe - there are simple workarounds for that, however.

Answered by: Daniel949 | Posted: 01-03-2022

Answer 4

The OP here, I've not finally managed to log in using my OpenID.


Point well taken, but for some Linux distros it seems to be standard to install application-specific data and application-specific modules in specific locations. I think that making these locations configurable at build/install time is a nice thing to do for people packaging my application. AFAICS “the pythonic way” in this case would force these packagers to apply patches to my code.

I'm also in the habit of writing applications where the executable part is a tiny wrapper around a main function in an application-specific module. To me it doesn't seem right to stick this application-specific module in /usr/lib/python2.5/site-packages.

Answered by: Adelaide130 | Posted: 01-03-2022

Answer 5

"I often find a need to put paths in my code" -- this isn't very Pythonic to begin with.

Ideally, your code lives in some place like site-packages and that's the end of that.

Often, we have an installed "application" that uses a fairly fixed set of directories for working files. In linux, we get this information from environment variables and configuration files owned by the specific userid that's running the application.

I don't think that you should be putting paths in your code. I think there's a better way.

[I just wrote our app installation tool, which does create all the config files for a fairly complex app. I used the Mako templates tool to generate all four files from templates.]

Answered by: Jack349 | Posted: 01-03-2022

Similar questions

mingw - Python Distutils

I was unable to install cython due to strict version numbering class of Distutils. For example binutils-2.18.50-20080109-2.tar.gz cannot be used along with

Python distutils, how to get a compiler that is going to be used?

For example, I may use python build --compiler=msvc or python build --compiler=mingw32 or just python build, in which case the default compiler (say, bcpp) will be used. How can I get the compiler name inside my (e. g. msvc, mingw32 and bcpp, respectively)? UPD.: I don't need the default co...

python - Getting distutils to install prebuilt compiled libraries?

I manage an open source project (Remix, the source is available there) written in python. We ask users to run python install to install the software. Recently we added a compiled C++ package (a port of SoundTouch -- go to trunk/externals in the source to see it.) We'd like the file that installs the base Remix libraries to also instal...

python - How to extend distutils with a simple post install script?

I need to run a simple script after the modules and programs have been installed. I'm having a little trouble finding straight-forward documentation on how to do this. It looks like I need to inherit from distutils.command.install, override some methods and add this object to the setup script. The specifics are a bit hazy though and it seems like a lot of effort for such a simple hook. Does anyone know an easy way to do th...

python - distutils and %post %postun

I am newbie. I am buidling rpm package for my own app and decided to use distutils to do achieve it. I managed to create some substitue of %post by using advice from this website, which i really am thankfull for, but i am having problems with %postun. Let me describe what i have done. In i run command that creates symbolic link which is needed to run application. It works good but problem is when i want to remove ...

Reasons to use distutils when packaging C/Python project

I have an open source project containing both Python and C code. I'm wondering that is there any use for distutils for me, because I'm planning to do a ubuntu/debian package. The C code is not something that I could or want to use as Python extension. C and Python programs communicate with TCP/IP through localhost. So the bottom line here is that while I'm learning packaging, does the usage of distutils specific fi...

file - Python distutils - copy_tree with filter

I want to copy a data directory into my distribution dir. copy_tree does this just fine. However, the project is also an svn repository, and I don't want the distribution to have all the .svn files that the data dir has. Is there any easy way to do a copy_tree excluding the .svn files, or should I just write my own recursive dir copy? I feel someone must have had this issue before.

distutils - How to pass flag to gcc in Python script?

I'm writing a Python extension in C that requires the CoreFoundation framework (among other things). This compiles fine with: gcc -o foo foo.c -framework CoreFoundation -framework Python ("-framework" is an Apple-only gcc extension, but that's okay because I'm using their specific framework anyway) How do I tell to pass this flag to gcc? I tried this, but it doesn'...

python - Custom distutils commands

I have a library called "example" that I'm installing into my global site-packages directory. However, I'd like to be able to install two versions, one for production and one for testing (I have a web application and other things that are versioned this way). Is there a way to specify, say "python stage" that will not only install a different egg into site-packages, but also rename the module from "exampl...

python - How to get the arch string that distutils uses for builds?

When I build a c extension using python build, the result is created under a directory named build/lib.linux-x86_64-2.6/ where the part after lib. changes by the OS, CPU and Python version. Is there a way I can access the appropriate string for my current architecture from python? Hopefully in a way that is guaranteed to match what distutils is cr...

Still can't find your answer? Check out these communities...

PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python