Auto-loading modules in Python

Since switching to vcsh I’ve been writing more personal scripts since they’re pretty easy to ship around to each machine. Plus more things have REST APIs and python’s slumber makes it dead easy to talk to them.

However I then have to make sure modules like slumber are installed since it’s not in the python standard library. This adds a level of awkwardness to the scripts in my ~/bin.

While looking for something else I came across this answer on stackoverflow and it fit what I wanted to do. Unfortunately when I tried it on python3 it didn’t work. The second import slumber failed with the same ImportError even though the pip.main line had succeeded. Looking around some more I found module reloading comes up a lot for Iron Python stuff. And the consensus there was that the correct thing to do was to restart the interpreter.

The lightbulb went off. Restarting the interpreter is simple to do with exec on unix-type systems.

So in the end I went with this:

1
2
3
4
5
6
7

import os.path
try:
  import slumber
except ImportError:
  import pip
  pip.main(['install', '--user', 'slumber'])
  os.execv(__file__, sys.argv)

This installs the missing python module and then reruns the script. There are two obvious additions here.

First, an environment variable that will say this has been tried. If so, it will abort instead of run os.execv under the theory that pip.main is failing but not firing an exception.

Second would be to record the missing modules, and then if the list of missing modules isn’t empty, install the missing ones and then do the os.execv (with the env check).

For now this works good enough. And this particular utility is one I’m likely to run from a few places so it’s nice that it will just work. One big caveat is that if you want to parse the output of scripts using this technique, the pip output will likely mess things up. Plus pip is rather notorious for breaking in various ways so keep an eye out on what the install noises actually say.