There are two questions related to Python's module system that come up repeatedly on #python, and don't have obviously good answers.
How do I reload a module?
How do I create a plugin system?
What is Exocet?
Exocet is a new way to load Python modules. It separates the act of naming a dependency from the act of creating a module object. As a result, more than one instance of a module can be created from the same source file. Also, when creating a module object, precise control can be exerted over what it's allowed to import.
Let's start with a code example.
>>> import exocet, httplib
>>> urllib = exocet.loadNamed("urllib", exocet.pep302Mapper)
>>> def HTTP_Ex(host):
... print "making HTTP connection to", host
... return httplib.HTTP(host)
...
>>> class _ModuleProxy(object):
... def __getattribute__(self, name):
... if name == 'HTTP':
... return HTTP_Ex
... else:
... return getattr(httplib, name)
>>> httplib_plus = _ModuleProxy()
>>> overriddenMap = exocet.pep302Mapper.withOverrides({"httplib": httplib_plus})
>>> urllib_plus = exocet.loadNamed("urllib", overriddenMap)
>>> print urllib.urlopen("http://python.org").read(121)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
>>> print urllib_plus.urlopen("http://python.org").read(121)
making HTTP connection to python.org
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Quick breakdown of what's going on here: loadNamed takes a module name and a mapper object, and returns a new module object. Note that this is very different from what import does; each invocation of loadNamed returns a new module object, unrelated to any previous ones.
The mapper object is responsible for intercepting all import calls made by the module being loaded. The pep302Mapper object used here just invokes Python's normal importing behaviour, caching loaded modules in sys.modules. But its withOverrides method creates a new mapper, one that looks in a dict first. The urllib_plus module is almost like the urllib module in this example, with one difference: the latter got the real httplib module when it executed the statement import httplib; urllib_plus got the httplib_plus module when it ran that statement. The other mapper that Exocet includes by default is emptyMapper, in which all import statements will fail. You can construct your own, providing any object under any name to be available to import statements in modules you load.
So after constructing these two modules, they're usable as normal. The only difference is that urllib_plus invokes our wrapper function when it calls httplib.HTTP(), producing the printed line before proceeding with its work.
So, how does this behaviour answer the questions I mentioned earlier?
Module reloading
As we saw last time, reload isn't suitable for real-world use; it changes things around too much in some places, not enough in others. With Exocet, you don't have to reload modules because you can just load them. When you want a new module with updated code, you just load it and have a new module object. All the old objects still exist as long as you want them to; there's no orphaned-instance problem. If you need new versions of the module's dependencies you can load them first and put them in the mapper so that they're visible to your new module. Old instances, classes, and modules can be removed in the normal way — by Python's garbage collection, when nothing uses them any longer.
Plugin systems
The normal use of Python packages and modules involves importing specific ones by name. However, sometimes you want to load a bunch of modules and call something in each. Since Exocet borrows code from twisted.python.modules, it's possible to iterate over a package and load each module in it:
>>> import exocet
>>> [exocet.load(x, exocet.pep302Mapper) for x in
... exocet.getModule("twisted.plugins").iterModules()]
[<module 'twisted.plugins.cred_anonymous' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/cred_anonymous.py'>,
<module 'twisted.plugins.cred_file' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/cred_file.py'>,
<module 'twisted.plugins.cred_memory' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/cred_memory.py'>,
<module 'twisted.plugins.cred_unix' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/cred_unix.py'>,
<module 'twisted.plugins.twisted_conch' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_conch.py'>,
<module 'twisted.plugins.twisted_ftp' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_ftp.py'>,
<module 'twisted.plugins.twisted_inet' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_inet.py'>,
<module 'twisted.plugins.twisted_lore' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_lore.py'>,
<module 'twisted.plugins.twisted_mail' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_mail.py'>,
<module 'twisted.plugins.twisted_manhole' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_manhole.py'>,
<module 'twisted.plugins.twisted_names' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_names.py'>,
<module 'twisted.plugins.twisted_news' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_news.py'>,
<module 'twisted.plugins.twisted_portforward' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_portforward.py'>,
<module 'twisted.plugins.twisted_qtstub' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_qtstub.py'>,
<module 'twisted.plugins.twisted_reactors' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_reactors.py'>,
<module 'twisted.plugins.twisted_runner' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_runner.py'>,
<module 'twisted.plugins.twisted_socks' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_socks.py'>,
<module 'twisted.plugins.twisted_telnet' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_telnet.py'>,
<module 'twisted.plugins.twisted_trial' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_trial.py'>,
<module 'twisted.plugins.twisted_web' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_web.py'>,
<module 'twisted.plugins.twisted_web2' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_web2.py'>,
<module 'twisted.plugins.twisted_words' from '/home/washort/Projects/Twisted/trunk/twisted/plugins/twisted_words.py'>]
In this case we used the default pep302Mapper since we wanted plugin modules to be able to import anything they wanted. However, a different mapper could be provided which provided different modules, limited them to only importing certain ones, or none at all.
Unit testing
Hardly anybody asks about this on #python, but they should. By loading the modules you test with Exocet, you can replace their dependencies with stubs without monkeypatching. Just provide overrides to the mapper object when loading the module, and you have a stubbed or mocked module to test without altering the behaviour of other tests.
Next steps
Exocet lets you load most existing Python modules, but in a way that gives much better control over what happens. This makes room for radically different ways of thinking about how to organize Python programs. Since this approach is so different, practice is needed to come up with new idioms for how to use it effectively. I've tried carefully to provide mechanism here with very little policy. There are a few kinks to still work out in how Exocet works in more complicated scenarios, but give it a try! It might be just the thing for your next IRC bot. ;-)
If you want to help with Exocet development, it happens here, on Launchpad. I look forward to your feedback.
4 comments:
The whole idea is interesting! Some doubts:
1) Why did you choose to use zope.interface? It seems overkill to me for what you're doing here, and many people wouldn't like to depend on it without a good reason.
2) What is "modules" ? It seems something like twisted.python.modules, was it pulled into its own distribution?
3) It would be really nice to have pkg_resources / egg integration; this way you could have multiple versions of a module installed, and
you could use *both* at the same time, achieving an effect similar to what OSGi does in Java. That would solve some problems with the standard library as well - you require a version, and exocet just pulls the best available version, whether it's in the standard lib or has been manually installed.
Is there any mailing list for discussing Exocet development around?
That's hilarious, as soon as I saw the plugin comment I thought to myself "That would perfectly resolve some of the issues I have with my IRC bot!"
Alan:
1) Exocet incorporates some code that already depended on zope.interface and I didn't bother to change it. z.i is a small enough and useful enough library that I don't feel bad about depending on it for everything. In my opinion, it belongs in the standard library.
2) Yes. twisted.python.modules was converted to a standalone project here: http://launchpad.net/modules
Exocet includes code from it.
3) I agree that multiple-version support would be pretty great. I am not so sure that setuptools/pkg_resources/egg are a good way to do it. If someone wants to contribute code for supporting that, I'd be interested.
I don't currently have a mailing list; I may set one up soon.
It looks like questions and bug reporting is disabled on the Exocet Launchpad project, or I just don't know how to use Launchpad, which is entirely possible.
I'm writing a MUD server that aims to be extremely reloadable. Exocet is working beautifully for the most part, but I'm having the worse time figuring out how to go two-deep with the code reloading.
For example, I use a ParentLoader class to load "parents" for things in-game (Rooms, Exits, etc), and this is done via Exocet here: https://github.com/gtaylor/dott/blob/master/src/server/parent_loader/loader.py#L30
This is an example of one of the parents:
https://github.com/gtaylor/dott/blob/master/src/game/parents/base_objects/player.py
You'll see that it is a child of BaseObject. I can reload that RoomObject class just fine right now, but I can't figure out how to reload BaseObject.
Attempting to stick an exocet import within that RoomObject module results in an error that looks like twisted getting freaked out, but isn't entirely helpful.
Perhaps I've just structured this wrong, but I'd love any pointers to doing this the right way.
Post a Comment