In the past week or so I finished implementing the core functionality of my project, importing modules using isolated state. Loaders and importers now accept an optional engine parameter. If it's not supplied, they fall back on a GlobalImportEngine instance which uses global state.

I also discovered that some loaders (namely for builtin and extension modules) call functions from the implementation of import in C (imp.*) and those are hardcoded to inject modules into sys.modules. This is a problem, but thanks to limitations imposed on these kind of modules (one copy of the module per process), it should be fine to place them into modules dictionary in an ImportEngine instance after they are imported by the imp module.

Finally, I realised that the test structure is more complex because the global import state needs to be preserved. So there was a number of context managers and other tricks which I didn't change before but I think now I got it right.

As a side note, I had some personal trouble in the last few days but things should be back to normal now and I'll try to post more often here.

So after two weeks of preparations, I spent last week migrating the core importing functionality to ImportEngine.

Since there the hierarchy of classes in importlib involved in __import__ functionality has to be pretty deep to maximise code reuse, the changes are spread over quite a few places:

• @module_for_loader
• BuiltinImporter.find_module
• BuiltinImporter.load_module
• FrozenImporter.find_module
• FrozenImporter.load_module
• _LoaderBasics._load_module
• _SourcelessFileLoader.load_module
• _ExtensionFileLoader.load_module
• PathFinder._path_hooks
• PathFinder._path_importer_cache
• PathFinder.find_module
• _FileFinder.find_module
• _DefaultPathFinder._path_hooks

In short, the engine argument has to be passed from the __import__ function down to the loaders. This is particularly tricky in case of path hooks (which are in fact implemented as a meta hook). All this results in substantial changes in the code and there are still a few places I need to iron out, and afterwards it will need solid testing.

Testing, testing, 1-2-3

One problem with using a custom import function is handling recursive imports (i. e. imports occurring in the module that you are importing). In order to handle this, I implemented a replacement __import__, which can be substituted for the builtin one (commit: https://bitbucket.org/jergosh/gsoc_import_engine/changeset/b189df886193).

I also spent a large part of last week figuring out how to make unittests take advantage of the ImportEngine. Resulting commit: https://bitbucket.org/jergosh/gsoc_import_engine/changeset/b39d1c0c3e53

Next step: new style finders and loaders.

First week and first commits!

With help from Nick and Brett, I implemented an initial barebones version of the import engine. The current implementation still uses the global import state, following Nick's advice to get the engine logic first.

Relevant commits:

https://bitbucket.org/jergosh/gsoc_import_engine/changeset/ab597598e4b7

https://bitbucket.org/jergosh/gsoc_import_engine/changeset/b55679f6b3c2

Next step: unittests.