Recently, I had the need to work with a plugin-based architecture, where the plugins needed to be shareable amongst different projects in an easy way. In order to do so, I decided to create a separate python module for each plugin, where the functionality of the plugin was implemented as a class within that module.

Originally, I had the idea that each project should just have one folder where all of these modules with plugins could be located in. However, after using an initial implementation for some time, I decided it would be better if each project would define a base folder under which all plugins would be located but that each plugin could be located in an arbitrary number of subdirectories under this main plugin directory.

Suppose we want to make a really basic system where each plugin will implement a method that applies a function over a given argument. The base Plugin class could then be as follows:

class Plugin(object):
    """Base class that each plugin must inherit from. within this class
    you must define the methods that all of your plugins must implement
    """

    def __init__(self):
        self.description = 'UNKNOWN'

    def perform_operation(self, argument):
        """The method that we expect all plugins to implement. This is the
        method that our framework will call
        """
        raise NotImplementedError

Writing a plugin that performs the identity function would require you to first set the description in the constructor and provide an implementation of the perform_operation method. An example implementation of this is as follows:

import plugin_collection 

class Identity(plugin_collection.Plugin):
    """This plugin is just the identity function: it returns the argument
    """
    def __init__(self):
        super().__init__()
        self.description = 'Identity function'

    def perform_operation(self, argument):
        """The actual implementation of the identity plugin is to just return the
        argument
        """
        return argument

Because we want to create a system where we dynamically load the different plugin modules, we will define a new class PluginCollection that will take care of the loading of all plugins and enables the functionality to apply the perform_operation of all plugins on a supplied value. The basic components for this PluginCollection class are as follows:

class PluginCollection(object):
    """Upon creation, this class will read the plugins package for modules
    that contain a class definition that is inheriting from the Plugin class
    """

    def __init__(self, plugin_package):
        """Constructor that initiates the reading of all available plugins
        when an instance of the PluginCollection object is created
        """
        self.plugin_package = plugin_package
        self.reload_plugins()


    def reload_plugins(self):
        """Reset the list of all plugins and initiate the walk over the main
        provided plugin package to load all available plugins
        """
        self.plugins = []
        self.seen_paths = []
        print()
        print(f'Looking for plugins under package {self.plugin_package}')
        self.walk_package(self.plugin_package)


    def apply_all_plugins_on_value(self, argument):
        """Apply all of the plugins on the argument supplied to this function
        """
        print()
        print(f'Applying all plugins on value {argument}:')
        for plugin in self.plugins:
            print(f'    Applying {plugin.description} on value {argument} yields value {plugin.perform_operation(argument)}')

The final component of this class is the walk_package method. The basic steps in this method are:

  1. Look for all modules in the supplied package
  2. For each module found, get all classes defined in the module and check if the class is a subclass of Plugin, but not the Plugin class itself. Each class that satisfies these criteria will be instantiated and added to a list. The advantage of this check is that you can still place some other python modules within the same directories and they will not influence the plugin framework.
  3. Check all sub-folders within the current package and recurse into these packages to recursively search for plugins.

Items 1 and 2 of the above steps can be implemented with the following code:

def walk_package(self, package):
    """Recursively walk the supplied package to retrieve all plugins
    """
    imported_package = __import__(package, fromlist=['blah'])

    for _, pluginname, ispkg in pkgutil.iter_modules(imported_package.__path__, imported_package.__name__ + '.'):
        if not ispkg:
            plugin_module = __import__(pluginname, fromlist=['blah'])
            clsmembers = inspect.getmembers(plugin_module, inspect.isclass)
            for (_, c) in clsmembers:
                # Only add classes that are a sub class of Plugin, but NOT Plugin itself
                if issubclass(c, Plugin) & (c is not Plugin):
                    print(f'    Found plugin class: {c.__module__}.{c.__name__}')
                    self.plugins.append(c())

The recursive part of the method is then implemented by calling the same walk_package method again on each subpackage found within the current package. We first build up the list of all current paths (either just the __path__ in case it is a string, or the elements if it is a _Namespace object. After that, we just initiate the recursive call for each of the elements. The complete implementation of the recursive part is then:

    # Now that we have looked at all the modules in the current package, start looking
    # recursively for additional modules in sub packages
    all_current_paths = []
    if isinstance(imported_package.__path__, str):
        all_current_paths.append(imported_package.__path__)
    else:
        all_current_paths.extend([x for x in imported_package.__path__])

    for pkg_path in all_current_paths:
        if pkg_path not in self.seen_paths:
            self.seen_paths.append(pkg_path)

            # Get all subdirectory of the current package path directory
            child_pkgs = [p for p in os.listdir(pkg_path) if os.path.isdir(os.path.join(pkg_path, p))]

            # For each subdirectory, apply the walk_package method recursively
            for child_pkg in child_pkgs:
                self.walk_package(package + '.' + child_pkg)

If we now implement two additional plugins DoublePositive (doubles the argument) and DoubleNegative (doubles and negates the argument) and place the modules for these two plugins in a subdirectory double under the plugins directory, our plugin collection should be able to handle these.

In order to test everything, we can write a very simple application:

from plugin_collection import PluginCollection

my_plugins = PluginCollection('plugins')
my_plugins.apply_all_plugins_on_value(5)

This application will first initialize a PluginCollection on the package plugins. When instantiated, this plugin collection will recursively look for all plugins defined under the folder plugins. After the initialization is done, it will call the perform_operation on each of the plugins with the value 5.

And behold, if you now run the application you will see the following output:

$ python main_application.py

Looking for plugins under package plugins
    Found plugin class: plugins.identity.Identity
    Found plugin class: plugins.double.double_negative.DoubleNegative
    Found plugin class: plugins.double.double_positive.DoublePositive

Applying all plugins on value 5:
    Applying Identity function on value 5 yields value 5
    Applying Negative double function on value 5 yields value -10
    Applying Double function on value 5 yields value 10

Without any direct reference to any of the plugin modules in your code, it automatically found all of the plugins..... MAGIC :)
magic

A complete working version of the above example code can be found in a separate Github repository.

Drop me a comment in case you think this is useful, or if you have tips, comments, or better approaches :)