Python Decorators

Recently I have been spending quite a bit of time learning/using Python for an API I'm implementing for a project at work. In order to create the API I have been using the Flask framework.

Whilst doing the development of this API I came across a need to better understand what decorators in Python do and how they work, for reasons I will explain shortly.

This blog post is not intended to recommend (or otherwise) either Python or Flask for an API project, but just to show a few things I have learned about how decorators operate.

Traditional web servers used to serve up content based on raw files on disk corresponding to the file paths given in the URL provided by the web browser. For example, if you pointed your browser to http://example.com/demo/test.html and the server had a document root of /var/www/html, a traditional web server would try to find the file at /var/www/html/demo/test.html and return it to the browser. Possibly doing some server-side parsing of the file for dynamic content.

In Flask, the content can be handled quite differently as it uses the concept of routes, which don't have any requirement to map to a physical file. Both the URL path and the mechanisms to generate/load the content required are all defined in code. You can have statically served content with Flask as well, but for the purposes of my API this isn't required.

So, what does this have to do with decorators?

Well, within Flask, the way in which you define the routes which the web server will respond to is by decorating the function which generates the content with a decorator provided by the Flask app. This way, Flask knows which function to call when the URL is hit.

This is all pretty cool and really quick and simple to get up and running. For example, a very simple Hello, world program looks like:

from flask import Flask

app = Flask(__name__)

@app.route("/api/helloworld")
def hello_world():
    return "Hello, World!"

app.run()

Running this using python3 app.py spins up a web server listening (by default) on port 5000 bound to the localhost/127.0.0.1 interface of the computer it's running on. You get some warnings that running the server this way isn't recommended and should only be used for development purposes. For what we're looking at here this is perfectly fine. If you do decide to run one in production you should really look at one of the WSGI frameworks which are out there (there are many to choose from).

The line @app.route("/api/helloworld") is the decorator in this instance and tells Flask that anytime a request comes through for http://localhost:5000/api/helloworld it should call this function and send back the returned data.

So, now that I've used the decorator and can easily create API endpoints, you may ask why I felt the need to delve deeper into the internals of what decorators do and how they operate. There are two main reasons I did this:

1. Curiosity
2. Testing

In terms of the curiosity, I like to understand as much as I can about how the frameworks/tools I am using do what they do. This is generally helpful both in terms of when things don't work they way I expect them to and trying to make the most out of the framework at hand.

With regards to testing, the trivial example presented above has very little which can go wrong. Some of the API endpoints I've needed to create, however, have significant amounts of logic involved and additional requests made by the code to external systems (mostly database queries and launching local applications on the OS). In order to ensure that the code functions as expected and continues to function as expected when I make changes to the codebase it's a good idea to ensure as much of the code as possible is covered by unit tests. That way if something stops functioning the way it used to you can discover it at code-creation time rather than run-time.

As the code I was writing grew and the number of endpoint my API had expanded, I started splitting the code into different files. I had a main app file which created the Flask app object and then handed it off to functions in other files to add their own routes to it. Whether this was the "correct" way to handle this or not I don't know, but it does what it needs to do for the moment, which is what is important to me. Going forward I may discover a new way to handle this and refactor the code. If I am making sweeping changes to the code as part of a refactor, the unit tests I have created will be invaluable to ensure that nothing I've changed breaks functionality.

The following files demonstrate how splitting the routes into different files works:

from flask import Flask
from routes import AddRoutes

app = Flask(__name__)
app.config["DEBUG"] = True

AddRoutes(app)

app.run()

def AddRoutes(app):
    @app.route("/api/helloworld")
    def hello_world():
        return "Hello, World!"

Again, running python3 app.py spins up a web server as before and the content served is the same as it was. You may have noticed the addition of the line app.config["DEBUG"] = True into the code. This tells Flask that we're running in Debug mode which give us a few nice features:

1. Automatic restart of the server if it detects code has changed
2. Browser based debugging if exceptions occur

Obviously don't run a production server with Debug enabled.

Although the example so far doesn't do much, it should still be possible to run unit tests on it to ensure that the returned values are as they are expected to be.

For this will will spin up a simple test_routes.py script:

from flask import Flask
from routes import AddRoutes

def test_helloworld():
    app = Flask("TESTING")
    AddRoutes(app)
    assert app.view_functions["hello_world"]() == "Hello, World!"

This can then be run with pytest (other testing frameworks are available), which shows that the test ran successfully and will show you how much of each file has been covered by the test:

python3 -m coverage report -m
Name             Stmts   Miss  Cover   Missing
----------------------------------------------
routes.py            4      0   100%
test_routes.py       6      0   100%
----------------------------------------------
TOTAL               10      0   100%

This unit test, unfortunately, has a dependency on Flask and involves it spinning up an instance of the Flask app class in order for the test to run. It also requires retrieving the function from within the view_functions object within the instantiated app. This probably causes the test to run slower than expected and also means that we're testing more than just our own code as a third party library is getting involved

In terms of timings, this test only takes 0.16 seconds to run but if we had thousands of tests to cover a significant proportion of the possible paths through the code, this could quickly add up.

So... What can we do instead?

Well, given that the AddRoutes function takes a parameter which defines what the object to add the routes to is, couldn't we just create our own class to use instead of relying on Flask?

Of course we could. In unit testing parlance these stripped down classes are referred to as Test Doubles and come in a variety of flavours. Such as Mocks, Fakes, Stubs, Spies and Dummies, as explained here.

Given that the AddRoutes method only relies on the route method in the Flask app, this is probably the only one we need to implement in order to get the test working without relying on Flask. This is used as a decorator in AddRoutes hence our interest today in decorators.

A quick lesson on decorators

So, now that we have established what we need to learn about decorators for, let's take a look at what they are and how we would implement one.

At a basic level, a decorator adds additional code around the function it is decorating. This could be in order to intercept and change the parameters with which the method is called, prevent the method being called, running the method multiple times, or doing additional work prior to or after the call is made, as a few examples.

It does this by defining and returning a function which is used in place of the function it is wrapping. The decorator takes an argument which represents the wrapped function, and the function within it takes the arguments which can be passed to the original function when it is called. The following code demonstrates this in action:

def decorator(f):
    def wrapper(*args, **kwargs):
        print("Called before function is run")
        f(*args, **kwargs)
        print("Called after function is run")
    return wrapper

def unwrapped_func(name: str):
    print(name)

@decorator
def wrapped_func(name: str):
    print(name)

print("Start unwrapped_func")
unwrapped_func("unwrapped")
print("End unwrapped_func")

print("Start wrapped_func")
wrapped_func("wrapped")
print("End wrapped_func")

At the top of the file you can see the implementation of a decorator. f represents the to-be-wrapped function. The function wrapper which is defined inside this function is what does the work here and takes args and kwargs parameters so that it can pass these along to the original function. The decorator function ends by returning the newly defined function.

Following on from that we define two more methods to demonstrate the functionality, unwrapped_func and wrapped_func which both provide the same functionality. It then proceeds to use these along with some debugging print calls to show the timings on what is happening:

python3 decorator1.py 
Start unwrapped_func
unwrapped
End unwrapped_func
Start wrapped_func
Called before function is run
wrapped
Called after function is run
End wrapped_func

unwrapped_func executes entirely as expected given its definition whereas wrapped_func produces the additional output as given by the wrapper function.

Pretty cool, huh?

In this example, however, we are only using a bare decorator. That is to say one which doesn't take any parameters. The route decorator from Flask, on the other hand, does take parameters, such as the URL path under which it is accessed, what HTTP methods are acceptable to use to call it, and so on.

Decorators with parameters are handled in a different way as there is a requirement to handle the parameters before it then handles the function. For this, the decorator returns a function to handle the parameters which then returns a function to handle the decoration logic. This is not necessarily an easy concept to get your head around and so I think an example may be helpful here:

def decorator(dec_param):
    def handler(f):
        def wrapper(*args, **kwargs):
            orig_arg = args[0]
            new_arg = "{} {}".format(orig_arg, dec_param)
            new_args = (new_arg,)
            f(*new_args, **kwargs)
        return wrapper
    return handler

@decorator("this")
def wrapped_func1(name: str):
    print(name)

@decorator("that")
def wrapped_func2(name: str):
    print(name)

wrapped_func1("test")
wrapped_func2("test")

Again, we start by declaring our decorator. This then defines and returns a function to handle the function to be wrapped, which in turn defines and returns the logic to be run when the function is called.

The wrapper function modifies the parameter provided to the call to use the parameter which the decorator was called with, resulting in the following output:

$ python3 decorator2.py 
test this
test that

At this point we're now armed with enough that we can proceed with our unit tests

Eliminating the Flask dependency

Now that we know how to implement a decorator which should be able to work in place of the Flask app's decorator methods, all that is left to do is create a class implementing the required functionality and use that in our test:

from routes import AddRoutes

class flask_mock:
    def __init__(self):
        self.known_routes = {}

    def route(self, route):
        def handler(f):
            self.known_routes[route] = f
            def wrapper(*args, **kwargs):
                return f(*args, **kwargs)
            return wrapper
        return handler

def test_helloworld():
    app = flask_mock()
    AddRoutes(app)
    assert app.known_routes["/api/helloworld"]() == "Hello, World!"

As we can see, we declare a class for our mocked out app implementation and include the route function with corresponding handler and wrapper methods. We have also taken the import for Flask out of the file.

The route function stores the reference to the hello_world function in a known_routes dictionary to allow this to be referenced and executed in the test code.

In reality, we don't even need to define and return the wrapper function as the method in question is never directly called (both in this test file and in the normal functioning of a Flask app), and so the code could be shortened as follows:

from routes import AddRoutes

class flask_mock:
    def __init__(self):
        self.known_routes = {}

    def route(self, route):
        def handler(f):
            self.known_routes[route] = f
        return handler

def test_helloworld():
    app = flask_mock()
    AddRoutes(app)
    assert app.known_routes["/api/helloworld"]() == "Hello, World!"

This test now functions in 0.01s which is a significant improvement.

Wrapping up

And so we conclude our little foray into the world of function decorators in Python. I'm sure there is plenty more which could be learned about them as I've only scratched the surface as far as needed for the use-case at hand.

I can see quite a few instances where being able to mock the Flask app is going to be useful for me going forwards, including having the ability to verify that the URL parameters in the route match with the parameters in the declared function.

Hopefully you all enjoyed coming on this little journey with me. Hopefully I'll be posting more soon.