Skip to content

The magic of template tags

Published on: December 4, 2007    Categories: Django

Over the last couple days I’ve spent some time discussing the word “magic” and exploring just what it really means, with an emphasis on the fact that a lot of “magic” in programming — though initially counterintuitive and not at all what you’d expect to have happen (and it’s precisely this reason which usualy makes “magic” a bad idea) — boils down to applications of fairly simple principles. As a real-world demonstration of that, yesterday we saw how to build a Python module object dynamically and make it work with import even though it didn’t exist anywhere on the file system.

When I mentioned Django’s “magic-removal” effort the other day, I mentioned that it didn’t quite purge all of the “magic” from Django:

there’s still a minor amount of magic in how custom template tags work, but it’s not nearly so bad and most people never even notice it (until they have a tag library which raises an ImportError and start wondering why Django thinks the tags should be living in django.templatetags, but that’s a story for another day).

Today is another day and we now have a little better understanding of just what “magic” can mean, so let’s take a look at the last little bit of real magic left in Django.

The magic

If you’ve ever tried to load a tag or filter library that didn’t exist, or wasn’t inside in application in your INSTALLED_APPS setting, or which had an error inside it which raised an ImportError, you’ve probably seen an error message like this: 

TemplateSyntaxError: 'no_library_here' is not a valid tag library: Could not load template library from django.templatetags.no_library_here

Which leads naturally to the question of why Django is looking in django.templatetags to find the tag library: shouldn’t it be looking inside the installed applications, hunting for templatetags modules there?

The trick

Unlike yesterday’s exercise, there’s no hacking of sys.modules going on here, and no magical generation of runtime module objects; when Django successfully loads a template tag library, it’s pulling it from the actual location on your file system where you put it. And, unlike the magic django.models namespace that models were placed into in Django 0.90 and 0.91, django.templatetags does actually exist; a quick look at the code shows the technique being used here (the following is, at the moment, the full content of django/templatetags/__init__.py: 

from django.conf import settings

for a in settings.INSTALLED_APPS:
    try:
        __path__.extend(__import__(a + '.templatetags', {}, {}, ['']).__path__)
    except ImportError:
        pass

This works because Python modules (or “packages” if you prefer that terminology, though it’s a bit loaded) let you assign to a specially-named variable — __path__ — which should be a list of modules. Each module listed in __path__ will, no matter where it’s actually located on your filesystem, be treated from then on as if it also exists as a sub-module of the module whose __path__ list it appears in.

So what this code is doing is simply looping through the INSTALLED_APPS setting and, for each application listed, trying to import a templatetags module from that application and add that module’s __path__ (meaning the list of custom tag libraries inside it) to the __path__ of django.templatetags. If the attempted import raises an ImportError (which usually indicates there’s no templatetags module in a given application), Django simply skips that one and moves on to the next.

The end result of this is that any custom tag libraries defined inside templatetags modules in your installed applications will — in addition to their normal locations — be importable from paths under django.templatetags.

Practical magic

In a lot of cases, apparently “magical” things really don’t serve any useful purpose and so can — and should — be removed in favor of more natural techniques. In this case, though, the “magical” extension of django.templatetags serves a very useful purpose: looping over INSTALLED_APPS and importing and initializing all the available tag libraries can be an expensive and time-consuming process. Doing it every time a template used the {% load %} tag would bring the performance of Django’s template system to a screeching halt, so we need to have some way of making it faster.

In this case, on option would be to maintain a cache of known tag libraries — maybe the same sort of thing Python does with sys.modules to keep track of which modules have already been imported and initialized — and that wouldn’t be such a bad idea. But extending the __path__ of django.templatetags works just as well, and makes for extremely compact loading code: you can simply take the name of the tag library the {% load %} tag asked for, concatenate it onto the string “django.templatetags” and try to import the result.

The __path__ method, then, gives the needed performance boost, and also has a useful side effect: since the list of all importable tag libraries lives in django.templatetags.__path__, it’s easy to loop through that list to find out what libraries are available (this is how the tag and filter documentation in the admin interface works, for example: it’s a simple for loop over django.templatetags.__path__.

Also, this “magic” doesn’t get in the way of normal Python imports: the module stays right where it was originally defined, and you can — if you need access to code within it — simply import it exactly as you’d expect, without having to go through the django.templatetags namespace.

Where it does cause problems

This does sometimes cause confusion, because there are cases where the unexpected “Could not load template library from django.templatetags” message can be a red herring that leads people down the wrong path when looking for an error. The most common case is trying to load a tag library from an application that’s not listed in INSTALLED_APPS, but there’s also a subtler issue. Since Django loops through INSTALLED_APPS trying to import templatetags modules, and treats ImportError as meaning that there is no templatetags module in a particular application, a tag library which — through bad coding or misconfiguration — exists but happens to raise an ImportError will be silently ignored.

This isn’t really a “bug” in Django, because the problem of handling situations like this — where, for example, you’re trying to import something to see if it exists, and an ImportError from another source gets in the way — is a long-standing issue for Python best practices.

Best practices for Django template tags

You generally won’t run into this problem unless you get into one of a few very specific situations, but it is useful to know that raising an ImportError from a custom tag library will cause it to “disappear”; this is often a bad thing, because custom tags are supposed to fail silently whenever possible (one design decision in the Django template system is that, in production, the types of template errors which can bring the site down should be kept to a minimum).

One easy way to accomplish this is demonstrated in the markdown filter in django.contrib.markup which, obviously, requires the Python Markdown module in order to function. This module isn’t in the Python standard library and isn’t bundled with Django, so there’s a very real chance that the markdown module will need to be separately installed before this filter can work properly. To detect and deal with a missing Markdown module, the markdown filter does the following: 

try:
    import markdown
except ImportError:
    if settings.DEBUG:
        raise template.TemplateSyntaxError, "Error in {% markdown %} filter: The Python markdown library isn't installed."
    return force_unicode(value)

This does several useful things:

  1. It makes sure the Markdown import happens inside the filter, rather than at the module level, which means an errant ImportError won’t make the whole library “disappear”.
  2. It wraps the import in a try/except block.
  3. In case of an ImportError from import Markdown, it suppresses the error in production, and falls back to a default of simply returning the input.
  4. When in development — i.e., when the DEBUG setting is True — it raises TemplateSyntaxError with a descriptive error message describing how to fix the problem.

The remainder of the code in the markdown filter can then safely assume that the Markdown module is available, and can act accordingly.

In general, combining one or more of these techniques will make your custom tag and filter libraries more robust and more useful in a variety of error situations, not just those where an ImportError can obscure a different underlying problem.

And that’s a wrap

I think I’m all talked out now on the subject of “magic”; hopefully at this point you’ve got a little better understanding of when and why things which appear to be magical can actually work on nothing more than very simple techniques, how it can be misleading sometimes to refer to things as “magic” when they might not be, and have a better understanding of how some specific instances of “magic” in Django (including all the ones we’ve removed) have been implemented.