Generic inlines and Django history

Published December 4, 2008. Filed under: Django, Misc.

The other day at work I stumbled across my first opportunity to use a relatively-new feature in the Django admin, one which turned what had looked like it would be a fairly nasty task into, basically, a five-minute job (plus staging, testing and deployment, of course, but that happens no matter how long it takes to develop the code). I’ll get to the specifics in a minute, but first I want to give a little bit of background on what, exactly, I was working on, since it’s sort of a fun trip back through the mists of Django’s history.

Once upon a time…

If you’ve read through some of the documentation on where Django came from, you know that it started out as a set of tools Adrian and Simon developed to make their lives at the Journal-World easier; they needed to deal with a couple of problems (most notably the tight deadlines and evolving needs of the newsroom) and so ended up writing a number of libraries to help speed up their development process and deal with some of the more tedious and repetitive aspects of writing web applications. As the Journal-World’s codebase evolved (by this time Jacob was on board as well), different parts of it started to move in different directions, ultimately producing two separate pieces of software:

  1. A set of applications and integration between them, optimized for handling the most common needs of news organizations; it’s since expanded beyond the confines of the Journal-World and is now a for-profit product called Ellington.
  2. A set of libraries and tools for developing web applications, on which Ellington is built; this is now open source and is the framework we call Django.

Every once in a while I like to go spelunking through our repository to look at the process which split Ellington and Django from each other and piece together some of the decisions that had to be made along the way (which bits became libraries in Django, which applications stayed solely in Ellington and which went open source into django.contrib, etc.). This is fascinating to begin with, but gets even more fun when you realize that the process of separating Django from Ellington didn’t end with the initial open-source release of Django; no truly proprietary code ever leaked out, but if you know where to look in older versions of Django you’ll be able to spot some references to parts of Ellington which weren’t completely cleaned up (and in fact, I’m pretty sure Django 0.95 was the first version to ship without any dangling pointers to Ellington in the code).

One good example is this bit of URL configuration which survived as far as the 0.91 release (where it shipped as django/conf/urls/registration.py):

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^login/$', 'django.views.auth.login.login'),
    (r'^logout/$', 'django.views.auth.login.logout'),
    (r'^login_another/$', 'django.views.auth.login.logout_then_login'),

    (r'^register/$', 'ellington.registration.views.registration.signup'),
    (r'^register/(?P<challenge_string>\w{32})/$', 'ellington.registration.views.registration.register_form'),

    (r'^profile/$', 'ellington.registration.views.profile.profile'),
    (r'^profile/welcome/$', 'ellington.registration.views.profile.profile_welcome'),
    (r'^profile/edit/$', 'ellington.registration.views.profile.edit_profile'),

    (r'^password_reset/$', 'django.views.registration.passwords.password_reset'),
    (r'^password_reset/done/$', 'django.views.registration.passwords.password_reset_done'),
    (r'^password_change/$', 'django.views.registration.passwords.password_change'),
    (r'^password_change/done/$', 'django.views.registration.passwords.password_change_done'),
)

Of course, ellington.registration (the original user-signup application for Ellington, now superseded by the open-source django-registration) wasn’t part of Django; it was only available as part of Ellington. These URL patterns referring to it, however, were evidently missed during the initial release of Django and weren’t cleaned up until quite a bit later.

A slightly less-obvious example is the original XMLField which shipped as part of Django’s ORM; certain content in Ellington is stored internally as XML and needed to be validated to ensure we could process it properly, so Django needed a field which could store XML and validate against a schema. By far the most human-friendly format for XML schemas is Relax NG Compact, but at the time there was no good Python library available for reliably processing and validating against Relax NG Compact schemas, so Django’s XMLField ended up with a dependency on Jing, a Java-based validation tool which could handle our schemas.

If you spend some time poking around in the 0.90 and 0.91 releases of Django, you’ll see more than a couple places where something like this managed to stick around past the initial open-source offering. With one exception, they’re all things like the URL patterns above or the Jing dependency; little bits of code which refer to or assume the existence of or interact with things which ended up on the Ellington side and not the Django side of the split (another example in 0.91 is the file django/parts/media/photos.py, which makes no sense unless you know how Ellington’s photo-thumbnailing worked at the time).

I said there’s one exception, though, and that exception is really interesting because it was a feature which literally got cut in half: part of it ended up in Ellington and part of it ended up in Django.

The “related links” feature

One largely-unsung but still incredibly useful (as in: complaints started almost immediately when it disappeared during a software upgrade) feature in Ellington consists of a fairly simple concept: for quite a few types of content, Ellington provides the ability to add a few simple hypertext links, with titles and publication dates, to arbitrary objects. These links are added and edited inline with the objects they attach to (in a fieldset labeled “related links”), and when I first started working at the Journal-World and saw it in action I thought it was just another model and a perfectly standard use of edit_inline. Here’s a screenshot taken from an admin page in an older release of Ellington:

Screenshot of 'related links' in the Ellington admin, looking like any other inline-edited model

But eventually I realized that, no matter what type of content the links were being attached to, the links themselves were all instances of one single model which was somehow being attached to, and edited inline with, virtually anything else in the database. That caught my attention, but I never had time to actually go and see how this was implemented; the feature worked, and I had genuine bugs to track down and new features to add, and so I just filed it away with an assumption that it worked like the comments application (which used a combination of a foreign key to a content type and an object ID to determine what to attach a comment to) and must have some custom admin code to make it look like edit_inline (a not unreasonable assumption; Ellington was, is and likely always will be rife with interesting uses and extensions of the Django admin).

Eventually I found the time to go digging and see how the related links actually worked. And I was blown away: though the model for the links was part of an application in Ellington, the trigger for enabling them was what looked for all the world like a custom Meta declaration, which was being recognized and processed by Django. Now, Django has never supported custom Meta options (the official list in the documentation is all you get, and trying to put anything else in Meta will raise a TypeError), but here was an option that never appeared in the documentation (in this case, the options documentation for Django 0.90 and Django 0.91, where the inner class was called META instead of Meta) and somehow still worked.

A model using related links would look like this (again, using an ancient version of the ORM where some things worked a bit differently:

class Foobar(meta.Model):
    name = meta.CharField(maxlength=200)
    
    class META:
        admin = meta.Admin()
        has_related_links = True
        verbose_name = 'foo bar'

This looks like a perfectly normal 0.91-era Django model, except for one thing: has_related_links = True, which would cause the related-link inlines to magically appear in the admin for this model. It turns out that there was still code in Django which knew about this option and, if you set it and had the application ellington.relatedlinks installed, would set up something resembling a generic relation between your model and the RelatedLink model (see, for example, this bit of Django 0.91 code, which provides part of the model-level support for related links). The automatic form generation used for the Django admin would also pick up on this and end up generating the necessary inlines for the links and handle saving them and filling in existing values when needed (see here for another part of the code).

Of course, this special-case support disappeared a long time ago, so this trick doesn’t work on a current version of Django. We still support quite a lot of Ellington installations running on older Django installs, however, so it wasn’t such a big deal until recently, when we wrapped up a major upgrade of our own sites (to a version of Ellington running on the Django 1.0.X release series; the process of how we got there from some of the most ancient Django installs on the planet is a story for another day) and suddenly started getting questions about why related links had gone away.

Related links in the modern age

Which brings me back to the present day, and specifically to what I was doing on Monday: finding a way to make related links in Ellington work like they used to, without the special-case support for them Django dropped several releases back. As it turns out, this is now surprisingly easy due to a combination of things which happened elsewhere in Django:

  1. The pattern of using a content type and an object ID to support a “relationship” to any other object in the database was formalized into generic relations.
  2. The Django admin was completely rewritten to use the new django.forms library (this was the newforms-admin project, which merged just a couple months before Django 1.0 landed) and became quite a lot more flexible.

There’s actually direct support now for inline editing of whatever generic relations you happen to have, and it’s really quite simple to set up. My development manager has cleared me to write up an example based on how related links now work in Ellington, but keep in mind that this will work for anything which uses generic relations.

First, let’s look at a simplified model implementing the links:

from django.db import models
from django.contrib.contenttypes import generic
from django.contrib.contenttypes.models import ContentType

class RelatedLink(models.Model):
    content_type = models.ForeignKey(ContentType)
    object_id = models.IntegerField(db_index=True)
    content_object = generic.GenericForeignKey()
    url = models.URLField('URL')
    title = models.CharField(max_length=255)

    def __unicode__(self):
        return self.title

So far, so good: just a basic model with a couple fields for the actual link bits, and a couple more to set up a GenericForeignKey, allowing a RelatedLink to be attached to any other object.

The next step is setting up an inline to use; with the new and improved admin, these are just subclasses of InlineModelAdmin, and are declared on the “parent” model’s ModelAdmin class using the inlines directive. So if all else fails, we could always just write a special-case InlineModelAdmin subclass which knew how to handle related links. But this need has already been anticipated, and django.contrib.contenttypes.generic already provides InlineModelAdmin subclasses which know how to handle generic relations: GenericInlineModelAdmin is the base class for these, and can be subclassed if you want to set up lots of custom behavior, and then GenericStackedInline and GenericTabularInline are pretty much exactly equivalent to django.contrib.admin.StackedInline and django.contrib.admin.TabularInline, except of course for the fact that they work with generic relations.

So in the admin.py file of the relatedlinks application, all we have to do is subclass the appropriate generic-relation inline and set it up to know about the RelatedLink model:

from django.contrib.contenttypes import generic
from relatedlinks.models import RelatedLink


class RelatedLinkInline(generic.GenericStackedInline):
    extra = 3
    model = RelatedLink

Now let’s turn to the classic example Django model: an Entry in a blog application. Here’s the model:

import datetime
from django.db import models


class Entry(models.Model):
    title = models.CharField(max_length=255)
    slug = models.SlugField()
    pub_date = models.DateTimeField('Publication date')
    body = models.TextField()

    class Meta:
        verbose_name_plural = 'Entries'

    def __unicode__(self):
        return self.title

Setting up an admin interface for it (in, say, blog/admin.py) is easy:

from django.contrib import admin

from blog.models import Entry


class EntryAdmin(admin.ModelAdmin):
    prepopulated_fields = { 'slug': ['title'] }


admin.site.register(Entry, EntryAdmin)

But now suppose we want to have related links available for Entry objects; since RelatedLink uses a GenericForeignKey it can work with any model we’ve installed, and it’s just a matter of tweaking the admin declaration for Entry to get editing of related links inline with entries:

from django.contrib import admin
from relatedlinks.admin import RelatedLinkInline

from blog.models import Entry


class EntryAdmin(admin.ModelAdmin):
    inlines = [
        RelatedLinkInline,
    ]
    prepopulated_fields = { 'slug': ['title'] }


admin.site.register(Entry, EntryAdmin)

The generic-relations code and associated inline classes it provides will take care of the rest, and the result will look like this (taken from a project I set up using exactly the code above):

Screenshot of 'related links' in the Django admin, looking like any other inline-edited model

One final bit of utility we can add here is a reverse relation on the Entry model to set up a slightly nicer API for accessing links from Entry objects:

import datetime
from django.contrib.contenttypes import generic
from django.db import models
from relatedlinks.models import RelatedLink


class Entry(models.Model):
    title = models.CharField(max_length=255)
    slug = models.SlugField()
    pub_date = models.DateTimeField('Publication date')
    body = models.TextField()
    related_links = generic.GenericRelation(RelatedLink)

    class Meta:
        verbose_name_plural = 'Entries'

    def __unicode__(self):
        return self.title

Note that here it’s generic.GenericRelation instead of generic.GenericForeignKey, and that the other end of the relation (RelatedLink) is specified explicitly, since it’s the other end that’s truly generic.

Now we have the nice API that normal relations get; given an Entry in a variable named e, we can use e.related_links.all() to access its related links (which, in turn, we can add, edit and delete inline with the entry in the admin).

And that’s the whole story

With code very similar to what’s listed above, I got the inline-edited related links working in Ellington again without needing the special-case support they used to have, and was reminded of a fun story of Django’s (and Ellington’s) development history which, so far as I know, had never been heard outside of the Journal-World’s offices. But now the story’s been told, an amusing bit of software history is out in the open and I’ve had a chance to explain a useful but underdocumented Django feature. Of course, it’ll also end up on my ever-growing list of things to add to the official documentation someday, but in the meantime I’ve got some ideas for putting generic inlines to use and exploring everything that can be done with them.