A little while back I released a couple of Django apps to the world, and I’ve got a couple more in the pipeline (whenever some more of my copious free time rolls around, I’ll finish them up and get them out the door as well), so I’ve been thinking a bit about best practices for making them as easily reusable as possible. And James Tauber has started up a mailing list for discussing a Django application repository (anyone who’s interested in that should join up and get the discussion going), where he’s posed some questions about that exact topic.
So here are my random initial thoughts and ideas on the matter.
This seems to be the one lots of people get hung up on: what format should Django apps be distributed in? Eggs are an obvious choice, and easy_install has wonderful integration with the Cheese Shop, but there are some significant downsides to eggs which make me feel a little uncomfortable.
So far I’ve just been putting things into Subversion repositories hosted on Google Code and thinking about making tarballs when I want to have an “official” package, but I’d love to see a better solution. Right now I don’t know what the right answer is to this one.
The Django tutorial walks you through creating a project and then putting an app inside that project’s directory, and this seems to be a pretty common idiom. But for distributing a reusable app, I don’t think this can cut it. For one thing, any file in your app that needs to import from another is basically stuck doing relative imports (since you can’t rely on the project name), and relative imports are evil. And that really starts to hurt when you need to do things like import from models.py when you’re inside a views module which has multiple files in it — relative imports can’t “go up one directory” to the app root to find things.
I think the best solution here is for standalone apps to be written with the assumption that they’ll live directly on the Python path, and so can use absolute imports stemming off the app name; e.g., myapp/views.py should be able to assume that from myapp import models will work. This also makes life easier for the end user of the app, because it’s quite a bit simpler to use a single installed copy of the app in multiple projects.
So far, I’ve provided example templates with both of the apps I’ve released; the pattern I’ve been following is to create a templates directory in the app, and then a subdirectory named the same as the app which will hold its sample templates. This seems to be a pretty good way to handle it, because it meshes well with the app-directories template loader; this way, a user who wants to get up and running with a minimum of fuss can just edit the sample templates in-place (to match up with site-specific block structure and inheritance, if necessary) and have things just work after that. I’ve also been getting into the habit of having custom views assume the presence of the appname template directory, and look for their template in there; generic views already do this, so it ends up being a unified convention for where the app’s views will find their templates.
Figuring out inheritance and block structure is harder; every site is going to have its own way of doing that, and it seems like it’ll be impossible to have truly “generic” templates without getting everybody to agree on some conventions. If we go that route, I think it might be nice to run with an assumed base.html which all templates can inherit from to get the site’s basic HTML structure, but I’m not sure if there should be some conventions for block names (though standardizing on a few block names might not be a bad idea; content is one that seems to be used a lot, so it might be a good candidate for this).
One more useful convention for templates is probably to name tag libraries after the application they belong to, possibly using multi-word names (with underscores) for apps which provide multiple libraries (e.g., you might have just one library — myapp — or several: myapp_foo, myapp_bar, etc.). There have also been a couple proposals for changing how Django looks for tag libraries, so that it’d become possible to do things like {% load myapp.foo %}, and that might be something worth looking into since it’d make this much easier.
In an ideal world, a reusable application will use the permalink decorator when defining get_absolute_url for its models, and will use the url tag in any sample templates it provides; that will allow on-the-fly reconfiguration of URLs without having to touch code or templates. Unfortunately, both permalink and url still have a couple issues to be worked out (mostly with cases where more than one URL points at the same view function), so there may still be cases where get_absolute_url needs to be hard-coded. When that happens, the application developer needs to document it clearly so that users will know they need to use ABSOLUTE_URL_OVERRIDES.
Some of this goes back to packaging, because a package format which allows dependency specification is really the best way to go here, but that won’t cover everything. Required settings, in particular, need to be documented, along with their acceptable values, so that users know what they’ll need to add to their project settings to use the app. Until, and perhaps even after, we have a standard package format, it might be a good idea to provide some sort of conventional way to check for dependencies; I suggested checking for this in the app itself, and I may go back and work on that a bit to provide a generic dependency-checking mechanism.
I’m sure there are more things which need to be worked out, and I’m sure there are people much smarter than me who have good ideas for how to do it, so pipe up in the comments, or join James’ mailing list and discuss there (I’ll be cross-posting this to the list momentarily).
Comments for this entry are closed. If you'd like to share your thoughts on this entry with me, please contact me directly.
Hi James,
Great post, thanks! This is a topic I’ve been pondering a lot, and here’s what I feel able to contribute at this point.
I found that, essentially, zipimport is the best thing since sliced bread. Most of my new apps — not just for Django — have a directory for packages/plug-ins that they parse at launch. Every ‘.pyz’ file found in that directory is added to sys.path. Each .pyz file is, really, a zipped Python module. This works beautifully, especially with Django, which is smart enough to use import paths in its URLs configuration, rather than filesystem paths. That way, any package dropped into that special directory can be imported with a direct import. It’s an absolute import; no need to fiddle with relative paths.
Up to now, I left the templates out of this; however, it would be very feasible to have a new template loader parse the .pyz files for the contents of a template/ directory in the .pyz packages — the native zip module of Python makes this easy.
I am not very much up to date as of yet on how shipping default settings for an app works in Django, however. Pointers about that would be appreciated, actually.
Outside of that, the only real issue that remains, as far as I can tell, is with URLs. Honestly, I am NOT satisfied with the idea of linking a model object to an URL. I’ve been fiddling with a system — a wrapper around urlpatterns — that would allow you to (optionally) name a (view, URL) pair in an explicit manner, which would allow you to point a model to the name of its preferred view, instead of either its view’s import path in urlpatterns or, god forbid, the URL itself. This is, however, no small deal, and I’m not very comfortable putting the whole thing forward, being mostly a Django newbie. Here as well, honestly, pointers would help.
With kind regards,
— S.
Hi James, Thanks for this post, you actually made a lot clear to me. You may be interested in my Google Summer of Code application about implementing a package repository for reusable Django applications – in the Django code base and with a community driven website: http://jannis.leidel.info
Best, Jannis
Nice, finally somewhere to discuss this.
Anyway, as for an archive format I’d go with *.tar since that’s just plain and simple, dependencies might be harder; a test.py might provided? That’s how I’ve done so far in projects that I distribute
I have similar issues with wanting to make my internal apps used to build a specific project more orthogonal for re-use. Things like templates becomes akward as well as settings and model imports. I find that I create monolithic projects when I don’t want to because of the investment in time of getting even internal distribution right when I have control over everything. I need to break that habit.
One problem with zip-like files is what to do with included media. You’d have to pull that stuff out of the archive and copy it somewhere. It’s nice to be able to symlink to it, or, in the case of development, to have it just work.
Packaging: I think the only option here is eggs because liking or not are somewhat became standard so having eggs in the PYTHONPATH will solve the import issue I guess
Imports: it’s not really true you can’t go up one directory. With Python 2.5 you can (http://docs.python.org/whatsnew/pep-328.html) but we know we must stick with 2.3+ so it’s not a real option. Just wanted to inform you
HTH
As far as standard template names goes, the tips that Sune Kirkeby originally posted on the wiki work reasonably well. I’m finding myself using that section’s practices regularly. I have some qualms about some of the other ideas, but the template inheritance structure works for me. There’s never going to be a perfect solution here; well-commented examples are probably the right approach.
Packaging is a tricky one. Supporting more than eggs is definitely to be recommended. For no other reason than eggs’ lack of integration with existing packaging systems and standard sys-admin practices. It’s one of the few places in Python land where it actually gets me angry to think about. Eggs are great for single systems with relatively few packages installed where you want to install to “standard” locations, but they scale badly. Now that I’ve no doubt ticked off the people who’ve drunk the Slurm and see nothing but goodness, I’ll back away from the egg issue whilst your comments veer wildly off-topic. This is probably why I don’t have comments on my site. Sorry about that. :-)
Would it be possible to change Django so included applications are added to the path, and allow an optional ‘as name’. Hence you could add application ‘admin_ver2’ as ‘admin’ and anything importing from ‘admin’ would get stuff in ‘admin_ver2’.
Henrik: not really, because Django can’t access the application unless it’s on your Python path to begin with.
Generally, what I do is have a single directory that I add to my Python path, and put everything in there so it’s directly importable; for example, djangosnippets.org uses two Django applications, Pygments, and Markdown, so there’s a
pylibdirectory on my Python path which lists out like this:Where
cabandregistrationare the two apps, andsnippetsis the project which runs the site. That makes it really easy to handle the imports.Paste Deploy does offer a system that addresses many of the packaging and configuration issues you present here, using eggs as the format. Yury Yurevich has restarted development on DjangoPaste as well: http://pythonpaste.org/djangopaste/ — which is basically just a Paste Deploy wrapper for Django.
It doesn’t address templating; I’d advise using a template search path for that, as it makes the workflow fairly simple — copy a template from the package to your local dir, then edit.
Yeah, I like Paste Deploy. I just don’t like eggs, so I’d use some other format and tell people to unpack it before use ;)
It’s a shame that Django, and python in general, doesn’t have the packaging and support mechanisms that Java has. When it comes down to it, configuration files like the ubiquitous web.xml and tools like maven would go along way to not only easing the reuse of python/django applications but also towards wider adoption.
justin: as someone who’s spent the better half of this week configuring, re-configuring, re-re-configuring an EJB3 container to do ddl schema generation properly, i wouldn’t be too quick to praise java’s methodolgy. this was using annotations, of course, so i wouldn’t have to configure things in XML files (as much)… turns out you still need to put an ejb-local-ref in the…….. WEB.XML file!!!! yeahh!11!! why didn’t it already know? they were in the same .ear file.
now, coming to django, being able to type a command and have this happen the first time is quite refreshing.
I’ve concluded that as far as the app space goes, django is monolithic. To be honest it’s not a problem for me, but if modular packaging is the goal for others; django could do with a products folder, and the loading is managed by django (no plugin architecture worth a salt relies on language import mechanisms). Like i said on the list last year, by the time you solve this problem, Django will have become an app server. Henrik’s problem is typical of what leads you to a plugin system.
My current problem with django is to do with another kind of monolith - generic views and templates afaict are designed to be globally used. That makes supporting things like multiple weblogs (a la your scaling django post last year) tricky.
I am definitely going to check your apps for idioms tho’.