Living without projects
The official Django tutorial walks you through the process of developing a Django application inside a Django project; in this view of the world, the “project” is seemingly the major focus, and the “application” just happens to be something you stuff inside it. For the purposes of learning and exploring Django — especially if you’re new to Python and unfamiliar with things like how to create your own modules and set up your Python path — this is a pretty good way to work, because it means that django-admin.py and manage.py can do most of the heavy lifting for you.
But in successful deployments I’ve seen and been a part of, this is pretty much never the way things are done; the concept of the “project” as a thing that’s worth dealing with and paying attention to, aside from its incidental duty of providing a settings file and a root URLConf, is pretty rare, and the focus is, instead, on applications. Malcolm touched on this today with an excellent write-up of application-centric development practices which you really should read, and I’d like to follow up today with some more thoughts on the same topic.
What do we mean by “project”?
The first thing to deal with is exactly what’s meant by this word, “project”, because it seems to have multiple interpretations depending on who you talk to. To many people, it seems to mean “the place where all of my site’s code goes”, which is something that’s sort-of true when you’re walking through the tutorial (if you ignore the use of the admin application and generic views, neither of which live inside your project). To others, it seems to mean more of a wrapper, a convenient place where you drop in some configuration while all the actual code — applications, templates, etc. — lives elsewhere (in the case of Python code, typically in application modules directly on the Python path). To yet another group, there’s no such thing as a “project” in most cases.
I tend to work almost exclusively in the latter style, and most real-world Django deployments I’ve seen or worked with have done the same. Of course, that may be a bit of a jump if you’re accustomed to thinking of projects as the place where everything has to go (and if you’ve worked with some other frameworks which don’t have a distinction similar to the one Django draws between projects and applications, it can be a natural thing to do), so let’s look at how an “application-centric” development style helps you out in the long run.
The pitfalls of project-centered development
As Malcolm points out, and as most people find out once they start developing moderately-complex applications, the biggest drawback of trying to put everything inside a project is that everything ends up coupled to the project; you either have to maintain that directory structure everywhere you use one of the applications (so that the imports, which will depend on the existence of the project directory, will work) or you have to do some gymnastics with your Python path to ensure that the individual applications are importable independently of the project. At which point the alternative — developing the applications directly on the Python path, independently of any particular project — really becomes attractive.
It’s also generally much easier to produce distributable applications when you’re developing them independently of a project; the two most common Python distribution mechanisms — distutils and setuptools — are both at their best when you have some flexibility to place distribution scripts (such as setup.py and a package manifest) in a directory above the actual application module; project-centric development impairs this severely.
So what are projects good for?
Aside from providing a place to keep configuration (e.g., the settings file and the root URLConf), there’s really only one use case I know of for projects containing actual code, and that’s when there’s some bit of code which is so coupled to a single deployment that it doesn’t logically make sense anywhere else. One common example is a site-specific “home page” view, which is logically part of the project it’s used in.
I don’t really use this myself, though; I typically have enough site-specific code that it makes sense to put it in its own module and treat it like another application. For example, I’ve written about the system of redirects I use on this site to handle the transition from an older URL scheme, and that code — along with a few other things specific to this site — lives in an application module named b_list, which is directly on my server’s Python path.
There is no spoon, and no project either
From this point it’s a fairly easy jump to just not bothering with “projects” at all, at least not as first-class modules in their own right. At work we manage a lot of Django-powered sites which use Ellington, our news-oriented CMS. Ellington is, ultimately, just a large collection of Django applications which provide different components of a news site, and so all of those applications live inside an encompassing module named ellington which sits directly on a server’s Python path. Meanwhile, the settings files for all of our sites live in a different directory (called, appropriately enough, settings), and root URL configurations for the sites live in a urls directory (though I’m simplifying a bit here for sake of example; there are some particular aspects of our setup that aren’t really relevant to this discussion).
This means that there’s really not anything, anywhere on any of our servers, which corresponds to a “project”, and it’s actually an incredibly flexible way to work; individual sites’ configurations can be edited easily, and new sites can be rolled out with a minimum of boilerplate (since all that’s needed on the Python side of things is a new settings file and a new URLConf).
How I work
For one more real-world example, I’ll go over how I manage my own personal development work, because I have a setup that’s pretty simple to work with, but which makes writing and maintaining my applications fairly easy.
First of all, I have a directory, ~/dev/personal/, on my laptop which holds the full setup of every application, including its parent directory for packaging purposes; for example, django-registration lives in ~/dev/personal/django-registration/. This directory, and the package directories inside it, are not on my Python path.
Meanwhile, a second directory — ~/dev/python-local/ — is on my Python path, and it holds all of the third-party Python software I use. Each of the package directories in ~/dev/personal/ contains a Python module which holds the actual application code, and I’ve just got symlinks from ~/dev/python-local/ into those directories, which means that the application modules are right on my Python path, with the packaging information conveniently out of the way. I do something similar for my checkout of Django, because I’ve got a copy of the full tree — branches, release tags, trunk and all — and there’s a symlink in ~/dev/python-local/ which points at the trunk/django/ directory of that checkout (and which can easily be pointed somewhere else if I need to work against a specific release version of Django).
From there it’s pretty easy to use various flexible solutions for settings files and URL configuration; I’ve got two hierarchies of settings and configuration data — one for my personal testing setups and one for the sites I deal with at work — and just point to whatever’s appropriate for a given situation.
And, again, there’s absolutely no need to have “projects” in this setup. Like Malcolm, I still do use a project module every once in a while as a quick way to get something up and running, but I inevitably factor everything out of it before I get close to deploying or distributing. The result is a simple — but still flexible and powerful — development platform which translates well into production deployment or packaged distribution, depending on what I’m working on.
Feed of latest entries
November 10, 2007
#
I have to say this is the most controversial post in my opinion so far. “and most real-world Django deployments I’ve seen or worked with have done the same” is just too opinionated, tho you are just mentioning your personal experience.
The point is, you registration case is too contrived, it simply falls on its face on my site, where I expect users to enter their mobile number, and registration involves mobile confirmation for instance. The registration app that I churned out, turns out to be too specific to my site, sure its a loft goal to write reusable modules, but almost none of the dozen apps powering my website would make any sense for anyone else. Why do I use apps? To divide them on smaller manageable chunks.
Your post just gives an idea that there is some flaw in my way of doing things, that is not decoupling apps from each other, by implying that all django experts you know only use app base development.
Re usability of apps can only be accomplished by catering to really common denominators, or requires too much thinking and planning. Consider django’s admin for example, its cool and all as long as you stick to doing almost exactly what it is intended to do. Registration app falls flat as soon as I had the mobile number, and verification. Comments app has its issues.
There is another reason projects are important in my opinion. I consider it a best practice to checkin django itself as part of my project. Reason being I have made a lot of changes to django, and this gives me a control, esp when I used to follow trunk. There have been backward incompatible changes in django on a fairly regular interval, and the only other way to ascertain that my code works against all developer machines is to convey to all developers the revision number of django trunk. This is quite important concern in my opinion, tho I am using .96.1 in my main project, in the next project or in the same project after I migrate my code to be compatible with trunk, I would be following trunk, and this is the advise I give to everyone as the release are quite infrequent, last release really pales in comparison to trunk.
On the same machine I develop my blog that follows trunk, and I develop that works against last release, that would be unnecessarily tricky unless I checkin the django itself with my project. And the few reusable django apps I are also checked in with my code, and they all combined make up as my project.
This sounds like best practice to me, at-least till the difference between django trunk and latest release is not as huge as it is, or when I would not be left with any reason to patch django code, and both of them are not going to happen anytime soon.
November 10, 2007
#
Amit, maybe we’ll just have to agree to disagree, but again — in Django deployments I’ve seen and worked with, which is a fair number of them — there’s typically been no need for a complex “project”; when there’s a lot of code tied to a single site, I agree it can make sense to put it into a “project” module, but personally I tend to just put that code into normal application modules and keep a settings file and root URLConf.
I’ve had the pleasure (take that sarcastically) of maintaining heavily-patched copies of Django, and personally I found it to be a nightmare; that’s why, both personally and at work, I’ve done my best to spin things out into separate applications whenever possible (e.g., I developed
comment_utilsinstead of maintaining a hacked-up Django with comment-moderation features).November 11, 2007
#
I agree that projects don’t make sense for certain websites, but I don’t think its fair to say they are never useful. A CMS is a perfect candidate for app centric development because it is really a mash up of reusable components. But I think your argument against projects falls apart when the “domain” of your website narrows. A lot of my apps are so site specific it doesn’t make any sense to not view them as part of a project — they would never be used anywhere else.
November 11, 2007
#
Ryan, notice how I didn’t say “never use projects”; I agree, and even stated pretty clearly, that for code that’s absolutely coupled to a single site, they can make sense. I just personally prefer to spin that stuff out into one-off applications, and keep the “project”, such as it is, down to the bare minimum of a settings file and URLConf.
November 11, 2007
#
I for one thank James for this insight into how we might make our development environments more productive. Whether or not his advice applies to my particular situation or not, it provides context from someone who has worked with this technology a while, and may be useful to me either in the present or future. Thanks James!
November 12, 2007
#
Well this post was an eye opener for me. I’m relatively new to using django on a higher level than just basic websites and I can now see how we can work out a few generic applications and template(tags) for basic menu structure. Along with a CMS and some generic models (article, page, etc..) we’ll be able to roll out sites in no time!
November 12, 2007
#
James, what you say makes sense to this Django n00b, but I still have the problem of where to put the media (js, graphics, video) for each app that’s independent of the project.
Do we have each app supply its own media in its subfolders, and then have a collection of links from an Apache-accessed top-level media directory?
What kind of approach do you see in common use here?
Thanks for any advice.
November 12, 2007
#
James, Can you explain how you organize templates within this structure? If two different sites use the same application but have different templates for that application, how is that handled? Is it similar to your media solution (see Chris R’s comment)? Do you make some assumptions in your templating, similar to those described by Adrian in this older post? Finally, how do you tie these things back to svn/cvs/vcs structure?
Thanks for your writing on this front. I’ve been collecting various resources on operational conventions around django, and several of your posts are in the list.
November 12, 2007
#
I’m using projects for my sites and found them quite useful because each site(project) has a lot of its own specific data: - web templates - as rule each site has its specific urls, context processors, models, etc - default settings file - some apps may have site specific changes - scripts In terms of subversion, all similar sites-projects are branched from the trunk. Yes, this introduces necessity of merge between projects, but on the other hand, it gives more control on how changes in the particular app affect sites.
November 21, 2007
#
Great stuff, James, thanks so much for this series. +1 to wanting more info about how you organize media and templates in a project-less structure. Maybe this is coming in a later post?
November 24, 2007
#
I find that the best way to arrange my Django code is to place all of my projects in a single directory, say “jeffs_django_sites”, and all of my applications in another directory, say “jeffs_django_apps”.
I place both directories “jeffs_django_sites” & “jeffs_django_apps” on the Python path. Then each application can be imported directly without reference to a project, and each project can be imported directly as well.
I maintain separate subversion repositories for “jeffs_django_sites” & “jeffs_django_apps”, which makes it easy to commit changes to all applications or all projects in one go.
I enjoyed reading your article - it helped me organize my thoughts and directory layout, something that always disturbed me, until now.
November 29, 2007
#
It took a bit of a push, but between this post and Malcolm’s I feel like I’ve made a definite jump with my Django ‘organisation’. In the past I’ve found myself implementing the same thing in a copy-paste fashion from my other projects code, and ending up with ten slight variations on a theme. Now I’ve sorted out my Python-path - and reasonable locations for settings, URL confs and applications - everything is much more accessible and geared towards me not repeating myself.
Now I’m working on one generalised and distributable personal messaging system for any number of sites, rather than ten variations which were all hooked into corresponding ‘projects’ in various nasty ways. There is no spoon, and it feels great :D.