So my little rant on AJAX support in Django apparently touched a nerve with a couple people; that means it’s time to write more about it.
One of the common points people have been raising, in comments and elsewhere, is that I shouldn’t rail against “hiding JavaScript from the developer” when Django’s ORM already hides SQL from the developer; from the perspective of a server-side developer, SQL is just as important, right?
Yes, SQL is just as important. But having ORM isn’t “hiding SQL”.
The intialism “ORM” stands for “object-relational mapping”, which should give away what it’s all about; an ORM system is not about keeping you from having to write SQL, it’s about providing a way to translate between objects in code and records in a relational database. In other words, ORM (in the simplest case) is a way of saying “this class maps to this table, and objects of that class map to rows in that table”. Instance attributes of the class can then be selectively mapped to database columns of varying types.
And the ORM systems in popular frameworks do pretty much that exact thing, falling mainly into two types of workflow:
In the first case you need to write and execute the CREATE TABLE statements yourself, so it’s hard to see how this “hides SQL”. In the second case you don’t have to (though you will, from time to time, want to edit them before they’re executed to do a little extra fine tuning), but you still need to be thinking in SQL terms as you create the model classes. Django’s model system enforces that somewhat further than necessary by using names like ForeignKeyField which serve as reminders of exactly how your models will map onto database tables and relations.
Once your database is created, you can, if you really want to, get by in Rails or Django without ever writing a line of SQL, but you’ll probably find yourself severely hampered because there are some queries you’ll want that just can’t be expressed cleanly in their ORM systems. For example, consider a discussion forum application written with Django, with the following models:
class Forum(models.Model): topic = models.CharField(maxlength=200) moderators = models.ManyToManyField(User) class Thread(models.Model): title = models.CharField(maxlength=250) started_by = models.ForeignKey(User) start_date = models.DateTimeField() forum = models.ForeignKey(Forum) class Post(models.Model): author = models.ForeignKey(User) pub_date = models.DateTimeField() body = models.TextField() thread = models.ForeignKey(Thread)
Now consider a common feature for such an application: listing the five most active threads in a forum, say over the past twenty-four hours. How would you get that using only Django’s ORM? I’m pretty certain that you can’t, and even if you could it’d probably be hellishly complex.
In straight SQL, on the other hand, getting a list of five “most active” thread ids in order of “activity” is a single query:
SELECT t.id AS thread_id, COUNT(*) AS score FROM forum_thread AS t INNER JOIN forum_post AS p ON t.id = p.thread_id INNER JOIN forum_forum AS f ON t.forum_id = f.id WHERE p.pub_date > [a datetime value twenty-four hours in the past] AND f.id = [the forum id] GROUP BY thread_id ORDER BY score DESC LIMIT 5;
Once you’ve got it you’ll probably want to feed that list of thread ids into Django’s ORM (using in_bulk) to get the Thread objects, but getting that list of ids was something you couldn’t do without writing SQL. Also, that reinforces once again that the primary purpose of ORM is not to “hide the database”, but rather to translate between database records and code objects — once you have the ids, you just ask the ORM to translate them into the appropriate instances of your model.
Hopefully that will put to rest the notion that having an ORM system is meant to “hide SQL from the developer”. No matter what you’ll end up thinking in SQL terms, and while you can try to never write any SQL it probably won’t get you very far; any ORM sufficiently complex to really replace SQL would, pretty much by necessity, have all the warts and complexities that drive people away from SQL in the first place.
Meanwhile, the whole point of the sorts of JavaScript “helpers” people keep clamoring for from Django does seem to be to hide JavaScript from the developer; hence the repeated lament of “why should I learn JavaScript just to do this effect?”
Going beyond the laziness involved in asking such a question — you’re a web developer and it’s your job to know this stuff, so learn it already — I also don’t think that automated “helpers” could ever do much beyond the most basic operations. And those basic operations are obscenely easy with the various modern JavaScript toolkits, so actually writing some JavaScript to get them still seems like the simplest and most straightforward way to go, and gives you a good foundation for developing more complex things.
Comments for this entry are closed. If you'd like to share your thoughts on this entry with me, please contact me directly.
not to belabour a point here.
a web developers job is to write applications which bring in money, not to write javascript code. just because I know how to do it, doesn’t mean I need to do it.
helpers will get the majority of the work done for me. If I hit a bug, or want a feature the helpers can’t do THEN I start coding javascript. the helpers just get me further in less time.
and at the end of the day. my time writing this app is what I am trying to minimize (so I can go and write another)
Ian, I’ll ask the same question I asked in my original post, the same question I asked of a commeter on your blog, the same question that no-one has yet answered:
If a framework should provide “JavaScript helpers” so you don’t have to write any JavaScript for common tasks, then shouldn’t it also provide “HTML helpers” and “CSS helpers”? Why waste valuable time writing a template or a stylesheet when, in most cases, I could just write something like
{% two_col_layout content_col contains entry_body sidebar contains entry_summary entry_category_list %}? That’d cover 80% of all weblog applications and layouts right there — why should we expect people to write any more code?yes. it should. it would be great for prototyping.
could you imagine telling your boss.. “i had this great idea last night, and here is a working prototype…
it looks a bit fugly.. but it is good enough to show to some marketing types, and if we can get support from them we can hire a designer to make it look nice and work on the UI a bit more.
Well, at least you’re honest ;)
When this is all over, shall we just have a framework that says “PUSH BUTTON TO MAKE WEBSITE” and reads your mind to find out what it should do? I’d imagine we could make a lot of money at that…
james. your misunderstanding my point. it isn’t about making a full site. it’s about getting a shell up and running.
right now django gets me going with generic views, and the admin stuff, but then leaves you to your own devices. if it could provide a default CSS, and a default ‘generic’ form/layout it would help people get something up and running faster. and you can prototype your app and how the pages would be structured a bit easier.
most large sites won’t use it, but for small things it will get things going faster. the main use would be for prototyping (the CSS/form generation). but sure for least-used pages it will probably stay.
James & Ian, I think you guys are arguing at the extremes here.
James - you seem to be arguing, that a “click this button to make site AJAXy” is wrong, yes, it is. e.g. do_ajax();
Ian - you seem to be arguing that writing every single bit of js from scratch is wrong, & yes, it is too. e.g.: try { xmlhttp = new ActiveXObject(“Msxml2.XMLHTTP”); } catch (e) { try { xmlhttp = new ActiveXObject(“Microsoft.XMLHTTP”); } catch (E) { xmlhttp = false; } } ….. etc ( first XMLHttpRequest-y thing I googled from http://jibbering.com/2002/4/httprequest.html )
I think most people would agree that both of your arguments are correct - but that’s not the issue here. What Django should be interested in doing, is coming somewhere between those two extremes, eg.:
$XHR = getXMLHttpRequestObject(); … now application specific code
That is, there’s a region where abstraction and a framework can help you the developer do that same old crap (browser detection, etc) quickly, and get onto doing the more difficult stuff. This is what the SQL ORM does, when the framework can’t do it, then we move onto raw SQL. It’s not about dumbing down things, but about making it easier to do the dumb and boring things.
—Simon
Ian,
I’m not entirely certain that’d be within scope for Django. Besides, my question wasn’t “do you want a throwaway HTML/CSS shell”, my question was whether HTML and CSS “helpers” should be provided along the same lines as JavaScript helpers, which are intended to be production-quality and remain in the final version of the application.
Simon:
No, Django absolutely should not be in the business of providing Yet Another Wrapper For XMLHttpRequest(TM). All the popular JS libraries have one, so (unlike the original development of Django, which took place before there were good resuable Python components) that would be a very bad case of reinventing the wheel.
Django should just be Django, and let people use whatever JS they want.
i personaly think of djano as a backend framework. html, css, and javascrpt are all out of the scope as far as i’m concerned.
(on the 770 so ignoring case. sorry.)
Maybe one middle ground is a bunch of nice cookbook type examples of using AJAX with Django to accomplish some of those simple things. I admit I’m no javascript expert and anything Django (or anyone) can do to help me get up and running quickly would be appreciated. I am smart enough to realize there may be a point where I get backed into a corner and need to roll up my sleeves or bring in an expert - then I will.
One thing I do think could tip the scales is if there is any kind of widespread improvement to the Django admin site. I think there are plenty of opportunities to improve it with Ajax. Once a toolkit is chosen and integrated, isn’t it a moot point?
Chris, in large part I think I’m now against the “integration” of any particular toolkit into Django in the sense in which most people mean that word. I think there are a lot of things we can still do to improve the admin, and we’ll undoubtedly pick a toolkit and use that for it (right now Dojo is looking like the winner since I’ve already done a lot of the initial work to get the admin to use it), but I’m hoping that doesn’t lead people to believe that they can only use that particular toolkit with Django.
It’s a tricky situation, trying to keep things as loosely-coupled and non-dependent as possible…
James: I’m not suggesting that Django WRITE another framework - that would be stupid, and definitely outside the scope of the project. I’m saying that there’s plenty of room to slot in an existing framework that complies with Django’s core principles like DRY, loose coupling, explicit better than implicit, etc.
You say “Django should just be Django, and let people use whatever JS they want”, sure but where does a framework stop? There used to be quite a clear client-side/server-side distinction going on, but things like XMLHttpRequest blurs that line. You can distinguish between display stuff and application logic, but then why are you arbitraily drawing the line at javascript? & in does that mean that django shouldn’t do the V part of MVC?
Let me reword your statement: - “Django should just be Django, and let people use whatever SQL ORM they want.” - “Django should just be Django, and let people use whatever template system they want.”
Django does - it’s apparently fairly easy to change the template system/db api ( never tried myself ),and there’s no reason that whatever JS library Django chooses wouldn’t be the same.
—Simon
James — great set of articles on this topic.
Concerning the criticisms — if the goal is not to avoid writing JavaScript, than what do people want to gain from the server-side writing JavaScript for them. There’s the “oh-but-I want-to-save-time” arguments, but I thought that’s what JS toolkits did — help write JS quicker (!) What reason is there for generating JS from the server-side except to avoid writing JavaScript?
james, it does not matter what kind of client side-ajax support django has (like helpers etc.) if at all. what really matters if someone comes to the django’s website, and new to django, there should be a tutorial which explains how it can be done. one way. as an example. why the support as it is. these things should be there to communicate the matter.
why it is important? - because you have to sell django and the rivals all have a solution for ajax. not presenting this argument and not presenting a possible solution means django is hard to use for ajax which is not true.
AJAX helpers are one of many things that I think could be rolled into a re-usable app instead of included in Django. Much like the admin and comments.
I agree with Gary on this. This is something that should be a contrib package, especially with the variety of javascript libraries out there. We could have one for each JS library depending on people’s preferences, given the chances of there being one for each library are slim to none but we can dream. :D
There are a variety of things that would be great to have already done for me, e.g. an autocomplete select form field. Other things, I wouldn’t mind the extra few minutes spent writing the code to allow for customization.
I do think there needs to be tutorials on this. But because of the amount of AJAX/javascript knowledge out there, I don’t think they need to be very indepth. It isn’t that hard to connect AJAX and Django, actually, it’s pretty darn easy but just undocumented.
One thing, not sure how many people have thought of this (probably quite a lot), but you can put template tags into JS files. Can probably to some neat things with that, such as themes for the animations or some other crazy stuff.
Hello, from the sidelines. :-)
I think Gary has a great idea… I think we’re reaching a compromise here by putting any “ajaxy helper stuff” inside a contrib’d app. Each app would be great as a light “helper” app for bulding prototypes on top of the generic views. I can see this kind of like how Ubuntu comes in 3 flavors - Ubuntu (Gnome), Kubuntu (KDE), Xubuntu (Xfce). Kind of funny how this pattern keeps reappearing… Reasonable people can’t agree when it comes to the visible stuff…
So there could be 3 helper contrib apps, each “ajaxified” on the client by the popular toolkits (Prototype, MochiKit, and Dojo), but they’d all plug-in to the standard serializer on the Django server side.
Is the serializer module on the server side done? Or is there more groundwork to be done before those Ajax-y contrib’d apps can start to flourish?
The argument of what a framework should include and shouldn’t include is really an arbitrary argument. The line could be drawn pretty much anywhere. The gist from you seems to be that the framework shouldn’t be doing the client side stuff (javascript, css, html) that the web developer should know enough and being doing those items him/herself? You say: “If a framework should provide “JavaScript helpers” so you don’t have to write any JavaScript for common tasks, then shouldn’t it also provide “HTML helpers” and “CSS helpers”? Why waste valuable time writing a template or a stylesheet when, in most cases, I could just write something like {% two_col_layout content_col contains entry_body sidebar contains entry_summary entry_category_list %}? That’d cover 80% of all weblog applications and layouts right there — why should we expect people to write any more code?”
Isn’t that the admin framework for Django? You define the models add a couple lines of code and poof you have a completely working application with html, css, javascript, the processing code everything. And the developer didn’t write a line of the code other than the model. How is that different from what you are saying Django shouldn’t do? Should the admin interface not be part of Django? Yes, technically it’s a contrib package but how many sites out there using Django don’t use it? I believe it was Adrian who said it’s the “crown jewels” of Django.
No, that’s not the admin application. The admin application is there because an administrative interface for site staff to use is something that’s needed by pretty much 100% of database-driven websites. It’s not intended to be a substitute for public views that actually display your content, or a substitute for building up templates and CSS for those views, or a substitute for anything other than, well, building an administrative interface for your staff to use.
So if I can sum up your argument it is bassically “javascript sucks, but that’s ok, html/css suck too”. Yes, I think we should have javascript helpers, and we should probably have html/css helpers too. Why is this such a blasphemy?
In every website you create you have to repeat the same idiom over and over again to handle all the popular browsers. If you are any good, you’ll just extract that idiom and reuse it accros your projects. Why not go that extra mile, and put it in django?
The whole point of high level languages is that you can write at a high level without caring about low level details. Checking what browser the user is using, and how you should handle the differences is really just an unimportant low level detail.
No, that’s not “summing up my argument”, that’s “putting words in my mouth”.
If someone called himself or herself a “professional web developer” but also said “I don’t want to learn HTML and CSS, someone should make my framework generate them for me”, no-one would take them seriously — part of being a professional web developer is being familiar with those technologies.
Similarly, part of being a professional web developer is being familiar with JavaScript. And one of the overwhelming reasons why people clamor for JavaScript helpers is, quite literally, “I don’t want to learn JavaScript, someone should make my framework generate it for me”.
With JavaScript, smart people have already done that and created JavaScript toolkits that are free for anyone to use. What’s so blasphemous about asking people to pick up one of those toolkits and use it? Why does a server-side framework also have to get into the mix?
It’s not that I don’t want to learn javascript or html/css, I know javascript, I know html/css. Yet I don’t want to deal with boilerplate stuff or browser incompatibilites. That’s low level stuff that gets in my way. That’s why html/css/javascript kinda suck.
Why not add a nice loosely coupled library to django that allows you to ignore that kind of stuff most of the time? One major idea of django is being able to develop a website very fast. Low level details that slow you down should be avoidable. That’s why I think it should come with a loosely coupled component that still integrates nicely with the rest of the system. I don’t really care wether that component is a javascript library or a bunch of javascript helpers that generate javascript code. Pick a javascript toolkit, make sure it works easily together with django, without coupling it too tightly. I really don’t see any problems with that.
Again, pick up a JavaScript toolkit like Dojo or YUI or MochiKit or Prototype, and run with it. Why do we need to bundle one when those are all available to anyone?