Helpers, scaffolding, tradeoffs and other stuff

An entry published by James Bennett on July 17, 2006, Part of the categories Django, Frameworks, JavaScript and Programming. 22 comments posted.

In one of the very, very few coherent things I’ve seen him say in comments posted here and elsewhere, one Lucas Carlson brought up the other perceived advantage of JavaScript helpers: they save time:

Sure it is possible to add javascript helper functions to Django, and yes that would speed up initial development times and reduce bugs since Python is usually more terse than JS… even for expert JS programmers.

I’m going to ignore the “fewer bugs” argument because it seems to me that writing code in one language which ends up outputting code in another language is something that, however terse the input may be, is inherently just begging for problems.

Ian Holsman touched on the idea of time-saving and development speed in a more coherent way, over a series of comments.

First:

helpers will get the majority of the work done for me. If I hit a bug, or want a feature the helpers can’t do THEN I start coding javascript. the helpers just get me further in less time.

Second:

could you imagine telling your boss.. “i had this great idea last night, and here is a working prototype…

it looks a bit fugly.. but it is good enough to show to some marketing types, and if we can get support from them we can hire a designer to make it look nice and work on the UI a bit more.

(Aside: I’ve used Django to do pretty much that exact thing.)

Third:

right now django gets me going with generic views, and the admin stuff, but then leaves you to your own devices. if it could provide a default CSS, and a default ‘generic’ form/layout it would help people get something up and running faster. and you can prototype your app and how the pages would be structured a bit easier.

So let’s talk about development speed for a moment.

The key word here is “initial”

It’s nice that Lucas recognizes this, because it’s the great deep dark secret of “helpers” of all stripes: the only time they reduce is that which comes between the start of the project and the first demo.

To understand why, consider a somewhat related feature that several people have asked Django to provide: the “scaffolding” offered by Rails and a couple of its clones. Scaffolding is a set of automatically-generated views/templates (in Rails, using the default template engine, a “view” and a “template” are pretty much the same thing — in other frameworks or other situations, not necessarily) which provide basic CRUD functions for a given data model. They’re provided with the goal of getting you off the ground quickly, and do so in two ways that I can think of:

1. They give you an immediate rough user interface which can be tweaked a bit and then shown to the client for feedback.
2. They provide a place where content administrators can begin putting stuff into the database while development of the application proceeds.

But somewhere along the road to deployment, there will be a moment when scaffolding just doesn’t cut it anymore and you’ll have to write the real views and/or templates for the project. This is intentional; as the name implies, scaffolding is meant to give you a place to stand while you build the actual application, not to give you a finished structure.

Generally, that moment will come after you’ve had your first successful demo and gotten the feedback you need to determine what the client really wants (as opposed to what the client initially asked for, which is never what they’ll actually end up wanting). So scaffolding gives you an incredibly fast path to your first demo and the feedback you’ll need to continue, but after that it’s pretty much useless and needs to be ripped out and replaced.

Back to JavaScript

Now, I’m going to go out on a limb and assert that the same will be true of pretty much any system of “JavaScript helpers” anyone ever develops. The reason for this is simple: there’s no such thing as “one size fits all” in web development.

Yes, you could provide a flexible set of “helpers” which just drop arguments into basic JavaScript functions, to get “flash a yellow fade behind the div with this ID” or “submit this form via XMLHttpRequest” or whatever.

But that’s it; for more complex effects — and sooner or later you’re going to need more complex effects — you have to buckle down and write JavaScript. Otherwise your “helpers” are going to get more and more complicated and less and less useful as you try to accomodate more and more needs, and very quickly you’ll arrive at something that’s not recognizably less difficult than using a good JavaScript toolkit.

Or — and I know this is a crazy idea — you could start out differently, by picking up a good JavaScript toolkit and learning how to use its equivalents of “yellow fade” and “AJAX form submit”. Then as your needs get more complex, you don’t have to throw away your helpers and start over with custom code; instead you can take the JavaScript you’ve been using and keep building on it with, well, more JavaScript.

To help illustrate this, let me talk for a minute about code generation.

DO NOT EDIT THIS FILE MANUALLY

One of the perks of my job — at least to me — is that I get to poke around in the company’s code repository, which has a full revision history of the code which eventually coalesced into Django. A few thousand odd revisions back, the proto-Django had features which, like scaffolding and helpers, would generate code for you; remnants of that are still to be found here and there in files which begin with the ominous message, “DO NOT EDIT THIS FILE MANUALLY. IT WAS GENERATED BY A PROGRAM.”

To touch on the “one size fits all” idea again, one of the problems with this sort of code generation is that you inevitably pay a hefty price in flexibility; the system is only as smart as you make it, and if you ever need to add “just one little extra feature” you need to re-wire the whole thing.

After a while, the effort and time spent rewriting the helper every time you need something just a little bit different start to get wearying and costly, and a better solution presents itself: take common pieces of functionality, and encapsulate their logic in a reusable way. Django’s generic views do exactly that, distilling and genericizing the logic of common operations into functions which are callable from anywhere. Where before the process of tweaking a generic view might have looked like this:

1. Find the program which generates the views for this operation.
2. Edit that program.
3. Test that it now outputs views which do what you want, without adversely affecting any other views it has to generate.
4. Re-generate your views.

Now it looks like this:

1. Write a wrapper function around the generic view.

Django’s admin application also used to do a lot of nasty on-the-fly work to programmatically generate its templates; people who’ve been following Django from the beginning will remember this, because it wasn’t fixed before the first public release. The “new-admin” branch ditched that and introduced a simpler system which avoided generating templates on the fly; now the admin is much easier to maintain, and is much easier to selectively extend and override by adding, removing or changing individual templates.

JavaScript “helpers” in frameworks are just “THIS FILE WAS GENERATED BY A PROGRAM” in slightly prettier clothing; they can only do what they’re programmed to do, and it doesn’t take long before the cost of re-programming them to suit your changing needs gets to be prohibitive.

Meanwhile, those pesky JavaScript toolkits that nobody on Earth seems to want to learn have gone and done the JavaScript equivalent of generic views: they’ve taken the common bits of logical functionality and encapsulated them in reusable ways. Sooner or later that’s what you’re going to have to do, so why not be doing it from the start?

Anyway, back to the real meat of things, which is what actually goes on when you use a helper.

It’s a trade

What’s really happening every time you use a helper, or scaffolding, or anything else of that nature, is that you’re making a trade: in return for decreased time until you can demo your application, you accept the penalty of having to go back later, rip out the automagical generated stuff and write real code later on. You were going to have to write real code no matter what, of course, but using the helper put that off until a little later on in the development process.

Hopefully by now you understand why that is: helpers, by their nature, are never going to be able to scale up to the complex effects people will need for production-quality applications, so whenever they’re used they’re eventually going to have to be replaced.

On “agility” and how helpers don’t help it

Of course, at this point plenty of people will be thinking that the process of starting out with scaffolding/helpers, then progressing toward real views/templates/JavaScript exemplifies “agile development”.

True agile development is a great thing; one of the biggest of its Big Ideas is that, once you start delivering, you’ll have the ability to keep on delivering, with consistent quality, as various aspects of the project change over time. Often this aspect of agility is also referred to as “sustainable development”.

In agile development circles you’ll often hear talk of a principle called “YAGNI”, which stands for “You Aren’t Gonna Need It”; much of the day-to-day work of sustainability consists in answering two questions:

1. What are we actually going to need?
2. When are we actually going to need it?

Answering these questions correctly is an art which can take a lifetime to master, and requires an acceptance of the fact that the answers will change as you go. The linchpin of the whole thing is finding an answer to a third, overarching question: how do we set things up from the start to make it as easy as possible to add new things as we need them?

Dissertations have been written and stupendously long flamewars have been fought over the answer to that third question, and there’s still no consistent right answer, just lots of general guidelines and a need for people who have smarts, experience and a finely-tuned sense for figuring out the specifics on a case-by-case basis. But I can tell you, here and now, that anything which involves the phrase “and somewhere we have to plan to pause, rip out what we’ve been doing and write the real application” is starting out with the odds stacked heavily against it.

Another of the Big Ideas of agile development is short timeframes for the various iterations of the project; rather than a monolithic eighteen-month development period followed by a gala product release, agile development focuses on quickly making a set of changes — with a timeline measured in weeks or even days — and rolling them out; lather, rinse, repeat.

But the time tradeoff involved in using helpers to kickstart your project seems to contradict the goal of agility; you could take a stab at incrementally pulling them out and replacing them with “the real code” over a few iterations, but that’s going to add additional planning complexity to your project as you try to figure out which bits need to get rewritten in which order. And it will feel as if, instead of being agile and incremental, you’re back in a long-term monolithic-rewrite frame of mind.

So, no. Starting with helpers and ripping them out when you need “real code” later probably doesn’t exemplify agile development.

On the other hand…

There is something you can do, however, that will promote agility: start out with real code. The sooner you ditch the training wheels and write real, reusable, extensible code, the better off you’re going to be in the long run.

That means that you, or someone on your team, will have to actually learn JavaScript, rather than asking the backend guy to write “helpers” you can use.

That means that you, or someone on your team, will have to actually mock up page templates at the very start, rather than asking the backend guy to generate “scaffolding” you can use.

That means it will take you a little bit longer to get to your first working demo, but if your people are good at their jobs the time difference isn’t going to be significant enough to warrant doing things any other way. And the long-term benefits in agility and sustainability will be more than worth it.

The end

This ended up being much longer, and taking much longer to write, than I’d initially planned, but now I think I’ve exhausted the topic of “helpers” and my feelings about them.

Any questions?

← Django tips: Hacking FreeComment Apropos of nothing →

On July 17, 2006, Jeff Croft said:

This ended up being much longer, and taking much longer to write, than I’d initially planned…

Perhaps if your blog application included some built-in “helpers” to get you started on the process of writing this post, you would have been able to develop it in a more agile manner.

:)

On July 17, 2006, James Bennett said:

Dude, what the hell are you doing up at this hour? Only Matt and I are supposed to be the developers who never sleep…

On July 17, 2006, Ian Holsman said:

you really have a chip on your shoulder about this don’t you ;-)

and in the mean time I’ve been wasting the last hour or two of my life getting some crappy js working.

On July 17, 2006, CC said:

I disagree, you could use the same logic with generic views and helpers. There is a point where generic views are not useful anymore and you have to write your own views.

On July 17, 2006, Mark Wubben said:

A good use for generating JavaScript on the server is for (strictly confined) communication with the scripts on the client. Being able to communicate from inside the main application code is a good thing in my opinion.

On July 17, 2006, James Bennett said:

I disagree, you could use the same logic with generic views and helpers. There is a point where generic views are not useful anymore and you have to write your own views.

No, actually that’s not the case. Consider a simple JavaScript helper which does a yellow fade behind a div with a particular ID; suppose it’s implemented like this:

{% javascript_helper yellow_fade mydiv %}

This will flash the fade on the div with ID mydiv. This is all well and good so long as all you need is the yellow fade. But what if you want to specify how long the fade lasts? Now you need to rewrite it to support

{% javascript_helper yellow_fade mydiv 2000 %}

Where the final value is how long (in milliseconds, which is how JavaScript does timing) to make the fade last. What if you want to specify a color other than yellow? At every stage you have to rewrite the helper, because there’s no way for you to build on top of it. And you get the joy of having to worry about compatibility with code you’d already written based on the older versions of the helper.

In other words, a helper by its nature is not extensible and is not reusable as a building block for more complex functionality.

With generic views, on the other hand, you can pile as much functionality on top as you like by just wrapping the view. Say, for example, you have documents for clients, and you only want each client to be able to see their own documents; the generic object_detail view doesn’t support this, but you can get it easily by wrapping the view:

from django.views.generic.list_detail import object_detail
def secure_create(request, id):
    doc = Document.objects.get(pk=id)
    if not request.user.id is doc.client.id:
            return HttpResponseForbidden
    else:
        return object_detail(request, queryset=Document.objects.all(), object_id=id)

And you’re done. You haven’t rewritten the generic view, so it’s still just as compatible with all other uses as it was before. The generic view is easily extensible and is easily reusable as a building block for more complex functionality.

See the difference?

On July 17, 2006, James Bennett said:

Mark, that’s sort of an edge case and, as you say, should be strictly confined to certain types of internal app communication. Which isn’t the same as a JavaScript helper ;)

On July 17, 2006, Coda Hale said:

James, having to rewrite your helpers all the time means that you’re writing inflexible helpers. From Rails:

 visual_effect(:highlight, ‘mydiv’)

With a duration:

 visual_effect(:highlight, ‘mydiv’, :duration => 2.0)

With a duration and a completion event:

 visual_effect(:highlight, ‘mydiv’, :duration => 2.0,
                                    :complete => visual_effect(:fade, ‘mydiv’))

There are plenty of ways of writing extensible functions; that’s not much of an argument against helpers. Obviously for the complicated stuff you’ll need to dip into Javascript, but I think writing macros for the 20% of Javascript functionality you’ll need most makes a lot of sense.

On July 17, 2006, James Bennett said:

Coda: at some point, the Rails devs had to wire all those options into the helper. And it’s great to have those options if you’re using the helper.

But what happens when you want an option that hasn’t been wired into the helper? Surprise! You get to rewrite it.

Or you could actually just write some damned JavaScript at the start, and build on it as you go. It’s so incredibly easy at this point to pick up Dojo or MochiKit or YUI or Prototype and just run with it that I can’t understand why people don’t.

On July 17, 2006, Scott Raymond said:

at some point, the Rails devs had to wire all those options into the helper. And it’s great to have those options if you’re using the helper. But what happens when you want an option that hasn’t been wired into the helper? Surprise! You get to rewrite it.

Actually, that’s not true, in the case of Coda’s example. Here’s the implementation of Rails’ visual_effect helper. It takes a hash of options and passes them directly on to the JavaScript output. So if you define a custom Scriptaculous effect that takes a new option, the visual_effect helper will support it automatically.

On one hand, I agree with one of your conclusions: just freaking learn JavaScript; you and your apps will be better off for it. But on the other hand, I think helpers have a place. In my experiece, I use JavaScript helpers throughout some big, successful apps — well into production, and they work great.

All of programming is about creating abstractions, which is all the helpers are. The art is simply determining where the abstractions belong. There’s no absolute answer — it’s the job of each developer to think critically about his application and find the right balance of forces. Ol’ Vitruvius’ triangle again: firmitas, utilitas, venustas.

On July 17, 2006, James Bennett said:

So if you define a custom Scriptaculous effect that takes a new option, the visual_effect helper will support it automatically.

In other words, if you write some JavaScript?

Or, if you want a new option on an existing effect, rewrite it?

Sounds like you’ve made my point for me ;)

Deep down in my soul, I believe that JavaScript helpers are a phase. A growing pain, if you will. The odious history of JavaScript has conditioned people to run screaming from it and implement any number of complex abstractions in the name of never having to write JavaScript. But eventually people will grow out of it and realize the advantages of writing Javascript all the way through (the toolkits help a lot with that), and that’ll be the end of that.

On July 17, 2006, Scott Raymond said:

Deep down in my soul, I believe that JavaScript helpers are a phase.

Fair enough. I guess we’ll just have to revisit this thread in a few years and find out who is right… and who is dead. ;-)

On July 17, 2006, Scott said:

I guess we’ll just have to revisit this thread in a few years and find out who is right… and who is dead. ;-)

Inconceivable! Did I just spot another Princess Bride reference on a tech blog?

On July 17, 2006, Alan Green said:

I hear what you’re saying, but… it feels wrong that, on one hand, Django provides so much support for the database models at the backend and the views in the middle, but, on the other, offers (almost) diddly-squat for user interface.

Some bright cookie should make a set of libraries, views, template tags and media and offer them as a downloadable extension. All it needs is someone that (a) knows what they’re doing with Django and Javascript, and (b), despite that, still believes JavaScript helpers are a good thing. ;)

On July 17, 2006, James Bennett said:

Alan, I tend to think that it’s mostly right for server-side frameworks to stay out of the front-end interface business. But beyond that, I don’t think an analogy between pre-packaged front-end components and the existing backend components would hold up; Django provides a lot of useful things to make it easier to write the back end of an application, but you still have to write the application. And similarly, Django provides useful things for the front end (the admin interface, templating, the formfields system) but you still have to write the application.

On July 18, 2006, Gasper said:

Pardon me, since I’m going off topic…

I’ve wondered for a long time … what is the correct way to use caching?

Where to cache, how to cache parts of pages, eg. for busy sites with personalized content where you can’t just cache whole page, since every user has his username on page. What is the best practice, where to cache parts and where not to cache.

I’ve caught some hints how not to cache from Simon Willison’s talk at Google, but I’m still puzzled.

And I’d love to see your article on that subject.

On July 19, 2006, Ahmad Alhashemi said:

Maybe we can solve this problem by providing a very thin and generic layer of Python over JavaScript code. Where every Python line will correspond to a line of JavaScript. It will be like writing JavaScript directly, only with a Python syntax.

More like automatic coce conversion than code generation helpers.

Or maybe we can implement Python in JavaScript then use Python directly in the browsers.

Radical? Yes. Impossible? No. Practical? I don’t know.

On July 19, 2006, Joseph Rose said:

This article’s points about scaffolding:

So scaffolding gives you an incredibly fast path to your first demo and the feedback you’ll need to continue, but after that it’s pretty much useless and needs to be ripped out and replaced.

is incorrect. When one uses a code generator to generate the scaffold, one does not have to rip out the code. One simply extends it.

Example: I recently wrote an admin panel for a classifieds site. I generated the scaffolding for the ‘classifieds’ model, which allowed me to do basic CRUD. I then added security, enumerable types, etc. At no point did I have to actually replace or delete scaffolding code. I simply added to it or modified it slightly.

Consider taking the same path with the Javascript helper tags. I drop a RoR Javacript tag to test functionality and demo to a customer. When I need to extend the tag, my javascript guy does a ‘view:source’ in his browser, copies the generated code out, and then begins extending.

No where in this cycle (for both Scaffolding and Javascript tags) does one just throw away code. Now I accept the fact that in some circumstances, generated code will be removed/thrown away. I just ask that you consider the reverse: Not all generated code is thrown away, and I suspect that a sizable percentage can actually be kept and just extended.

Now, take my one gripe about Django: The admin panel is extremely difficult to extend in certain circumstances. Take, for example, a need to implement ownership. If you want a ‘user’ to only CRUD his own files, how do you add this functionality to the Django Admin panel? Not so simple.

In a scaffold, you simply check the logged in user id versus the object’s associated user id and, because this code was generated for you, but you now own it specifically for this object, none of the changes you make will be reflected anythere else in the admin panel.

On July 21, 2006, Le Roux Bodenstein said:

I agree.. Javascript helpers are just bad. Too few people know how to write proper javascript (in fact - few people realise that it is actually quite an interesting language - see http://simon.incutio.com/slides/2006/etech/javascript/js-reintroduction-notes.html). I think that has to do with the fact that 1) back in the day, javascript did in fact suck, 2) the average web developer knew even less about javascript back then (so there were many sites featureing poorly written code snippets and tutorials) and 3) frameworks provide helpers ;)

Well written javascript libraries make helpers completely obsolete, code generation (the type that writes bits of source code) inevitably comes back to bite you and really - all these examples are just silly - you save almost no typing. The only javascript code generation we need is JSON :)

I think Django shouldn’t provide built-in helpers and they shouldn’t favour a specific javascript library/toolkit. Someone else can make some 3rd party helpers that works prototype/dojo/mochikit/whatever and I think then everyone should be happy.

Helpers (imho) are ugly!

end rant.

On July 21, 2006, James Bennett said:

Joseph:

If your JavaScript guy’s first response to the output of a Rails helper isn’t to rewrite the whole shebang, get a new JavaScript guy. The sheer amount of ugly, old-school inline JS it seems to generate makes me extremely wary.

The Django admin is kind of a tricky case, because it’s not really intended to be an interface for end users who operate under extreme permissions limitations; the fact that the trigger for being able to log in to the admin is called is_staff is, I’ve always hoped, a clue about that.

Nice to see you’re talking about using scaffolding as encapsulated logic, rather than as code generation, though. If you’re talking in my terms, I’ve won without you knowing it ;)

On July 24, 2006, Joseph Rose said:

If your JavaScript guy’s first response to the output of a Rails helper isn’t to rewrite the whole shebang, get a new JavaScript guy. The sheer amount of ugly, old-school inline JS it seems to generate makes me extremely wary.

Cut him some slack, he is a student intern :)

The Django admin is kind of a tricky case, because it’s not really intended to be an interface for end users who operate under extreme permissions limitations; the fact that the trigger for being able to log in to the admin is called is_staff is, I’ve always hoped, a clue about that.

The use we intend for the interface is for staff only, of course. We are a large company, however, and concerns are raised that an ignorant user (we just call them users, here) would post under another person’s name or edit someone else’s work without permission.

I’m actually suprised that Django didn’t take a Unix-like ‘everyone/group/owner rwx’ approach to the admin interface from the beginning. Would’t it just be adding a one-to-one foreign key to a ‘permissions’ object for each model? Sounds simple, but what do I know?

Nice to see you’re talking about using scaffolding as encapsulated logic, rather than as code generation, though. If you’re talking in my terms, I’ve won without you knowing it ;)

Everyone has their slip to the dark side, I suppose…

:)

On July 28, 2006, Paul said:

Your “start out with real code” comment reminds me of the Pragmatic Programmer book and their distinction between prototype code and “tracer bullet” code — the former being something you write to throw out (which one does need to do sometimes); the latter, production code with narrow functionality that you expand later.

The terminology isn’t great, but it’s an interesting distinction.

Comments for this entry are closed. If you'd like to share your thoughts on this entry with me, please contact me directly.

Copyright © 2006-2008 James Bennett. Learn more about the author or about this site. Subscribe to feeds to get updates.