Every so often, a TV producer who wants to get ratings will air a “documentary” about some wonder of the ancient world. Say, the great pyramids at Giza. The formula for this “documentary” is pretty simple: you get a bunch of people from modern, industrialized nations to go crawl over these huge ancient monuments in Egypt and speculate on how those ancient Egyptians managed to build them. And, inevitably, a lot of these people will throw their hands up and decide that the Egyptians must have had help from aliens or mystical “pyramid power” or some other magical or quasi-magical force, because the average modern Westerner doesn’t know the first damn thing about how to build a huge pyramid and line it up just right so that it points at Betelgeuse on the Feast of St. Swizzen.
And if it’s a really well-done documentary, there’s a thirty-second segment where somebody shows up with some sticks and string and demonstrates how you can lay out a perfect square of pretty much any size, aligned with pretty much any earthly or astronomical feature you like, and optionally also shows how you can move gigantic rocks around with nothing more than logs and some grease.
Another example comes out of the Nazca Desert in Peru, where there are a bunch of lines scratched into the ground in strange patterns; it’s only when you get up into the air over the desert that you see how the lines actually form gigantic figures in the shapes of animals and people. Crackpot theories about these lines abound: aliens who supervised the designs from their flying saucers are a popular explanation, as are magical flying shamans who could levitate above the desert to make sure the lines were drawn correctly.
But if you avoided the people who inevitably get interviewed for documentaries about the Nazca lines, and went instead to a draftsman or painter who was trained before the age of computers, you’d get a boring explanation of how you can start with a small drawing and scale it up to any size you like, using tools that aren’t complex at all. Back in the 1980s, three guys from Kentucky drew a 440-foot-long Nazca-style condor on the ground using nothing more than ropes, sticks and their brains.
There aren’t a whole lot of professions these days which require this sort of knowledge or thinking; as a culture, we’ve largely forgotten that Archimedes showed how, with a lever long enough and a place to stand, you could move the planet. And even sailors, who used to live and die by applying the principles of one of the classical simple machines, have mostly gotten out of the business of lifting huge things with them. Today they just use cranes.
Which isn’t to say that these things are “simple” in terms of the knowledge you need to apply them. The actual machinery you need to, say, lift a five-ton granite block off the ground is “simple” in that it’s just ropes and pulleys, but the knowledge of math and physics — even if you’re not thinking of it in those terms — required to properly set up and operate those ropes and pulleys is pretty complex; an ancient Egyptian would probably have spent years as an apprentice pyramid-builder in order to learn the ropes (pun intended) and be able to carry on the work. In that sense — the knowledge and understanding required to operate ostensibly “simple” devices — the ancient world had some pretty advanced technology.
Those of us who live in modern, industrial, computerized societies don’t really have a clue anymore about all that advanced technology. We like to think of people like the ancient Egyptians as fairly primitive folks, and we reinforce that idea by latching on to the way they believed that, for example, drying your corpse out in the desert and wrapping it in linen could grant you eternal life. So most of us, when faced with a great damn pyramid looming up out of the sand, get confused and assume they must have used magic.
Which brings me to Arthur C. Clarke, whose Third Law famously states that
Any sufficiently advanced technology is indistinguishable from magic.
What’s important to realize here is that this principle applies to computers just the same as anything else. Computers aren’t really any more “advanced” than the systems used to build the pyramids or lay out the Nazca lines; ultimately, they boil down to really simple parts chained together in complex ways. Just like the wonders of the ancient world, it’s all about having the knowledge and understanding to apply the relatively-simple machinery in just the right way. And, for sufficiently complex knowledge, it’ll end up looking like magic.
If you sit down today, with a recent copy of Django, and set out to define a data model, it’s pretty straightforward. You might represent a blog entry like so:
from django.db import models class Entry(models.Model): title = models.CharField(max_length=250) body = models.TextField() pub_date = models.DateTimeField()
You’d stick that in an application named blog, run syncdb and start adding entries to your database. And when you needed to get them back out again you’d do something like this:
from blog.models import Entry entry_list = Entry.objects.all()
But that’s on a modern copy of Django. At the sprint over the weekend, Jacob gave a short historical talk outlining the evolution of Django’s model syntax, going back all the way to the days when “Django” was a set of modules in our private repository known as “the CMS”. Even those of us who were using Django right after its initial public release have largely forgotten how things used to work: with the blog entry model, for example, you couldn’t do anything so nearly straightforward as from blog.models import Entry. You would have done something more like this:
from django.models.blog import entries entry_list = entries.get_list()
Which immediately raises questions like “why is there a module named entries when I didn’t define one, and how did it get out of the spot where I put my models file and into django.models?” The answer was that Django did an immense amount of under-the-hood work aimed at eventually throwing away the model class you’d defined; instead of using that directly, Django would build one or more entire modules of code — complete with module-level constants, methods and exception classes — and hack Python’s import mechanism to make it live in the django.models namespace.
This was, in Python terms, pretty advanced technology, and even to an experienced Python programmer it was sufficient to be indistinguishable from magic. That’s why we got rid of it and switched to the model system we have now, the one that leaves your model classes right where you put them and which mostly works by having lots of things happen to the base Model class which inherit into its subclasses. The development effort which replaced the old system with the new was called “magic-removal” precisely because we’d gotten to that indistinguishable-from-magic point where even people who really knew Python could be stumped by what was going on inside Django.
Thanks to the magic-removal effort we’ve pretty much gotten rid of the indecipherable and counterintuitive things Django used to do; there’s still a minor amount of magic in how custom template tags work, but it’s not nearly so bad and most people never even notice it (until they have a tag library which raises an ImportError and start wondering why Django thinks the tags should be living in django.templatetags, but that’s a story for another day).
There is, however, a fair amount of reliance on features of Python which aren’t necessarily obvious to someone who’s just starting out with the language. The way we set up the database API methods to accept the right keyword arguments for your model’s fields and relations, for example, must seem awfully strange if you don’t know how Python’s variable argument lists work. And, yes, the way a QuerySet can be sliced, indexed into and iterated over as if it were a list probably seems strange if you don’t know how Python lets any class emulate a sequence or container.
These are somewhat advanced uses of the language; they’re not things you’ll learn in the first five minutes of a Python tutorial, and some people write Python for months or even years without ever noticing these features or knowing how they work. They’re a form of moderately advanced technology, in other words, and so it’s tempting sometimes to label them “magic”.
But, just as you can lay out a pyramid or a 400-foot condor using nothing but ropes and sticks, or move a multi-ton block of stone with an arrangement of pulleys, the trick here is not in the complexity of the technology: it’s as simple as accepting **kwargs in your argument list and inspecting the resulting dictionary, or implementing a few specially-named methods on a class. The complexity here is knowledge and experience of working with Python; just as anybody could have gone through an apprentice-pyramid-builder stage and come out of it knowing how to lay out the structure and move the rocks, anybody can go through an apprentice-Python-programmer stage and come out knowing how to make a function take any arguments you want to throw at it, or how to make a class behave like a list.
And so calling it “magic” really is about as accurate as calling the pyramids or the Nazca lines “magic”; if you put in the time to learn how it works — and in the case of Python, you don’t need to spend years as an apprentice to do this, since the technology doesn’t require a whole lot of knowledge and is pretty well-documented — it’ll make sense and, like the guy on Fox’s When Pyramids Attack who’s just seen a perfect right angle laid out with a few pieces of twine, you’ll wonder how something so simple could ever have seemed so complex.
Comments for this entry are closed. If you'd like to share your thoughts on this entry with me, please contact me directly.
This was a beutifull post! Thanks to put evrything in perspectif. Very philosophical
Wow. Nice post. I must admit I’m usually turned off by “magic” in frameworks b/c when you get into the more advanced portions, it gets really weird. Nice to know Django is moving away from that. Thanks!
Another fantastic post. If you want to be good at your trade… you need to know your tools :)
Way to turn what could have been a boring 2 paragraph soap box into a far more entertaining read.
In case anyone wondered: James studied philosophy in school.
Nice post. :)
Do they still teach kids what simple machines are in public schools? Or do we just start them out with PCs now? When I was in 6th grade, we had to make complex machines out of just the three simple ones (lever, inclined plane, pulley). That kind of knowledge is useful as you go through life, even if only for a subject to draw awesome analogies from ;-)
Excellent post James.
Great post, “magic” is a weakness of many frameworks and it was interesting to read how you got rid of it in django.
Yeha, i note that “way” to inform the mass public about the ancient history. The problem is that nobody tell the big amount of imagination required to draw the conclusions on that shows.
For example, anything about pre-historic is almost pure invention. Nobody there, not write testiomonials and the drawings in caves are so bad that for sure the guy was on crack ;).
Or the actual interest in re-write the historical events, from the Bible or Egypth or anything else… that it, rewritend with a full of magic too ;)
I think is the human nature not-stop in face of ambiguity. We need teach the zend of python to everyone! So when we simply not know, and can’t use any past reference (a writed myth could be more usefull than no more that a bunch of objects in bad conditions), then the show could be shorter, to the point and cost a lot less more:
Start: How the egyptians build giza? Answers: We surely not know. We have a lot of crazy ideas and a couple of samples in how we could build a giza now. But for sure, the old people know and that mean their are smart. We, instead can talk about all dimensions, shapes and materials used but none important as why & how. Ending: Tanks to see a real historic lesson. Watch next episody: Why everything you know about dinosours is as reliable as a hollywood movie!
Sometimes, I wonder if that object that the “expert” tell us was used for ritual not was a simple toilet!
;)
In the other hand, is great to see how django is build under a very mature fundation. That is the thing that help me going further when I face a problem, I see the code is a bless. In fact, I do a sql 2000 port in no time thanks to that - and that requiere a lot of magic, but help A LOT when the “egypcians” that build that “giza” are still alive-
I’ve never understood why it’d be so difficult to build those pyramids. It must have been alot of work and required alot of people, but I don’t think the builder had to worry about any of those things. And they did have unlimited amount of sand available. They could have just buried the pyramid as they built it and then removed the sand when the structure was in place, couldn’t they? Incidentally, that seems to resemble the old “Django Magic” that you talked about in your article. :)
Oh, and I’m just starting out with Django and Python after working with Java for ages. It truely feels like magic. And speaking of magic and flying people: http://xkcd.com/353/
cheers,
Jo-Erlend Schinstad
I find one good way to unravel the magic is to just face your fears and actually Google for something like kwargs or meta the first time you see it, rather than sweep it under your mental rug and assume you don’t need it because it scares you.