Clarke’s Third Law
Every so often, a TV producer who wants to get ratings will air a “documentary” about some wonder of the ancient world. Say, the great pyramids at Giza. The formula for this “documentary” is pretty simple: you get a bunch of people from modern, industrialized nations to go crawl over these huge ancient monuments in Egypt and speculate on how those ancient Egyptians managed to build them. And, inevitably, a lot of these people will throw their hands up and decide that the Egyptians must have had help from aliens or mystical “pyramid power” or some other magical or quasi-magical force, because the average modern Westerner doesn’t know the first damn thing about how to build a huge pyramid and line it up just right so that it points at Betelgeuse on the Feast of St. Swizzen.
And if it’s a really well-done documentary, there’s a thirty-second segment where somebody shows up with some sticks and string and demonstrates how you can lay out a perfect square of pretty much any size, aligned with pretty much any earthly or astronomical feature you like, and optionally also shows how you can move gigantic rocks around with nothing more than logs and some grease.
Another example comes out of the Nazca Desert in Peru, where there are a bunch of lines scratched into the ground in strange patterns; it’s only when you get up into the air over the desert that you see how the lines actually form gigantic figures in the shapes of animals and people. Crackpot theories about these lines abound: aliens who supervised the designs from their flying saucers are a popular explanation, as are magical flying shamans who could levitate above the desert to make sure the lines were drawn correctly.
But if you avoided the people who inevitably get interviewed for documentaries about the Nazca lines, and went instead to a draftsman or painter who was trained before the age of computers, you’d get a boring explanation of how you can start with a small drawing and scale it up to any size you like, using tools that aren’t complex at all. Back in the 1980s, three guys from Kentucky drew a 440-foot-long Nazca-style condor on the ground using nothing more than ropes, sticks and their brains.
There aren’t a whole lot of professions these days which require this sort of knowledge or thinking; as a culture, we’ve largely forgotten that Archimedes showed how, with a lever long enough and a place to stand, you could move the planet. And even sailors, who used to live and die by applying the principles of one of the classical simple machines, have mostly gotten out of the business of lifting huge things with them. Today they just use cranes.
Which isn’t to say that these things are “simple” in terms of the knowledge you need to apply them. The actual machinery you need to, say, lift a five-ton granite block off the ground is “simple” in that it’s just ropes and pulleys, but the knowledge of math and physics — even if you’re not thinking of it in those terms — required to properly set up and operate those ropes and pulleys is pretty complex; an ancient Egyptian would probably have spent years as an apprentice pyramid-builder in order to learn the ropes (pun intended) and be able to carry on the work. In that sense — the knowledge and understanding required to operate ostensibly “simple” devices — the ancient world had some pretty advanced technology.
Those of us who live in modern, industrial, computerized societies don’t really have a clue anymore about all that advanced technology. We like to think of people like the ancient Egyptians as fairly primitive folks, and we reinforce that idea by latching on to the way they believed that, for example, drying your corpse out in the desert and wrapping it in linen could grant you eternal life. So most of us, when faced with a great damn pyramid looming up out of the sand, get confused and assume they must have used magic.
Which brings me to Arthur C. Clarke, whose Third Law famously states that
Any sufficiently advanced technology is indistinguishable from magic.
What’s important to realize here is that this principle applies to computers just the same as anything else. Computers aren’t really any more “advanced” than the systems used to build the pyramids or lay out the Nazca lines; ultimately, they boil down to really simple parts chained together in complex ways. Just like the wonders of the ancient world, it’s all about having the knowledge and understanding to apply the relatively-simple machinery in just the right way. And, for sufficiently complex knowledge, it’ll end up looking like magic.
Django’s old-school “magic”
If you sit down today, with a recent copy of Django, and set out to define a data model, it’s pretty straightforward. You might represent a blog entry like so:
from django.db import models class Entry(models.Model): title = models.CharField(max_length=250) body = models.TextField() pub_date = models.DateTimeField()
You’d stick that in an application named blog, run syncdb and start adding entries to your database. And when you needed to get them back out again you’d do something like this:
from blog.models import Entry entry_list = Entry.objects.all()
But that’s on a modern copy of Django. At the sprint over the weekend, Jacob gave a short historical talk outlining the evolution of Django’s model syntax, going back all the way to the days when “Django” was a set of modules in our private repository known as “the CMS”. Even those of us who were using Django right after its initial public release have largely forgotten how things used to work: with the blog entry model, for example, you couldn’t do anything so nearly straightforward as from blog.models import Entry. You would have done something more like this:
from django.models.blog import entries entry_list = entries.get_list()
Which immediately raises questions like “why is there a module named entries when I didn’t define one, and how did it get out of the spot where I put my models file and into django.models?” The answer was that Django did an immense amount of under-the-hood work aimed at eventually throwing away the model class you’d defined; instead of using that directly, Django would build one or more entire modules of code — complete with module-level constants, methods and exception classes — and hack Python’s import mechanism to make it live in the django.models namespace.
This was, in Python terms, pretty advanced technology, and even to an experienced Python programmer it was sufficient to be indistinguishable from magic. That’s why we got rid of it and switched to the model system we have now, the one that leaves your model classes right where you put them and which mostly works by having lots of things happen to the base Model class which inherit into its subclasses. The development effort which replaced the old system with the new was called “magic-removal” precisely because we’d gotten to that indistinguishable-from-magic point where even people who really knew Python could be stumped by what was going on inside Django.
Modern magic
Thanks to the magic-removal effort we’ve pretty much gotten rid of the indecipherable and counterintuitive things Django used to do; there’s still a minor amount of magic in how custom template tags work, but it’s not nearly so bad and most people never even notice it (until they have a tag library which raises an ImportError and start wondering why Django thinks the tags should be living in django.templatetags, but that’s a story for another day).
There is, however, a fair amount of reliance on features of Python which aren’t necessarily obvious to someone who’s just starting out with the language. The way we set up the database API methods to accept the right keyword arguments for your model’s fields and relations, for example, must seem awfully strange if you don’t know how Python’s variable argument lists work. And, yes, the way a QuerySet can be sliced, indexed into and iterated over as if it were a list probably seems strange if you don’t know how Python lets any class emulate a sequence or container.
These are somewhat advanced uses of the language; they’re not things you’ll learn in the first five minutes of a Python tutorial, and some people write Python for months or even years without ever noticing these features or knowing how they work. They’re a form of moderately advanced technology, in other words, and so it’s tempting sometimes to label them “magic”.
But, just as you can lay out a pyramid or a 400-foot condor using nothing but ropes and sticks, or move a multi-ton block of stone with an arrangement of pulleys, the trick here is not in the complexity of the technology: it’s as simple as accepting **kwargs in your argument list and inspecting the resulting dictionary, or implementing a few specially-named methods on a class. The complexity here is knowledge and experience of working with Python; just as anybody could have gone through an apprentice-pyramid-builder stage and come out of it knowing how to lay out the structure and move the rocks, anybody can go through an apprentice-Python-programmer stage and come out knowing how to make a function take any arguments you want to throw at it, or how to make a class behave like a list.
And so calling it “magic” really is about as accurate as calling the pyramids or the Nazca lines “magic”; if you put in the time to learn how it works — and in the case of Python, you don’t need to spend years as an apprentice to do this, since the technology doesn’t require a whole lot of knowledge and is pretty well-documented — it’ll make sense and, like the guy on Fox’s When Pyramids Attack who’s just seen a perfect right angle laid out with a few pieces of twine, you’ll wonder how something so simple could ever have seemed so complex.