Rambles around computer science

Hacking versus programming

When I was an undergrad I used to consider myself a pretty good programmer. By the time I started at the Lab as an RA I had over 18 months' full-time professional experience in software development, as well as lots of hobbyist stuff when I was younger, and I'd felt that I was at least as good a programmer as most of the people I'd worked with. However, pretty much the moment I started at the CL, and was faced with the prospect of Xen development, I realised two things.

Firstly, apparently unlike many, I didn't delight in programming for its own sake. I hated being told to “just go and play with [some code]” without a problem to solve. To be motivated, I needed a real objective, and one which I had a clue about how to make progress towards. Secondly, I was suddenly and severely short of the skills I needed to make progress in my work. Since for most of my RA year, the only brief I had was vague and mountainous (“implement some graphics virtualisation for Xen”), and doing even the simplest things with Xen seemed like a titanic struggle, motivation was sorely lacking for most of my time as an RA. Things did pick up eventually, as I learnt to set my own objectives, build up my skills and, in truth, avoid Xen wherever possible, but it was too late to achieve anything particularly worthwhile.

Now, of course, I realise that I wasn't lacking in “programming” skill, but that there is an entirely different set of skills that I did lack: “hacking” skill. I've subsequently spent large parts of my PhD time, as well some of the earlier RA time, acquiring these skills, and they are definitively not the same as programming skills. Most programming, according to introductory texts and (it seems) much professional development practice, involves starting with a concrete brief, and designing implementing a bunch of code more-or-less from scratch, using languages and tools over which one has some sort of choice. Hacking, by contrast, involves taking a vast mound of already-implemented code, and making some relatively small, clean changes to make it do new stuff.

Hacking is a very useful skill, of course, because you don't start from scratch. The experienced hacker can achieve great results in relatively modest time. However, it takes lots of experience to be able to do anything: experience with the wide variety of programming styles, languages, tools, build systems and version control systems which your target codebase might use; experience working with other people's codebases, knowing how to find the relevant parts while ignoring the irrelevant parts; knowing where or how to find other code to base your attempts on; finding shortcuts by adapting someone else's already-implemented solution to your problem or a similar one.

A first observation: hacking is very suboptimal in software engineering terms, because it takes smart and/or experienced people to do -- not just to do well, but to do at all in any realistic sense. It also tends to yield awkward outputs: patches that rot, forked codebases, copy-and-paste jobs. Given that hacking is essentially a compositional process -- making new code to work with old code -- there's clearly a need for better compositional tools and infrastructure. And, fortunately, that's exactly what I'm working on for my PhD.

The second observation: it would appear that hacking is only learnable by doing. But, despite the lazy conclusion drawn by many, that doesn't mean that we can't make any effort to help people learn it! (I say “help learn” and leave it up to you whether that counts as “teaching” -- though I'd say it's the very best kind of teaching there is.) At the very least, you can help people with time-distilled hints and tips, by giving them a more experienced buddy to work with, and by building a structured path of successively harder problems with which to build their competence and confidence.

So, with this in mind, could we, or should we, aim to teach incoming young researchers how to hack? Of course, we already make sure they can program, but there seems to be a wide variety of hacking experience -- at least inasmuch that I didn't have much of it when I started, but that didn't make me a bad candidate. I suppose this is what Master's degrees are for, at least in systems research, so it's rather a shame we don't have one here at Cambridge. (Don't get me started on Part II projects, which are definitely no substitute.) Fortunately though, this will probably be fixed within the next couple of years -- progress at last.