Hackers and Fighters

© Dr Mark Tarver, 2007
Ars Programma and Lambda Associates

This essay was prompted by a post on comp.lang.lisp by Joe Marshall.

Some of the people in this group have, or are pursuing, advanced degrees in computer science. Some have bachelor's degrees. Some have degrees in related fields. Some have degrees in completely unrelated fields. Some have dropped out before getting a degree. Some are autodidacts.

Is there a correlation between studying computer science and knowing about computers? Do people gain anything by pursuing an advanced degree in computer science, or would they be as well served by saving the money, buying a PC, and teaching themselves? There is a romantic ideal of the untutored hacker that program circles around MIT grad students. Is that absurd or commonplace?

This reminds me of another debate which takes place in martial arts - about martial arts students and street fighters. Geoff Thompson touched on this in his excellent and extremely funny book 'Watch My Back'. Geoff is a trained martial artist, a karateka, a grappler and the victor in over 200 street fights as a bouncer.  In his book he compares two styles of fighter - the school-trained martial artist and the untutored but hardened street fighter.

The street fighter fights in a ring with no rules and no boundaries.  The school-trained martial artist trains for a circumscribed contest with known parameters. What Geoff says is roughly this: the street fighter, if transplanted to the tournament ring, will often fare poorly.   Contrariwise the trained martial artist in no-holds fighting will generally be taken out by a good street fighter unless his level of skill is very high and he has trained for combat situations. However, the combination of the two, the street fighter who has also formally trained in the dojo, will surpass either.

This is also true for programmers.  There are a number of people who have a natural gift for programming and just pick it up; in the same way that some people have a gift for fighting and pick that up.    As programmers they will surpass the average level ability CS student in a straight programming contest.  These naturals may not have passed through a CS course or sat a single CS lecture, but they can walk the walk and do the job.   These people we can call 'street programmers' - hackers without any formal qualifications, who have picked up their knowledge on the fly and through self-teaching, but who can blow most competitors away when it comes to code productivity.

I started my career as a street programmer. In 1984 my father brought round a BBC model B microcomputer. Running with 32K of main memory, I learned to program in BBC Basic. Armed with this slender knowledge, I talked my way into a small software company and was given 'in-house training'. My in-house training consisted of a DBase II manual and a coffee machine. My immediate boss was in charge of a pile of flakey in-house legacy DBase II systems which they wanted to get rid of but could not. My job, for a princely £5000 per year was to keep this mouldering pile of legacy code upright.

What I learnt in this job were street programmer's skills. Being able to digest a manual without any lecture course to help me. Being able to dive into a pile of manky code and come up smelling sweet. Being able to locate and fix an errant bug before the client blew a fuse. Also being able to take a drink at lunchtime and still be useful in the afternoon (a challenge this one). It was high-pressure and low pay and I quit after a few months, just before in fact, I would have got a raise. But it was a useful in teaching me what is required in a commercial software house. After that I migrated towards computer applications at university and there it was a very different story.

In academia you do not get promoted for being a good hacker but for writing papers and getting grants.    There are even professors who actually look down on implementors as blue-collar workers who exist to implement the grand designs of people like themselves. Typically coding is thought of as the preserve of post-doctoral RAs. Hence the metric by which street hackers assess each other does not function here. During my time at the LFCS, Rod Burstall actually felt impelled to give a lecture to a bunch of recycled mathematicians doing CS on why implementors were important .   They were not interested - as one put it "Hacking code is not why we're here."

Coming into academic computing from street programming, I was unimpressed by this attitude. It took me several years and several dead end turns before I came to accept the idea that actually getting to grips with the theoretical end of CS was well worthwhile. Like many streetwise programmers I had become dependent on and over-confident of, my ability to solve problems by simply improvising Lisp code on the computer. I was a rough and tumble guy who would mix it with the biggest problem sans textbook. No problem was too big or tough that my street programming skills couldn't take it out.

Except that wasn't true. As my ambitions grew, I grew to recognise that metal-bashing programming was not going to take out all of my problems and that by importing a little of the conventional wisdom from the learning of the greats, I could improve my problem-solving skills. After all, who wants to reinvent quicksort? So I could be seen at the terminal with a CS book (generally AI) paging the text and pouring out reams of Lisp. Theory and practice were melded together. I was becoming not a mere brawler, but that more potent combination of theorist and bareknuckle code monkey. I was transforming, day by day, into the programming equivalent of what Geoff Thompson has described as the most potent fighter - the theory-hardened street fighter.

In following this path I was lucky. I was lucky to have the time to do it. I was lucky not to be weighed down with the nonsense of a conventional CS degree and I was lucky because I was using Lisp. And I was following a path that was well trodden because the best people in CS have followed exactly this path. These are the people who ride above the divide; who have a grasp of theory and make it work on the computer.  

If you check out the history of computing, a lot of the old aces in artificial intelligence (like Newell and Simon of GPS fame), were accomplished street programmers. There's a reason for that. The reason is that AI programs are difficult to predict and often the only way to tell if your program works is to run it and see. Hence you need to have both theory (a clear idea of what you are doing) and implementation skills if you want to do AI. The old timers of CS were physicists, mathematicians and what-have-you (Simon was a political scientist) who were accomplished street programmers. Joe wonders whether such people are romantic ideals. They were not romantic ideals, they were real living, breathing people; and many of them are still around.

Officially these old timers are the kind of persons professors are producing from CS courses; people well grounded in theory and practice.  However it does not work like that. The martial arts analogy comes back to mind again.  

If you train in martial arts, you'll find there are some schools of martial arts which are essentially empty.  These are schools where the students train in katas which have to be memorised with little or no understanding of application.   These schools avoid subjecting their students to any combat situations and emphasise artifical grading systems with little relationship to combat effectiveness.    The graduates of these schools are not a match for the experienced street fighter.

Unfortunately a lot of CS departments are like these empty dojos. In CS a lot of student time can be taken up in courses which require little or no programming expertise.  In the UK it is quite possible to walk off with a CS degree and be quite hopeless at programming.  Why is this?   For several reasons.

First, a pick-'n-mix approach to education which allows weak programmers to avoid programming-intensive courses.

Second, CS courses at university are rarely bleeding edge.     Staying ahead of the game, knowing all the latest Java tools, being up at 1 a.m. on forums swapping information with other street programmers, is not what professors want to do after they go home at 6 p.m.    With the weight of admin, teaching, reading and scribbling papers to be done there is simply not the time for the street programmer's lifestyle.   So students learn what the professor feels comfortable with or get palmed off on an instructor.   And people who do know the bleeding edge can get better money (and often better conditions) in industry. You can graduate from these courses just by jumping through the hoops. Unfortunately a lot of very bright students become disenchanted with this type of course.

Third, is the conservatism of CS courses that comes from always needing a consensus to originate anything.    A good example of this came up in my old department in the early '90s. Back then the language of choice for teaching students was Pascal. Nearly all the staff knew Pascal had to go - it was out of date.  I attended a meeting in which computer scientists chased the question of how to replace Pascal.   Several hours later Pascal was still on the curriculum because they could not get a consensus.   Seven years later I attended a meeting on the same subject with the same lack of agreement and the same arguments. It took eight years to move from Pascal to C++. Trying to do anything innovative in teaching at University or instituting change is a gruelling business.   If your course is foundational then your teaching is tied to the courses which list your course as a requirement .  Try to do anything radical, even a change of approach, and you can be in trouble with the Old Guard.  I know; I learned the hard way.

This resistance to change was always part of the university character (and human nature!) because universities have been, for the greater part of their history, repositories of the Conventional Wisdom. The material that got taught there was carefully filtered through a long laborious process of peer review, starting with workshops, conferences, and finally into book form. It took about 6 years for Prolog to move from research paper level to the point were there was a useful introductory book (Clocksin and Mellish's 1981 text Programming in Prolog). Universities are not places for learning the latest thing, but places for learning what has already passed into received opinion.The advantage of this traditional model is that it filters out the bogus and the weak. The disadvantage is that much that is new, useful and exciting is not transmitted and for a fast-moving discipline like computing, the model seriously sucks.

Traditionally universities had a mechanism for offsetting the ossifying effects of traditionalism. This powerful counter-potion was called academic freedom and it gave the professor the right to lecture on what interested him. So wonderful and theatrical lecturers like Feynmann and Wittgenstein gave sparkling lectures on physics and philosophy. Wittgenstein's Cambridge lectures became historically famous and created a whole school of brilliant post-war philosophers.

Sadly universities have lost this aspect at the very time when they most have need of it. Wittgenstein today would be simply out of a job. Courses are centrally planned and bureaucratically delivered - often from canned standard texts. Teaching loads at the junior end are heavy and the only way to deliver five or six courses a year (the standard in many American colleges) is to reach for the shelf and open a canned course with canned slides and a set book. Professors do not have the time for their students.

And all this is a damned shame, because, fundamentally, learning and discovery is a social process. To rely purely on a language manual and an idea is to ignore all the painful learning processes that previous computer scientists went through in order to arrive at their state of understanding. It invites the repetition of all the past mistakes and the reinvention of old ideas. Unfortunately the state of university CS departments is such that students would be better off in my situation - having a computer and a good book and the time to do their own thing.

So to answer the post: yes, the street programmer is generally better than the CS graduate - at programming because CS courses are often so woeful.  And the professor will beat the street programmer hands-down in the black arts of churning out papers.  But in the CS departments which are not empty dojos, the graduates who have absorbed the wisdom of their intellectual fathers, and learnt to apply it under pressure will surpass the street programmer.