Separating Programming Sheep from Non-Programming Goats

I found this blog entry to be very interesting. It’s about a paper that has been written on distinguishing those who can program from those who can’t. (The blog entry contains a link to the paper itself, too, which is here.)

I should say that the intent of the paper is not elitist. “We can program, and you can’t, so don’t even bother to try.” But programming is something that people seem to have either a natural ability for, or… well… don’t. As the authors of the paper say:

Learning to program is notoriously difficult. A substantial minority of students fails in every introductory programming course in every UK university. Despite heroic academic effort, the proportion has increased rather than decreased over the years. Despite a great deal of research into teaching methods and student responses, we have no idea of the cause.

It has long been suspected that some people have a natural aptitude for programming, but until now there has been no psychological test which could detect it. Programming ability is not known to be correlated with age, with sex, or with educational attainment; nor has it been found to be correlated with any of the aptitudes measured in conventional ‘intelligence’ or ‘problem-solving-ability’ tests.

That quote is followed closely by this sentence, which I rather enjoyed:

We point out that programming teaching is useless for those who are bound to fail and pointless for those who are certain to succeed.

The authors talk about three significant hurdles that students encounter, when learning to program: assignment and sequence; recursion/iteration; concurrency. They then go on to say this, which I found very interesting:

…. Assignment and sequence … hardly look as if they should be hurdles at all: storage of / remembering information and doing one thing after another are part of everyday patterns of life and thought, and you might have expected (as at first do most teachers) that students’ experience could be analogised into some kind of programming expertise. Not so: it is a real hurdle, and it comes at the very beginning of most programming courses.

I’m sure, when I was first learning to program, that I became frustrated with fellow students, when they couldn’t figure out how assignment works. I guess this paper explains both sides of the coin: why my fellow students couldn’t get it, and why it seemed so obvious to me.

But enough preamble. Let’s talk about the test itself. It consisted of questions such as the following (click to see a bigger version):And the authors found these results:

We could hardly expect that students would choose the Java model of assignment … but it rapidly became clear that despite their various choices of model, in the first administration they divided into three distinct groups with no overlap at all:

• 44% used the same model for all, or almost all, of the questions. We call this the consistent group.
• 39% used different models for different questions. We call this the the inconsistent group.
• The remaining 8% refused to answer all or almost all of the questions. We call this the blank group.

We did not interview our subjects to determine anything about their group membership, so we do not know whether students chose consciously or unconsciously to follow one strategy or another, nor how conscious choices (if any) were motivated, nor what any particular choice meant to a subject who made it. We have no information about how group membership correlates with earlier education, employment experience, age, sex, marital status or indeed anything else.

The great thing about the study is that the test was administered twice: Once before the students had had any computer programming studies, and once after they had completed their first course. (The authors aren’t really sure whether it was planned to give the test prior to instruction, or if it was a happy accident.) They found that the results were fairly consistent: People who fell into one of the three groups before taking the course tended to fall into the same group after taking the course.

Interestingly, the authors then talked about using these groups to predict who would be good programmers, and who wouldn’t. I would have expected that the “consistent” group would be the most successful programmers, but I would have been in the minority:

Speculation is part of science, though, and those to whom we have recounted this tale have usually been ready to try it. Told that there were three groups and how they were distinguished, but not told their relative sizes, we have found that computer scientists and other programmers have almost all predicted that the blank group would be more successful in the course exam than the others: “they had the sense to refuse to answer questions which they couldn’t understand” is a typical explanation. Non-programming social scientists, mathematicians and historians, given the same information, almost all pick the inconsistent group: “they show intelligence by picking methods to suit the problem” is the sort of thing they say. Very few, so far, have predicted that the consistent group would be the most successful. Remarkably, it is the consistent group, and almost exclusively the consistent group, that is successful. We speculate on the reasons in section 7.

And what do they feel are the results of this study? First of all, they feel that they have a predictive test, as to who will be a good programmer, and who will not.

[One of their graphs] suggests that it is extremely difficult to teach programming to the inconsistent and blank groups. It might be possible to teach them, if we concentrated on trying to persuade them to see a programming language as a system of rules (though the evidence in section 6 below seems to suggest otherwise).

The consistent group seem to be much easier to teach. We speculate that they divide into two groups: the very successful ones find programming easy and may perhaps be those who ought to follow a conventional computer-science education, with lots of formal content; the moderately successful perhaps are the software engineers, those who can program but can’t imagine that they will ever enjoy it, and are content to submit to management discipline and to be drowned in UML (ugh!).

As a side-note, why does everyone hate UML?

I don’t know how scientifically accurate this study will turn out to be, or if the authors will attempt to try it in a larger experiment with more subjects, but I found it interesting nonetheless.