What do programmers really need to know?

(Cross posted at my java.net blog)

Joel on Software recently posted the following observation:

"...there's nothing hard enough about Java to really weed out the programmers without the part of the brain that does pointers or recursion..."

"But beyond the prima-facie importance of pointers and recursion, their real value is that building big systems requires the kind of mental flexibility you get from learning about them, and the mental aptitude you need to avoid being weeded out of the courses in which they are taught. Pointers and recursion require a certain ability to reason, to think in abstractions, and, most importantly, to view a problem at several levels of abstraction simultaneously. And thus, the ability to understand pointers and recursion is directly correlated with the ability to be a great programmer."

I respect and admire Joel, and he has been hiring programmers for years, so I don't doubt his asserted correlation between pointers, recursion and great programmers... But....

There's a lot of lousy software "out there".

That's not a fair or precise statement, so let me retract it and offer the following:

A lot of software "out there" is great, but it contains pervasive security flaws.
A lot of software "out there" is great, but it is way too expensive to maintain.

Here's some supporting evidence:

• Flaw in Microsoft's Windows Meta File Vulnerability in Graphics Rendering Engine Could Allow Remote Code Execution announced December 28th, 2005
  
• Microsoft Internet Explorer PNG Image Rendering Buffer Overflow Vulnerability released June 16, 2005.
  
• Firefox and Mozilla IDN buffer overflow security issue disclosed on September 8th, 2005.
  
• Annual software maintenance cost in USA has been estimated to be more than $70 billion.
  
• The Cyber Security Bulletin 2005 Summary lists 5,198 reported security vulnerabilities last year.
  

What's up with this? Most of this software is not written in Java, so presumeably the authors learned about pointers and recursion.

Maybe mastering a "hard" programming language isn't much of a factor after all.

What do programmers really need to know in order to produce exceptional software?

I agree with Joel that the abilility to "to view a problem at several levels of abstraction simultaneously" is very important... But...

I think that really exceptional programmers have the ability to focus on concrete requirements. Programs are written to solve concrete problems. Abstractions have a way of taking on a life of their own, and that can lead to software that is hard to maintain when a "concrete" requirement changes.

I'll admit that it is a stretch, but even "buffer overflow" problems can be characterized a symptom of misunderstanding a basic concrete requirement (the program should gracefully handle unexpected input).

In some instances, the overhead associated with bounds checking allocated memory is unacceptable, but it is hard to see how a browser would need performance optimized to such a level.

One way to keep focused on the concrete requirements is to test compliance with those requirements all throughout the development life cycle. If you start with a test plan, you'll focus on passing the test.

The great programmer may be the guy who is most adept at writing the "right" test plan (You don't want to waste time passing meaningless tests).

Maybe teaching Test Driven Development is as important as teaching pointers and recursion?

What do you think programmers need to know?