Why is so much software so bad?

(Cross posted at my java.net blog)

Why is so much software so bad?

I've read or heard this question thousands of times, and I repeatedly ask this question myself (I am sometimes embarrassed to be a programmer when my non-techno-friends struggle with bad software). Robert Martin's blog entry on No Next Big Thing bemoans the sorry state of software quality, and prompted me to ponder this question again.

James Gosling's blog entry on Sharpening the Axe includes a confession that is a telling clue in our search for some of the causes for bad software. In Gosling's words:

"Often tool building is far more fun than actually doing the job at hand."

Many of the great programmers that I know are more passionately curious about building tools than about using tools. Actually doing the job at hand is sometimes an afterthought... and that can lead to bad (over-engineered/convoluted/overly complex) software.

Note: I will be using the word "tools" very broadly; "scaffolding" and "frameworks" are tools in the context of this blog entry.

Tools fall into two broad categories:

1. Tools that are built for the tool builder's use
   
2. Tools that are built for someone else to use
   

James' blog entry talks primarily about tools that he needs (or wants) to accomplish a specific task. Frequently these tools are disposable. It would be great if we honed these "I need it now" tools to a fine edge and kept them stored in our toolkits for future use, but that doesn't happen very often.

We tend to crank out "I need it now" tools with little thought towards "I'll need it again".

Corrective Behavior: When you find yourself tempted to build an "I need it now" tool, first ask yourself: "Will I need it again?"

If the answer is no, don't spent much time or effort on the tool.
Building any tool is a distraction from implementing the functionality that you've been asked to deliver. Bad software can result from spending too much time in preparation, and not enough in execution... it's a balancing act.

If the answer to the question "Will I need it again?" is yes, then you may be justified in spending more time and effort on the tool. I say "may be justified" because there are a million and one caveats to consider. If you will need a tool again, then someone else may have already created a tool that you could use for the same purpose. Because we like to build tools, we don't always check around to find out if a tool is already available.

Corrective Behavior: Before building a tool, look for a similar tool.

Actually, you should get into the habit of looking for tools long before you need them. Outside the world of software, I am a hardware store junkie. I love to browse the Harbor Freight catalog and prowl the aisles of Home Depot. My wife has learned not to roll her eyes when I rave about gadgets for driving screws at right angles or hanging a picture at a 32.7 degree angle. Most of the tools that I come across I will never need... but every so often I save my self a lot of time and effort because I knew that a tool existed to accomplish a specific task.

If you can't find a pre-existing tool and you do decide to build an "I'll need it again" tool, then you really have to start thinking about usability and reusability.

Tools that you will need again begin to resemble tools that you build for someone else to use. You have to start taking into account the learning curve needed to master the tool. You have to take into account the ways in which the tool can be misused. You have to start thinking about incorporating user feedback into your process for refining and maintaining the tool (With my fading memory, I won't remember building the tool, so when I need it in a few months I too will be a "first time user").

Corrective Behavior: Before finalizing a tool's design, explain to someone else how to use it.

The perceived quality of a tool has as much to do with how you use the tool (the tool's interface) as with how well it is built. You can craft a widget out of titanium alloy, but if the users can't figure out where to hold it they won't use it (and they'll probably tell others that you built a bad widget).

Many of the interfaces that we come up with for our tools are overly abstracted. We genericize our tools to the point of hyper-configurability. Configurability is great, but when we configure something wrong the result is often buggy software. Every layer of indirection between a requirement and the code that implements the requirement is an opportunity for introducing errors.

Much of the bad software out there could be due to layer upon layer of slightly mis-configured tools.

Corrective Behavior: When building a configurable tool, spend time to simplify the configuration process.

Recently XML configuration files have become all the rage to the point where I am beginning to wonder if it is possible to write and deploy a Java application without authoring an XML configuration file. It's always going to be a judgment call with regards to what should be configured via XML, what should be configured via annotations, and what should be configured via customized code. Whatever your choice, you must consider the likelihood of configuration errors and take steps to prevent, detect, and correct them. Often times this will require the creation of a supporting tool. Often time this leads to the cycle of building tools to build tools that build tools. If any of the tools is buggy, the end-user product may well be bad software.

At this point I need to back-track and loudly assert that the tool-building related factors that I have discussed are not the primary reasons that a lot of software is bad. Shoddy requirements and poor management are certainly bigger culprits in many instances... but we always need to look at ourselves as part of any solution.

While writing this blog entry, I started thinking about professional standards and a professional code of conduct. Using Google, I did a quick search on "code of conduct" and came up with a link to the following: ACMCode of Ethics and Professional Conduct.

The ACM's code is lofty and noble in it's tone, and seems to borrow a bit from the Hippocratic Oath.

I am going to digress from my tool-building-centric theme and lift my final Corrective Behavior directly from the ACM code:

"Quality professional work, especially in the computing profession, depends on professional reviewing and critiquing. Whenever appropriate, individual members should seek and utilize peer review as well as provide critical review of the work of others."

Sounds like a good topic for a future blog entry....

Bestselling Business Ethics Books

Privately famous

(Cross posted at my java.net blog)

Most folks are familiar with Andy Warhol's prediction: "In the future everyone will be world-famous for 15 minutes". My 15 minutes have come.

I was at the express checkout (10 items or less) of the the HEB on Red River (that's a supermarket near UT here in Austin) when a young man (whose name turned out to be Ben) approached me:

Ben: Excuse me sir, I know this is going to sound strange, but are you a programmer?

John: (caught slightly off guard) Why, yes I am.

Ben: Are you a java.net blogger?

I'm not sure what exactly I said at that point... but it had something to do with my wonderful java.net mugshot and my recent blog on "Too old to program?".

I really touched a nerve with that blog, and I am not surprised in the least. Ben told me that every programmer he knows starts having the same fears as they approach 30. Imagine, a career where you are "over the hill" within 10 years of starting. We might as well chosen to be Olympic Gymnasts.

Back to my story... when Ben excused himself and walked away, the checkout guy remarked: "Wow, a famous guy came through my checkout line!"

"Famous Guy!" Wow, This must be my 15 minutes! Who'd of thought that it would be in a checkout line, and in response to a cropped mugshot? Oh well, enjoy it while it lasts.

My online UserID is always "johnreynolds" if I can get it, and some variation on that theme if that UserID is already taken. I've never been all that interested in adopting a nickname or pseudonym, even though "John Reynolds" is a very common name.

The downside to this is that anything that I write online can be traced back to me. Every opinion that I put in print may come back to haunt me at a job interview, or maybe even in a checkout line ;-)

The upside to this is that I'm more careful about what I write. It's like living in a small town, anything that I say will get back to the neighbors, so I need to be careful what I say about them.

Other folks choose to be more circumspect about their identities. For example, the Tapestry Table Component was authored by MindBridge. I was under the mistaken impression that MindBridge was a pseudonym for Howard Lewis Ship when I first published my Tapestry examples. I still don't know what MindBridge's real name is (and I wouldn't divulge it if I did know), but I can tell you that he's a great coder and a very generous person.

In this day and age, privacy is an illusion.

A piece of spam recently arrived in my inbox that caught my attention. The subject line read:"John, your current checking account balance is $1,234.56" (I've changed the amount to preserve my privacy). The amount was correct. Needless to say, I was less then enthused to realize this information was public.

All in all, I am less interested in preserving my privacy then I am in insuring my identity. Think about it, how can I prove that I am me? My education and employment records are tied to a Social Security number... but there is nothing tied to that Social Security number that identifies me as me. There's no picture, no finger-print, no DNA on file; just a number and some information that is common knowledge.

I know that many people fear a National Identity Card, and I can sympathize; Governments are notorious for turning oppressive. I hold the other view; I want a National Identity Card, and I want my DNA and finger-prints on file.
Without a trusted identification authority (and I know that nothing is foolproof), it's just too easy for someone else to pretend to be me, and too hard for me to prove that I am me.

Fortunately, java.net has changed all that. All that I have to do is point folks to http://weblogs.java.net/blog/johnreynolds/, cock my head to one side and crack a toothy grin, and voila!, proof positive that I am me! Isn't fame great?

Too old to program

(Cross posted from my java.net blog)

I was drawn to E-Surfer's recent weblog entry that asks the question: "Could I Still Pursue Software Development Career When Getting Older?"

E-Surfer's "friends" warn him that:"when getting older I would not be competent to pursue software development because of slowness of thinking, difficulty reading and writing, and so on."

I will be 48 years old on my next birthday, and I have been professionally programming since I got my BSEE degree from Rice University back in 1980. I freely admit that my memory is not what it once was, but I am pretty sure that "slowness of thought" and "difficulty reading and writing" will never be the primary reasons that someone leaves the programming profession.

I started programming when many programmers were also digital logic designers (hence my electrical engineering degree). While I was in school, the 6502 microprocessor and the Z80 microprocessor became widely available, and almost everybody that I knew was wire-wrapping chips together on perfboard to build their own "personal computers".

Cheap microprocessors transformed the world, and resulted in an explosion in the number of professional programmers.

An odd personal side effect of starting my career at the dawn of the PC age is that I have always been one of the oldest programmers at every company where I have worked. I was dealing with being "the old guy" before I hit 30.

For me, the biggest obstacle to staying in the programming profession has been the frenetic pace of change. Programmers must continually learn new lanquages, new operating systems, and new devices just to stay employable. For example, in '85 my MSCSE focus at the University of Texas at Arlington was on computers that were optimized to execute Lisp. Lisp programming positions were always few and far between, so I relied on assembler, Pascal, 'C' and FORTH to pay the bills. By the 90's, C++ was on the scene, and along with it the need to leap from structured programming to the OO paradigm (thank heavens!).

The advent in the mid-90's of the Internet and Web-based applications once again negated much of what I had previously mastered. Along came Java: first applets, then servlets and JSP, and now a plethora of client and server side technologies (most of which have something to do with XML).

My point in relating all of this is that programmers must commit themselves to life-long learning. With the possible exceptions of a few safe-harbors (like mainframe COBOL), practitioners must keep abrest of changes and be prepared to discard their tools, languages, and paradigms every 2 to 3 years (at best). With the dawn of EJB 3.0 will you ever again use the esoteric EJB 2.0 home and remote interface knowledge that you needed to pass the J2EE certification exam?

I have never been "put off" by the demands to learn new things, but I do get very cranky in the realization that much of what I must learn is about reinvented wheels rather then better wheels. I have probably learned over 20 ways to implement what is essentially a modal dialog box. Where's the value in that?

Rather then failing health and mental accuity, I think burn-out is what does most programmers in. I think we just get tired of fighting the same old battles and move on to other things.

Fortunately, I believe that the profession is on the verge of getting much, much better (in terms of sustainability). As I've blogged before, adolescence is over and the industry is maturing. I do not expect the pace of change to slow, but I do expect that the nature of the changes will be more manageable.

Much of my optimism is pinned to the renewed interest in Service Oriented Architectures and Business Process Management.

Programming has long suffered from two major faults:

1. The mappings between requirements and implementations have been vague at best
   


2. The elements of our programs are too tightly coupled to each other (and to languages, frameworks, etc.)
   

With SOA (done right) the business services that programmers develop will be accessible to clients that are clueless about the implementation.

With business processes running on BPMN engines the mapping between the business process and the requirements will be crisp (and language neutral).

BPM and SOA will be good for the professional programmer because they will bring to the foreground the skills that are not tightly coupled to specific languages, operating systems and devices.

Programmers are problem solvers. Programmers are logical thinkers. Programmers can figure out why it doesn't work. Programmers like to build new things and make old things work better.

Perhaps some folks think that limits programming to young folks, but I respectfully disagree.

Adolescence isn't all that it's cracked up to be

(Cross posted at my java.net blog)

Brian Marick recently blogged: "I worry that the adolescence of computers is almost over, and that we're settling into that stagnant adulthood where you just plod on in the world as others made it, occasionally wistfully remembering the time when you thought endless possibility was all around you."

Brian's high school memories must be a bit fonder then my own ;-)

I understand Brian's sentiments, and share many of them, but I'd like to speak up and praise the merits of maturity. Endless possibilities were never around us... we always have been and always will be bound by our basic natures.

Adolescents are head-strong and righteous. Every idea that catches their fancy is new and original. Every disagreement is passionate and personal. A lot of yelling and shouting takes place.

Adolescents are oblivious to the possibility that this behavior might not be a good thing as you can see here in Exhibit A:

"One last piece of wisdom, the tone of the lists can be very aggresive. People here are usually talented and pig-headed or equivalently from Russia (with love to spare). So the discussions look like flames... well they are flames. As long as your information is tight and of good quality we don't care about the arrogance. It's a playground, it's all a big playground. So e-n-joy it."

Witness our own Java community's recent bickering over EJB versus JDO, Swing versus SWT, Tapestry versus JSF, Eclipse versus Netbeans, etc. There are many legitimate concerns underlying all of these disagreements, but very few true roadblocks to compromise and cooperation.

Arrogance is very destructive. I know, I was an arrogant jerk for years and still suffer relapses (it's kind of like being an alcoholic, you're never really cured).

So growing up is a good thing: The end of adolescence opens up real if not endless opportunities, and you've got the wisdom to know which opportunities really should be pursued.

I think that Java's maturity could be really great. Java's adolescence was kind of awkward... the growth spurts left him uncoordinated and the hormone swings ruined his complexion. His legs were always longer then his pants and his socks never matched. Much of Java's "philosophy" was convoluted, esoteric, and just plain wierd.

Java has baggage, but baggage can be discarded. Java can refocus on making the easy things easy and the harder things possible. Java can calm down, become more effective, and be more fun to work with.

Being an adult is a good thing.