Managing the Garbage Collection Process

For those of you who follow my Java oriented blogs, don't continue reading this if you're looking for information on Java's Garbage Collection... I'm going to blog about real-world trash today:

In my neighborhood in Austin, we have weekly curb-side garbage collection. The city provides me with a large plastic bin that has attached wheels, and on Friday mornings they'll empty this bin and haul away the garbage if I've remembered to drag the darn thing out to the curb.

Within the Reynolds' household, "taking out the trash" is a weekly Process whose Activities have been assigned to me by the boss my lovely wife Teri. "Taking out the trash" isn't a complicated process (in theory)... there are only a few steps:

1. Gather trash from the various trash cans scattered around the house.
   
2. Wheel the trash bin from the side-yard to the curb in the morning (before the garbage truck arrives).


3. Wheel the trash bin from the curb to the side-yard when I get home from the office.
   

In practice, "taking out the trash" almost never works smoothly. Step 1 (gathering the trash) almost never occurs without an Escalation Task Notification from my supervisor that the Task is overdue. For some reason she always remembers "Tomorrow is Trash Day" long before I do (selective memory, no doubt).

There's also the problem that Step 1 almost never meets its SLA (Spouse Service Level Agreement)... I inevitably forget to empty some of the small trash cans in some out-of-the-way rooms (like the kitchen, bathroom, living room, etc.). Once again, Escalation to my supervisor is almost always required (She usually ends up doing it herself).

Obviously, the Management Team of the Reynolds' household ought to invest in a good BPM system and turn this into a Managed Process ;-)

My little Process is only a small Sub-Process of the City of Austin's Garbage Collection Process. They've divided the City into geographical districts that are serviced on different days of the week. Everything goes off like clockwork except for a few times a year when Holidays cause the schedule to slide by a day or so, and during special periods where the volume of trash increases dramatically (Christmas, Bulky Trash Day, etc.).

The City is pretty good about sending out schedules far in advance of any changes in service, but you still see a lot of trash bins out at the curb on "slip" days. Some folks (me) just can't seem to keep the schedule straight.

As a Managed Process Guy this is obviously a deplorable situation. We ought to Analyze the Current Process, Model an Improved Process, Implement the Improved Process, Monitor Its Performance and then repeat these steps until we Perfect the Process.

If each trash can in my house were actually a Robot (something like a Roomba), then the trash cans could gather themselves... rendezvousing on the side porch to empty their contents into the City's trash bin.

The City's bin would also be a Robot, in constant touch with the City's fleet of Robot Garbage Trucks. The Robot Trucks could signal the Robot Bin when in the neighborhood... the Robot Bin could signal its little flock of Robot Trash Cans, and I could watch TV for a few more minutes.

The only problem with this Grandiose Process Improvement Scheme of mine is that the Return On Investment is most likely non-existent. It's a great idea (from my perspective) but the Cost will never justify the Benefit).

Cost... That's the real point isn't it?

What does the Current Process cost? Are there opportunities for reducing that cost?

To answer either of these questions you are going to have to define "cost"... Cash outlay, time spent, bad feelings, etc. Once you've defined "cost" then you are going to have to monitor the factors that contribute to "cost"... Only then can you begin to have an objective discussion on how to make the Process better.

So... The first step towards resolving the Garbage Collection Process pitfalls in the Reynolds' household and the City of Austin is to Manage the Process "Just Enough" to Monitor it. Hopefully the cost associated with my forgetting to perform my assigned tasks will justify the purchase of those Robot Garbage Gatherers, but somehow I doubt it ;-)

UPS Envy - The Visible Process

The number one reason why companies invest in BPM is because they have "UPS Envy"... I just made up that statistic, but I'll bet that it's not far from true...

If you've ever ordered anything online and had it shipped by UPS, then you've probably seen the UPS tracking screens. As the buyer, you can see exactly where your package has been, when it was there, and where it's going. Every drop-off, every pick up, every mile... from the shipper to your door step... It's GREAT.

I've never seen the "Shipper's View" of UPS, but I'll bet that it's phenomenal too. You can probably see all of the packages that you have shipped in a consolidated view. Everything in real time, letting you sleep comfortably knowing that your customers are going to get what they ordered on time... or giving you the warning that you need to preemptively head off the disappointment of a late shipment.

UPS itself probably has a "Gods-Eye View" of every package in their system... Great for day-to-day operations, and no doubt with historical analysis features to determine the optimum number of trucks, trains, and planes that were actually needed to get those packages delivered on time.

This is what BPM promises... The ability to track each step of all of your processes, and to be able to analyze past metrics to plan for a better future.

1. Model the Process
2. Manange the Process
3. Monitor the Process
4. Analyze the Process (darn... I can't think of a synonym for Analyze that starts with "m")
5. Modify the Process

If your BPM implementation doesn't provide these capabilities, then you've missed the mark.

In almost every Managed Business Process that you implement, you will probably need to provide the following "Current State" views (at a minimum):

Originator's View

This is analogous to the Purchaser's UPS tracking view. You initiated a Process, and you want to know how things are going. More than likely your primary concern is in knowing when the Process will complete (which implies knowing what steps are left to complete) and you probably also want to know how things have gone so far (what steps have been performed, and who performed them).

Participant's View

This is similar to the Originator's View in some respects... but you are generally more interested in all of the Processes that you've participated in rather than a single Process. In many cases, you are interested in the Processes that may "come back to haunt you"... Processes that are still "in flight" where you may be required to perform additional steps (or defend past actions).

Manager's View

This could be the Manager of the complete Process, or the Manager of a Group of Participants who own a Swim Lane of a Process. In some respects this is a Cumulative Participants View, but there is an emphasis on spotting overdue tasks that might lead to an SLA (Service Level Agreement) violation.

Note that all of these Views could span Multiple Processes. The Originators, Participants, and Managers will often be involved with many different types of Processes simultaneously.

I have emphasized the "Current State" in describing these Views, but obviously (especially for the Managers) Historical Views are also quite useful.

Process Engines are primarily concerned with making sure that the right tasks are assigned to the right participants at the right time... but they must also support the generation of all the Views that I've mentioned. This can be accomplished in many ways... but how ever you do it, make sure that you're satisfying that "UPS Envy".

The Best Way to Build a House is the Worst Way to Implement a Process

I love analogies, and sometimes a bad analogy can be just as useful as a good one...

A House is NOT a good analogy for a Process, but contrasting the best way to build each of them is kind of fun.

The "best"way to build a house is something like the following:

1. Build the Foundation
2. Build the Frame of the house
3. Add Roofing material to keep out the rain
4. Run the Plumbing and Electrical
5. Drywall the Interior
6. etc. etc.

This is the most Pragmatic and Practical way to build a house, and probably the most cost efficient.

The only problem with building a house this way is that you have to live somewhere else until the house is finished. Nothing in the house is finished. Everything in the house is in a state of flux.

If you've read many of my blog entries you probably see where I am going with this... but let's continue.

There is another way to build a house. It's not used very often, but it does work:

1. Build a room... completely. Foundation, Frame, Roof, Electrical, Plumbing, Drywall, etc. In essence a "mini-house"
2. Build another Room
3. Continue until you have all the Rooms that you need

The advantage to this approach is that you can move into the first room as soon as it is finished. You have a place to live... and you are getting a "return on investment" from all the money that you've spent so far.

There are many downsides to the "One Room at a Time" approach, but I think the most obvious is this: You don't have a "full featured house" when the first Room is completed. Your "mini-house" is not as functional as a "complete" house.

To offset this you should build your rooms in terms of importance. What Rooms are absolutely essential?

Personally, I would build the bathroom first, but that's just me :-)

As you build additional Rooms, in the order of most essential to least essential, your House gets increasingly closer to the "Dream House" that you've longed for. Unfortunately, while each new Room is added you are essentially living in a construction zone. Workmen keep tramping through your living Room (once it's built) and there's a lot of hammering and sawing going on to disturb your peace.

Building a house this way probably costs more too. It's much cheaper to have an electrician and plumber work on the whole house at once then on a room-by-room basis.

For all it's down sides, I think the "One Room at a Time" approach can have a lot of merit when building a House... each Room tends to be more what you really want than when you try to do everything at once.

In the realm of Business Process implementation, I am convinced that the equivalent to "One Room at a Time" is the best way to guarantee success when building a Managed Process. I've blogged about this many times before (Drive the Path )... and I'll probably blog about it many more times in the future.

When approaching a Complex Process, identify the Activities along the most important paths and make them (and their associated Decision Gateways) functional first. When you are done, you will have a Managed Business Process and you can start using it immediately. You may choose to wait until more of the Process has been implemented before "turning it on", but just like the "One Room at a Time" approach to House building, the parts that are complete are ready to use.

This is particularly important when building Managed Processes, because the Business probably does not have "somewhere else to live" while the work is going on. There's a problem with their existing Process, or they wouldn't be spending the money to Manage it. They need something useful as soon as you can get it to them... and ultimately it is their decision (the Business Folks) whether or not to "move in early".

Lame analogy or not, Houses and Processes are both usually built for someone else - and satisfying some of the clients needs as soon as you can is almost always a good idea.

---

Another, perhaps more subtle advantage of starting with a Functional Process and adding additional Business Value over time is that it conditions your Business folks to think of their Process as a Continuous Work In Progress. Instead of a Static Process Definition that never changes, they will come to view their Process Definition as something that Improves over time... That sounds a lot like Continuous Process Improvement, doesn't it?

Tips for the Business Process Developer - Iterative Development

Iterative Development is by far the best way to implement your Managed Business Process... unless your definition of "Iterative Development" is different from mine ;-)

(Obligatory notice: I am an employee of Lombardi Software, but the opinions and advice presented here are my own.)

I've had a surprisingly difficult time conveying my own definition of "Iterative Development" in the past, so I thought I'd take a stab at explanation via analogy... Let's compare your Business Process to a trip from Austin Texas to El Paso Texas:

The most important aspect of my trip is arriving at my destination. No matter what interesting things may happen on the way: If I don't end up in El Paso my trip has failed.

The same is true about your Business Process... No matter what else goes on, there is an objective to your process and you have to accomplish that objective.

The first iteration of your process... like the first iteration of my trip to El Paso... should implement just enough to get you through all the steps of the process. No extraneous details, but enough to get you from the beginning to the end. Start your iterations with the the Flow aspects of your Process. What are all the places that you need to go?

Austin is a long way from El Paso... lot's of lonely miles on the highway. Fortunately, we have family in San Angelo Texas, and it's not too far out of the way to make a detour when we are headed out west: Based on how much time we have, the weather and road conditions, and how long it's been since we've seen our family in San Angelo, we make a decision whether or not to take a little side trip.

Your Business Process, like my trip from Austin to El Paso, probably has multiple paths that it can take. There are Activities that are always performed, and there are Activities that are occasionally performed based on "current conditions" and other factors. Implement a place holder for each activity and capture the decisions that are necessary to pick the proper path.

At this point I need to take a step back and be a bit more explicit about what I've defined thus far regarding my journey from Austin to El Paso. I have done just enough to get to all of the places that I need to visit. I haven't yet filled in any of the other details. I've got the directions on how to get there... I haven't worried about where to get gas, where to eat, etc.
You've defined the Flows through your Process, and you've implemented just enough stuff to get you through all the activities that make up the Process. You can now "run" the Process and step through each of the Activities... but the Activities don't really do much... They just prompt the Participant's for the minimal information necessary to get to the next step.

That's your First Iteration: All of the "Happy Paths" through your process work. There's not a huge amount of business value yet... but you do have a managed process in place. All of the steps of the process will get executed in the right order... and the decisions at each step will be captured to guide the process along the correct path.

Your Process logic is now correct, so it's time to start iterating through the Data aspects of your Process. You will need to know all of the information that is gathered during each Activity in the Process. You will need to know all of the information that is presented to the Participants during each Activity in the Process.

I hope that you see where I am going with all of this... Iterative Development of a Process must be done with respect to the Whole Process. Each Iteration delivers a more functional process. All of the Activities of the Process have to be functional, or you don't have a working Process. Each Iteration should improve functionality across the whole process (in general).

Contrast Iterative Development (as I've defined it) with Incremental Development.

Incremental development focuses on fully implementing each of the Activities that make up the Process in some order. I see folks following this approach a lot when a particular Activity involves integration with an external system... the IT guys will always want to tackle that part of the Process first. I also see it when one Activity dominates the Process... such as preparing a very involved application. It's our nature... We focus on the parts of the Process that we depend on. We focus on the parts of the Process that we are responsible for.

The Incremental approach is analogous to fully planning my trip from Austin to Junction... then fully planning my trip from Junction to Ozona... then fully planning my trip from Ozona to Fort Stockton... then fully planning my trip from Fort Stockton to El Paso (with maybe a little side trip to Fort Davis). Nobody denies that we'll (eventually) need to know all of those details... but by following that approach we won't be able to make any sort of journey from Austin to El Paso until the last step of the journey is fully defined.

Incremental development is how Process oriented projects get bogged down. This is how Process oriented projects just drag on, and on, and on. Your development team ends up focusing on Activities of the Process, rather than on the Process itself... and you often end up with Activities that don't make sense when the Process is finally finished.

Iterative development means building a process that works, then making it work better, then making it work better, then making it work better... It's an incredibly effective way to build your Managed Business Process, and there's no reason for the cycle to end after the initial deployment... You just start calling the cycle Continuous Process Improvement.

Tips for the Business Process Developer - Getting Tasks to the Right Participants

Recently, I was working with a client to implement a managed Loan Origination Process using Lombardi's Teamworks BPM suite...

(Obligatory notice: I am an employee of Lombardi Software, but the opinions and advice presented here are my own.)

As with any real-world process, there were a few interesting "gotchas", but on the whole it was a pretty standard process that many of you would easily recognize.

Modelling the Loan Origination Process using BPMN diagrams was fairly straight-forward, but when we got to the point of implementing Task Assignment and Routing things got interesting. Let me very loosely paraphrase some roughly similar Task Routing requirements (that I made up for this blog entry):

"When a Loan Application comes in, make the application available to all of our Credit Officers who have sufficient Lending Authority to process the loan, and who haven't already met their quota of applications for the week. Of course if everybody has already met their quotas for the week, then let everyone have an opportunity to work on the new application.

One more thing, only a few of our Credit Officers can approve loans over $100,000, so don't route low value loans to them unless they are the only available credit officers.

And don't let anyone claim more than 3 applications at a time. Our Credit Officers are compensated for the number of loans that they process, and some of them will claim all the applications that they can in hopes of earning more money."

These Task Assignment/Routing requirements are kind of involved, but they make sense in the real-world. If you can't handle real-world problems in your managed business process then you aren't going to make many business people happy; So how do you satisfy complex Task Routing/Assignment requirements like this?

There are three key elements to any business process:

1. The steps and flow of the process
2. The data the process generates and consumes
3. The participants who perform the process

Task Routing deals with the Process Participants: How do you know who the right participants are and how do you assign tasks and route information to them?

All BPM suites have one form of task list or another that holds all of the tasks that are assigned to a user and all of the unassigned tasks that the user can claim. Sometimes the task list appears on some sort of a Portal. Sometimes the task list is integrated with a mail client like Microsoft Outlook. Often task notifications are emailed to users. Task Routing in a process is all about getting the right tasks to show up on the right user's task list at the right time.

A BPMN swim lane indicates the Role (in the process) of the Participants who execute any of the Tasks in the lane. These BPMN Roles can be very specific, both at the process definition level and at the process instance level. For example, an individual who is acting as a Reviewer in one instance of a process may be acting as a Requestor in another instance of the same process.

In many processes, a single task may be routed to a group of people rather than assigned to a single user. Let's refer to this group of users as the Task Group. All of the Task Group members are eligible to claim the task until one member of the group claims the task.

If you've dealt with Authorizing a User in a "normal" application, then Assigning a Task in a process will seem very familiar. When people log on to web-based applications there is almost always a two step process occurring under the covers: Authentication and Authorization.

Authentication is conceptually simple: Is the user who they say they are? This is usually handled by prompting for a user id and a password. If the two match, then the user is someone who can log on to the system. In Single Sign On environments, authentication is generally handled by a token (such as a SAML assertion ) that vouches for the user's identity.

Authorizing a User is often a bit more involved than Authenticating a User: Is the user allowed to perform a specific task or view some specific information?

Authorization is less straight-forward than Authentication because a single application may be written to service users with significantly different permissions to perform tasks and to view information. If the logged on user has specific permissions, then they can perform specific features and view specific information. Conversely, if they don't possess the proper permissions, then they cannot access some of the application's features or view some of the information.

The relationship between Task Assignment and User Authorization is pretty obvious:

Authorization logic is applied to determine whether or not the logged on user can perform a task or view some information.

Task Assignment logic is applied to determine the group of users who are authorized to perform a task or view some information.

Many organizations use an LDAP repository like Active Directory to hold the Authorization information associated with users. In this model, users are assigned to specific groups and Authorization is fairly straight-forward. Assigning a BPMN swim-lane to users who are members of a specific LDAP group is a no-brainer.

Unfortunately, in the real world the membership of a Task Group can be very dynamic... Here is an example slightly modified from the loan origination process that I worked on:

• The Task should be assigned to all Credit Officers with a Lending Authority Limit of more than $50,000 who have processed less than 5 applications this week.
  

At first glance, it seems reasonable to create an LDAP group for Credit Officers with Lending Authority over $50,000... but it's really not. The required Lending Authority Limit is actually based on the amount of the loan. You need a Credit Officer with authority greater than the amount of the loan, so this is really an attribute of the Credit Officer rather than a specific group that they belong to. The second half of this requirement, "who have processes less than 5 applications this week", is an extremely dynamic categorization... There's no way that you would store this sort of dynamically changing data in an LDAP repository.

To a Business Process Analyst, specifying a Task Routing rules like these are very straight-forward. To the Business Analyst, it's extremely clear who should be eligible to claim the task, and who shouldn't be able to claim the task.

To a Business Process Developer, trying to implement a Task Routing rule like this is a nightmare. It's usually very unclear how to access the data necessary to implement this rule.

Enter the Task Routing Service...

The "right" answer to solve a complex routing problem like this is to develop a custom Task Routing Service to determine the list of users who should be given the opportunity to complete the task. If the conditions for eligibility are very dynamic (if they could change in a few minutes) then it's also a good idea to develop a related service that will tell you if a specific user is eligible to claim a specific task.

When a task is ready to be performed, invoke the first service to get all of the eligible users.

When a user claims the task, call the second service to determine if they are (still) eligible, and return the task to the pool if they aren't.

Encapsulating the Task Routing Policy (the logic to determine the eligible users) in a separate service frees us from having to worry about the details of determining those users while working on the other aspects of the managed process. Our custom Task Routing services can implement any routing policy that is necessary.

Refining the Task Routing policies is often one of the areas with the most penitential for improving the overall performance of a process. If tasks are languishing on a queue, waiting for someone to claim them, then nobody is happy. If "simple" instances of tasks are routed to the "most skilled" participants, that's probably not the best use of resources (and once again nobody is happy).

The Task Routing Service that I've described for this loan origination process requires access to information from several sources:

• Information from the current process instance
  
• Information about all of the possible participants
  
• Cumulative information from "related" process instances
  

Getting information about the current process instance is never a problem. Since the Task Routing Service is called from within the process itself, any relevant information can be supplied.

In this specific process implementation, the possible participants (the Client Officers) are defined in a database table... There is not a group in LDAP that distinguished the possible participants. The same table holds attributes of the Client Officers that are needed, such as "Lending Authority".

Cumulative information from related process instances is often the hardest to come by. This is generally the information that is used to load-balance the workload of each participant, and the total workload of any participant may be dependent on many types of processes (not all of which may be managed). Frequently, when a project is begun to implement a managed business process, some of the information necessary to implement a complex Task Routing Policy will not be available (and may not become available before the first release of the project).

So what do you do if the information that you need for your Task Routing Policy isn't available?

Do the best that you can with what you have.

The first release of the Task Routing Service for the Loan Origination Process that I have described will be implemented as a Human Powered Service. Prior to this BPM project the task routing was performed by a person. When new Loan Applications were received a central "Traffic Cop" would review each application and assign it to a specific Credit Officer. This manual assignment process was actually working quite well since the number of Credit Officers was small and the "Traffic Cop" was really good at the job. Although the ultimate goal is to automate the task, we are "punting" and will create a UI for a human to make the routing decision in our Task Routing Service.

Here's another key reminder: Don't spend a lot of time and effort (and money) implementing a complex Task Routing Policy unless it really is needed. Back to my example, the human "Traffic Cop" routing service really may be the right solution for the present. Until the "Traffic Cop" is overwhelmed by volume, replacement of this routing service with an automated one might not provide a reasonable ROI.

Task Routing in a managed business process is often very simple and easy to implement with the out-of-the-box functionality of a BPM suite. For those times when it's a bit more complex (or maybe even insanely complex) don't despair. With a well-designed Task Routing Service you can handle almost anything.

Tips for the Business Process Developer - Drive the Path (process and data flow)

Any tools can be used wrong, and I believe that's the reason many developers hate BPM. They just don't know how the BPM tools should be used... and I'd love to rectify that situation.

If you grok Process Driven Development you will love BPM. If you don't, then you'll try to use your BPM tools like a traditional application development environment and you will end up with a mess.

It's all about the Process...
Before you begin development, discover the answers to these questions in this order:

1. What are the steps?


2. What are the possible paths through the the steps?


3. What data controls the path through the process?


4. What data flows through the process.

When you know these answers, then start your development efforts with the process diagram and use that to drive your development path in this order:

1. Model the process flow.


2. Define the data that controls the process flow.


3. Build "just enough" user interface to allow you manipulate the data necessary to step through all the process paths.

It's absolutely critical that you do these steps in this order at the outset of your project... and once you are done you've got to do the following:

Drive through ALL of the process paths with your Business Sponsors.

If you follow these steps, then you will expose the flaws in the process flow long before you've wasted any time coding up a fancy UI that is WRONG.

The process should drive the UI, but often our haste to show a sexy screen to our Business Sponsors the UI ends up locking us into a bad process model. We've all made this mistake, and we know from experience that it screws things up for the rest of the project's life.

Everyone focuses on UI. It's the most visible aspect of a project and the easiest to have an opionion about... But in truth the best UI in the world won't make the wrong process right.

Yes indeed... the UI development tools in some BPM suites often require super-human efforts to build the sexy AJAX powered screens that are all the rage, and often you'll end up developing your fanciest UIs with supplemental tools. But that's not why some BPM projects fail...

BPM project's fail when the project team doesn't follow the steps of Process Driven Design that I've laid out in this blog entry. If you've had problems with BPM... try it my way next time. I think you'll be pleasantly surprised if you do.

The BPM Elevator Speech

A few years ago I posted a short blog entry "The SOA Elevator Speech" to try to distill SOA into talking points that you might be able to cover on one elevator ride. With that posting in mind, here's my attempt at explaining BPM as concisely as I can...

• BPM stands for "Business Process Management". You will find some folks who say that the "M" stands for Modelling, and a few others who say that the "M" stands for Monitoring. In my opinion it really stands for all three so we might as well call it BPM³.
• BPM systems deal with processes that include both tasks that Humans perform and Automated tasks.
• BPM systems execute Process Definitions. Process Definitions are usually derived from Process Diagrams... for example a BPEL (for people) Process Definition can be derived from a BPMN Process Diagram. There are many different graphical process notations and process definition languages, but the goals are generally the same.
• BPM systems provide a Process Centric view of the code base. Rather than focusing on object hierarchies, the focus is on the relationship of the components in the context of a specific business process. This feature comes in particularly useful when debugging a process as well as when the need arises to modify a process.
• BPM tools generally have a Business Relevant focus rather than a Programmer Relevant focus. For example, a typical programming suite might include a profiler for analysing the execution efficiency of code... a BPM suite is more likely to include a profiler for analysing the efficiency of a process (based on historical or simulated metrics).
• The goal of a BPM system is to enable Continuous Process Improvement. Analysis of existing process behavior combined with simulation of potential process enhancements can be used to identify and improve the business process.
• BPM benefits from SOA, but Process Tasks are not necessarilly reusable services. Many tasks in a Business Process are not reusable in other processes, and many SOA services do not correspond to Process Tasks.

That's already too much material to cover on one elevator ride... I'd better give it a rest for now or you're likely to take the stairs the next time you see me on an elevator :-)

Why do Java developers hate BPM?

Java developers hate BPM.

The preceding sentence is (of course) intentionally tailored to be controversial. People tend to read controversial blogs, and I'd like you to read this one. Now that I have hopefully grabbed your attention I'll tone it down a bit...

A lot of Java developers hate having to use BPM tools instead of the object oriented tools that they are comfortable with.
I work on a lot of BPM projects, and I work with a lot of other folks who work on a lot of BPM projects, and we have all encountered resistance from traditional Java developers. Java developers (in general) would rather use frameworks like Struts and Spring than be saddled with the constraints of a BPM suite.

Java frameworks like Struts and Spring are in the background... they provide just enough support to "set your creativity free" so that you can be a real programmer. You can build almost anything with Spring or Struts (if you've already mastered the intricacies of Java). They are light-weight, they're agile, and they look sexy on your resume.

BPM suites are in-your-face. They rob you of your creativity. They dictate to you how you will develop your application.

BPM suites make programming boring. They force you to use point-and-click and drag-and-drop tools to design your process diagrams, data models and forms. What's worse, they actually encourage Business People to model processes and design forms on their own... Fortunately most Business People are too intimidated to use these tools, but it does open the door for them to look over our shoulders and meddle in our affairs.

That certainly doesn't sound like something that real programmers would like, does it?

I'm not being subtle, am I?

BPM suites are a threat to traditional Java programmers. These suites are far from perfect, but even in their current state we can see where things are heading. The days of the Java Guru as indispensable are fading... We've used Java to build tools that make knowing Java itself less important, and that's opened up competition for us from folks who didn't spend years learning Java.

We're victims of our own success... programming isn't as hard as it used to be... and that's going to cost us.

That's why Java developers hate BPM.

Continuous Process Improvement and Composite Applications

At the 40,000 foot level, it's hard to see the difference between the "Services" in SOA and the EJBs of Java EE. Java's (remote) EJBs were heavily influenced by CORBA, and both were intended to generate a wealth of remotely invoke-able business oriented components.

There are of course many differences in the technical implementation of Web Services, EJBs, and CORBA objects... but there are not that many conceptual differences. Yes, formatting parameters with XML did open the door of programming language neutrality much wider, but is XML really the killer concept leading Web Services to triumph where EJBs and CORBA objects faltered?

The lasting value of any component technology isn't really due to technical merit (think back on the original Visual Basic components)... it is due to the philosophy and mind-set associated with those who use the components. If the community of users adopts the "right" attitude, the technology will flourish despite technologically superior alternatives (think how long the Mac survived despite under-powered CPUs).

EJBs and CORBA didn't succeed in fostering the "right" attitude for success. Authors crafted components... but those who used the components frequently didn't "get it". Most EJBs ended up tightly bound to specific applications... languishing as just another tier in an "N-tier" application.

The success of SOA is going to hinge on whether or not programmers really grasp the difference between traditional and Composite Applications. Many will grumble that they've been "composing" applications for years by wiring together components. That's true... but the devil is in the details.

There are components, and then there are components. There's wiring components together, and then there's wiring components together. Composite Applications in the SOA context have very specific ideas on what constitutes a component and on how they should be wired together.

SOA is about satisfying business needs, and most of what business needs concerns understanding and improving business processes. I am fairly sure that you have heard the term "Continuous Process Improvement". (CPI) ... but in the event you haven't, CPI is what makes winning companies win. CPI is what propelled Japan's industries from bombed out shells to Global Domination.

Composite Applications enable continuous process improvement. Traditional applications all too often "cast the process in stone". Although software is theoretically easy to change, the sad reality is that many companies are "stuck" with their applications due the the high cost (and risk) of making changes.

Components in Composite Applications closely map to steps in Business Processes. Components are wired together to build processes, but the wiring is "loose". Think plugs and sockets instead of solder.

The "right"type of components and the "right" method for connecting them together go a long way towards enabling Continuous Process Improvement, but it's still not enough:

If you can't see it, you can't fix it.

"It" in this case refers to the activities or services that constitute a Business Process. Composite Applications are not a bunch of Web Services with point-to-point wiring. Composite Applications rely on orchestration engines to coordinate the flow of information between components... enabling rapid change of these applications. Enabling change is a big plus... but how do you know what to change?

Continuous Process Improvement relies on metrics. Monitoring leads to Analysis leads to Improvement. If you don't have the metrics, your attempts to improve a process are based on hunches rather than facts... and that's where the orchestration engine comes in by enabling the efficient gathering of process relevant metrics.

With process relevant metrics, processes can be analyzed, proposed changes can be simulated, and Process Improvement can truly be Continuous... (assuming that your Composite Applications can evolve quickly)

In the final analysis that's the only reason why business care about all of this SOA stuff... improving their businesses.