Robert V. Binder

How a Big Ball of Mud becomes a Black Hole: Why Software Architecture and Process Matters

July 11, 2011  |  Blog, Business, Process, Software Products

Cosmic matter spiraling into a black hole

Once upon a time, I had a Software 25 company with a struggling division as a client. They were selling and supporting an integrated enterprise system brought in by acquisition. This product, at version 7.0, was dominant in its market. But, with apologies to Gresham, bad software was driving out the good: more and more development time was spent fixing bugs and less and less on new features. The competition had no such problem. Their feature set was thin but had lots of sexy new stuff. My client was getting killed: in new sales situations, they almost always lost. An alarming number of existing customers were defecting, fed up with too many bugs. The story was much the same for the other divisions.

To reduce bugs and cycle time, I was asked in to fix the software process. The 7.0 system itself (features aside) was the classic big ball of mud. The process was well-organized chaos lead by the maintainers — coder heroes who had a vested interest and pride of place in their cruft. They also had lots of bile for “****ing ***hole consultants and all that theory bull****.” Windows were smashed in the parking lot.  Nonetheless, I started work, made some friends, produced a roadmap, and started to chip away at the cruft.

Besides the maintainers, another project team was developing an all-new next generation system that would trounce the competition and protect the installed base. This dream team (selected from the maintainers) was working in a “secret” location, but nothing about it was secret. The dream team was equally adamant that they didn’t need no stinkin’ process. The maintainers were convinced that things would only get worse. They were right, but not in the way they expected.

There was not enough cash flow to maintain 7.x and develop the next generation king-of-the-hill app. The business had become an economic black hole. I think the management team must have recognized this and decided that the solution was to sell the company. About 3 months after I started, my client was acquired by a Software 10 company, whose business model was simple: Buy established software companies with a large installed base, minimize operations and development, and maximize license revenue stream. Save for a skeleton staff, the maintainer heroes, the dream team, and the ******* consultants were all shown the door on the day the deal closed.

I don’t how this worked out for the third owner, but I’d guess they probably got a modest ROI. That was enough as they were mainly interested in using the cash flow to support more acquisitions (the better to repeat the whole cycle again.) About five years later, they were busted for severely hinky accounting.

In retrospect, I think selling the company was the right call. The dream team couldn’t have released anything soon enough to regain share and the black hole was sucking in more resources than it produced. Even with the career disruptions of several hundred folks, selling the company probably minimized overall losses for everybody.

From a software product management point of view, this shows how hard (and important) it is to balance the long term costs of ad hoc development and short term pressures. It is very easy to collect a lot of smart and energetic programmers and produce a 1.0 and even 2.0 app that works and sells. If the product really takes off, you get a cargo cult effect – whatever weirdness that happened to be produced becomes worshiped. I don’t know of any situation where this hasn’t resulted in a system artifact and “culture” that only knows how to do more of the same, i.e., produce even bigger balls of mud. When they get big enough, they become black holes.

Software architecture and software process matters, especially if you’re in it for the long haul. Would things have been different if I’d been able to help them get an in-control software process established a few years earlier and extensible software architecture? Maybe, but it would have been an uphill battle.


No Comments


  1. Another Day, Another $440 Million - Robert Binder's Blog and Professional Portfolio

Leave a Reply

Comment moderation is enabled, no need to resubmit any comments posted.