Complexity Should be a Last Resort

I haven’t listened to this CD since I was in high school, but there are some parts I remember so clearly. Like standing in a long line at the bank and asking for change of a nickel.

George Carlin once quipped, “How often the simple solution will elude us.” George was making a joke as part of his “Keeping People Alert” bit from years ago, but he may as well have been giving a lecture at a conference called “Digital Transformation 2021” or something equally buzzy in the IT space. There’s something about modern IT delivery that seems to gravitate towards the most-complex, byzantine way to accomplish something, and I think that’s a real mistake that we in the industry need to actively correct for.

In order to achieve any kind of solution, you need to first define the problem. I feel like I’ve been in more than my fair share of meetings recently both in and outside of the office where complex “solutions” are being tossed around before the problem is properly or well defined. The challenge with that is that if you don’t clearly understand the business or technical issues, you will often develop a solution that either doesn’t solve the problem or solves it in such a backwards way that you’ll end up suffering with the downstream architecture or support problems for years – or in some unfortunate cases - decades. If a problem is well defined, a simple, efficient, and timely solution might become apparent sooner in the process.

I have found that the time it takes to understand a problem goes down rapidly when you try to figure it out around a whiteboard. So, if you’re pressed for time - draw!

I think one of the reasons that we waste time trying to solve poorly defined problems is that we don’t take the time to actually understand what the real problem we’re trying to solve is. Put simply, we often don’t know precisely what we’re trying to accomplish. This seems like an easy issue to avoid, but I have seen firsthand how time-pressure on a team can create genuine pressure to short-cycle the analysis phase of any engineering process. The irony of course is that by cutting out proper analysis, you end up spending substantially more time on the backend fixing what you should have built right the first time.

I mean, in a lot of ways what I’m saying is that the oldest adage in the world about building something still applies to modern technical solutions: you should measure twice and cut once. Only in this case, measuring means understanding what exactly you’re trying to do or what business problem you’re trying to solve. Often, if you understand the problem well enough and have a broad enough view there’s a straightforward solution. It’s not always technically simple (because if it was, we wouldn’t need nearly as many IT folks in the world) but at least conceptually, the solution should be easy to understand as why its set up or configured in a certain way.

As a silly example, I’m reminded of a time many years ago when I was an intern working as a (very) junior-level Oracle applications developer. There was a need to get some information about inventory at one of the warehouses to the manufacturing team to ensure that we were planning production appropriately. I got a quick overview of the issue and thought “I know exactly how to fix this problem,” and ran off to start building triggers, forms, and views so that I could get the data from the inventory tables to present into the manufacturing forms. I banged away on this for a day and a half until someone came over to me and asked why I still didn’t have that inventory report.

I was confused – saying that I was working on getting a live view of the data spun up so that the manufacturing folks could see it in real-time. (Please keep in mind that we’re talking about a year that started with a “19,” so this wasn’t nearly as straightforward as it would be in most modern apps.) I got a sideways glance, and then got a lesson that I like to think I haven’t forgotten to this day. The story I was told goes something like this.

Suppose that you’re an engineer and you show up on one side of a rather large valley and you are told that cargo has to be delivered to the other side. You have a background in bridge design, so you start designing and building a bridge that goes across, that way the trucks can get freely from one side to the other when the cargo comes. So, you start doing that, and months later you get the bridge mostly built. Unfortunately, someone comes by, notices the road, and helpfully tells you that the cargo is coming on a train. You do the calculations, and find out that with some extra reinforcing, the bridge might be able to handle the load of the train and extra tracks, so you start working on the reinforcements. Unfortunately, that’s going to take more time that you have, and ultimately the train shows up before you’re finished.

You scramble, trying to figure out how to finish the job quickly – and so you cut corners on the last part of the project – the final leg of the bridge is shoddy and will allow a train to pass over it only once before collapsing – but at least you can get the cargo across and then fix the bridge after the train has passed. In a mad dash, you finish the bridge, and tell the conductor that the train can cross. He looks at you a little funny and says, “this seems like a lot of effort to get this mule across the valley.” You realize that this whole time, the cargo is a mule – which can make the trip across the valley without a bridge, or a train, or anything else. It can just climb down and climb back up.

“Oh… just uh… just this mule, huh?”

You never bothered to find out what exactly you were supposed to do, and so wasted a ton of time. Even worse, you ran out of time and built a shoddy bridge at the last second that may or may not work for the trucks it was originally designed for, but even if it does will require significant maintenance just to stay standing.

If you don’t see how that relates to life in modern IT, then you either haven’t been around long enough or work for an organization that is running so perfectly at every level that they never run into any communication challenges. What I didn’t think to ask of my internal customers back when they needed that inventory data was what the ongoing need was. It turns out that there was already a process but it got messed up just that one month and so they needed a manual report. I didn’t understand their problem properly, and so attempted to provide something far more complex than the simple report they needed

I wish I could say that this was the last time I ever made that mistake – it for sure was not. I likely made this mistake this week and probably last week, too. My guess is that you have also done so. As technologists I think many of us have a natural inclination to see ways that we can apply new tools to a problem before we fully understand what we need to build. The vast majority of the time, this results in us building something larger or more complex than the situation or business actually requires. Just think about it – have you ever seen a developer start spinning up a fully redundant database cluster when you’re pretty sure that the business process in question would work okay with a simple spreadsheet? I know I have. Let’s not talk about all the downstream support that we’ll need to put around that new cluster. Patches and upgrades for life!

So, what do we do about this? How do we make sure that we’re building the most straightforward solution that meets the need? I think the most important thing is to spend a lot of time making sure that you know exactly what the business challenge is, what role current processes play in the issue, and what other processes or functions are dependent upon a solution. Once you understand this, examine the problem from multiple sides. Sometimes the best solution isn’t a technical one; for example, it may be redoing a business process or flow. What’s important is that you take the initial time to understand the problem in order to provide the most straightforward solution.

If your problem is how to turn the lights on and off, and this is your solution… you are part of the problem.

Of course, sometimes you really do need to build something complex. Occasionally, a new business challenge will require you to develop massive scalable platforms to deliver applications to millions of users. You can’t run Netflix on a smartsheet, after all. But not every problem is as challenging as how to stream an enormous amount of 4K content tuned to the preferences of millions of users to every continent on earth as efficiently as possible. Sometimes, you just need to get a mule to the other side of the valley. All I’m suggesting is that you put in the effort to know if the problem before you is a mule or a global content delivery platform.

My guess is that it’s somewhere between the two, and if my experience has taught me anything it’s likely closer to the mule.

Questions for reflection:

• What was the last project that you saw that was overbuilt versus the scope of the problem?

• Does your IT team spend enough time making sure they know the business challenge well enough to explain it to others?

• When you come across a super complex process that you didn’t design or implement, what’s your first reaction?