It’s Not Complicated

It was 2:45 PM, I had just walked into the office after being out at appointments all day. The team I was working with at the time had a major issue: a customer’s main software was down and their server was doing funky stuff.

One engineer had in-depth diagnostics pulled up that showed that, well, everything was broken. Another engineer was overseeing the work but didn’t have much else to add.

After a few minutes, the issue was escalated to me.

One of the things I’ve learned troubleshooting problems in life, tech, personal, financial, IT support services and managed IT services, anything , is that when everything appears to be going wrong there is usually a single root cause. If it’s foundational, then everything will break and it will appear very complicated.

I quickly noticed a file on the desktop. Its name implied that a team prior to us had been working on the server and turned off a Very Important Feature.

I turned Very Important Feature back on again.

Five minutes later everything was up and running and all of those complicated diagnostics were showing fine again.

Takeaways

• Simple root causes underly problems that appear complicated. I knew to look for the simple thing because with so many things broken there had to be a foundational issue. In relationships, its often simply that two people aren’t communicating, in finances revenues don’t match expenses. There is an inverse relationship between the number of things on the radar and thing causing them.
• Observation beats technical know-how. I wasn’t the smartest guy in that room, that was the other engineer. I wasn’t the most experienced, that was the one watching him. I was just the one to stop and look around at the situation before diving headfirst into the weeds.
• Bad decisions multiply. I still don’t know who turned off the Very Important Feature. My guess is that either the customer did or a vendor in a related space to ours. Either way, the decision to allow a team less experienced team than us take the first crack at the issue cost this customer more money than if they had called us in the first place.
• Culture matters. The first two engineers did the exact right things in escalating the issue as rapidly as they did, which ultimately resolved the issue quickly. They both avoided target fixation and worked in a team that valued the results over who got the credit. The culture of the team produced a process that brought value to our customer.