If you’re looking for commonality between any two entities that appear to be at complete odds with each other, look no further than our nation’s political process. Typically, both sides of the aisle will agree that something is indeed wrong, however, their solutions may be on the opposite ends of the spectrum.
Do we fix an issue by regulating it because there is no oversight or do we deregulate it because it has too much oversight? No one can seem to agree on the approach, so we end up in endless arguments over right and wrong and the problem doesn’t go away. Usually, it gets worse.
In this vein, we have a similar issue with competing methodologies and solutions in the IT industry. Nothing aggravates experienced IT professionals more than walking into a new network where incredibly poor choices were made.
The ensuing cleanup to ensure uptime, longevity and performance can cost a company a lot of time and money even if they’ve just purchased relatively new equipment. This article is a guide as to why poor choices are not only made, but also defended, in the IT industry and how it can lead to IT implementation failures.
The Setup
Before we dive into the reasons, let’s set up a fictitious company that is actually based on a scenario my colleagues and I walked into when we were called in for a possible breach and other very poor decisions.
We’ll call them “The Decepticons Inc.” because it really fits and why not!? The Decepticons make “Energon cubes” that are proprietary to them, and thus, they own quite a bit of intellectual property and have it stored on their on-premises server farm.
When we were called in, The Decepticons were experiencing severe performance issues and weren’t sure why, since they recently purchased newer servers and firewalls. Through our investigation, we found out that the Russian mafia had hacked into the server farm of The Decepticons and were slowly uploading all of their intellectual property to Russia.
After fixing the breach by temporarily dropping in our own firewalls, (which we knew were not compromised), we went to work trying to figure out just how this happened and how their network was fully configured. What we found was staggering.
Choices in equipment that were inadequate for the performance needs of a network that big, configurations that were both incomplete (aka just enough to get data moving but not protect it) and completely indicative of novice or inexperienced thinking.
The Decepticons had a C-level executive, we’ll call him Starscream, that not only fancied himself the CEO (he wasn’t) but also took it upon himself to drive this initial configuration by reading about cool new technologies and then trying to implement them through a process we can only refer to as Frankensteining.
In our consultation meetings, Starscream defended his choices for the network and how they were configured. While Starscream was technically correct in that everything on the network worked in terms of user uptime, he was missing the forest for the trees in terms of security and performance. So with this primer in mind, let’s discuss why IT personnel disagree on solutions to the same problem!
Ignorance is the Mind Killer
To start with the first obvious issue, many IT personnel simply do not know how to properly configure equipment for a full or complete setup. Further, many are not aware of other competing equipment that could be superior and better performing.
In our case above, Starscream chose a firewall that did not have the performance to handle their user load and session count but also did not have adequate defenses. The firewall had no Unified Threat Management (UTM) capability, nor any kind of advanced Intrusion Prevention System (IPS) that would have vastly mitigated the chances of the Russian mafia breaking in.
In our discussions with Starscream, he disclosed that he was not aware of these capabilities nor had he gone looking for available options. This underscores a major problem experienced IT professionals have when working with new or prospective clients: Our job is part educator in this respect and it’s a tough hurdle to clear when having to tell the client that their expensive firewall doesn’t really protect them in the way it should.
These conversations lead to anger, usually directed at the previous IT “professional.” In this case, the CEO of The Decepticons, let’s call him Megatron (obviously), really let Starscream have it for not doing his homework in picking adequate defenses for the network. This company’s financial loss was massive due to the trust put into an ignorant individual.
Complacency Doesn’t Help Either
This is a major problem in the IT industry. IT professionals learn the technology they’re used to and then fail to adopt new or better technology for their customers or companies. In essence, they get pigeonholed into the same cycle of hardware and software, fall behind in the ever-changing world of technology advancements and, most importantly, the threat landscape that has already surpassed their level of technical knowledge.
This is most prevalent in non-IT companies with their own in-house IT employees. The company will not invest in newer technology, opting to use the “if ain’t broke, don’t fix it” model and, as a result, they fall behind. Unfortunately, so does the educational standards of their IT staff. We see this all the time.
Take Starscream, for example. In this particular case, he was a big fan of older Cisco PIX and ASA models, pre-UTM. Because his company invested no money in education and because he knew these technologies well, when it was time to select a new firewall, he stuck to what he knew. He couldn’t, or refused to, see that there were a slew of Next Generation Firewalls (NGFW) that would protect The Decepticons much better than a traditional ASA.
Due to lack of education on the latest Cisco operating systems, Starscream configured the new ASA incorrectly and opened up several security holes, which left critical servers in the infrastructure open to attack. To completely seal the deal here, he also didn’t keep his server farm up to date. As well, several core servers were running on older, unpatched operating systems that were running on an older version of VMWare with a hypervisor that had the default password attached to it. Needless to say, Megatron wasn’t happy.
Denial of a Better Solution is No Way to Go Through Life, Son
When these issues were brought to light, Starscream went into full CYA mode and began challenging that there was, in fact, better technology and methodology out there for The Decepticons to use despite all available evidence to the contrary.
We also see this all the time as well. It’s Ostrich Syndrome for IT. The sad part is that many companies will take the word of their employee or colleague over the more experienced IT professional they hired to enumerate the problems in their network.
In fact, recently, I had this exact scenario. After performing a security audit and finding an incredibly poor configuration by their existing IT contractors, we had a big meeting with everyone. In this meeting, I pointed out that while a decent UTM firewall was purchased, it actually wasn’t licensed and running with any UTM functions at all, basically making it an expensive router anyone could buy at Best Buy.
We even took screenshots of the configuration to show the client, because we couldn’t believe it ourselves. Their existing provider said that we were lying. Then he said they were actually routing this company’s traffic through their own UTM, which was also a lie since we reviewed their entire configuration and could prove this wasn’t the case.
Then they blamed their own customer by saying they actually declined purchasing the licenses due to cost! This last one their CEO actually believed, even though his staff was disagreeing with him in the meeting in front of everyone. Needless to say, this company did not leave their current provider and it’s honestly to their detriment. I told this CEO that even if we never spoke again, he needs another IT provider stat.
It’s sad because I know they’re going to be hit and it’s going to cost them quite a bit of money to recover from it.
Okay, Okay…Fear is also the Mind Killer
This last point shouldn’t be confused with ignorance, complacency or denial, though it definitely feeds into each of these. When you have a person like Starscream, who exemplifies all of the above, it’s important to understand the motivation for such reactions and positions when confronted by the reality of the situation.
Starscream may be a nice person and mean well, but he secretly realizes he has been outclassed; for whatever the reasons, his knowledge and capability haven’t kept up with the evolving industry of technology. He doesn’t want to lose his job, and I have no desire to displace him, providing his company is willing to train him to adopt the correct technology to keep them current and safe.
This is, sadly, another stance we see all the time. Instead of approaching the situation in a collaborative manner, the person will get defensive and close off their thinking to anything that may challenge the domain they’ve previously ruled. No one is perfect, not even me (though I do, on occasion, like to think so). In this vein, I approach every technology challenge as an opportunity for learning and collaboration. We shouldn’t mind others disagreeing with us, providing their positions are supported by logic and reason. We do ourselves, and the people we help, a grave disservice by failing to learn and grow or by shunning more effective solutions in the name of ignorance, denial, complacency or fear.
Pro versus Pro… Fight!
I’m sure by now many of you are asking yourself; “Hold on, now. What if you have two IT professionals offer two entirely different approaches to solving an issue?”
That’s a very valid question! This does happen from time to time in the IT world, because when it comes to philosophy on how to achieve success, there are many different paths. Do we virtualize an infrastructure or go with physical servers? Do we build a traditional server farm or do we hyper-converge? Virtual hosting or containerization? The possibilities seem endless and sometimes there is no incorrect answer.
In cases like this, it really comes down to two major factors: experience of the professionals and a good, old-fashioned pros and cons list of sorts. Professionals that have experience with physical servers, but not virtual, will obviously be inclined to choose a configuration using physical servers. Same with a professional who knows virtual more than physical.
Both solutions will ultimately be optimized by said professional, however, without that pro/con analysis of each solution, options and features can be overlooked or missed, to the detriment of the customer. This is why I highly recommend any company looking for solutions find a consultant that is well-versed in multiple solutions to help illuminate the right path for the company. This is perhaps one of the most critical functions I play for my clients and customers since it will directly impact their ability to grow and scale for the long term.
You will note that I did not add “pride” as a reason why IT professionals may disagree. Pride, in and of itself, isn’t really a bad thing. However, it goes out the window when serious problems crop up. It’s hard to take pride in the network you built when it’s crawling with crypto infections thanks to a poor choice in defense.
At the end of day, we’re all human. No matter what the issues are and why we may approach problems differently, the ultimate goal is to fix any IT implementation failures and do it in the best way possible. Let’s just hope the most experienced and logical solution is the one used!