I have a good friend who is in the IT Project Management area. He called the other day to say that his update was fully successful, having been distributed to over 1 million pieces of hardware without one call in. I congratulated him and he felt totally satisfied that he had accomplished something pretty fantastic. I want to believe that he did.
I did not want to tell him that his euphoria is probably misplaced. You see, according to “the law of averages” (in other words statistics), if you have no comments from a distribution of 1 million users, there are several issues:
- The call center ignored the calls concerning any problems with the software update
- There were no calls because people thought the problems would clear with time
- The call center explained that there was a new software update and the client should give it 24 hours or so to clear up
Of course, this is only three of many possible alternatives; and this is why I think this way. As a practicing statistician, I truly believe that outliers exist and should be noted and studied. In the past, outliers have been ignored and, as a result, calamity occurred. Just a few examples:
- Someone wins a major championship 7 times in a row (big outlier and never studied)
- The levies in New Orleans are not upgraded because it is not a good cost benefit analysis number (never questioned, and never fixed)
- A million users (or pieces of hardware) are upgraded and NO ONE has a complaint
What we fail to do, as people and especially as IT professionals, is to question good fortune. I have heard many an IT person (including me) talk about “if it ain’t broke, don’t break it.” Instead of rechecking the software, we are satisfied with knowing that no one has complained. We do this because we are busy and onto the next project, upgrade, etc. and do not have time to rehash an old upgrade. I understand those issues and do not blame the project team. Who I do blame is the data analyst, who should be studying this anomaly and try to figure out why this upgrade is so “good.” By not studying these outliers, we are denying ourselves the real analytic questions and, consequently, not really exploring the very nature of data analysis.
What will happen as a result of not studying this issue? Maybe nothing, or maybe the next upgrade will see a landslide of issues, some from the current upgrade and some as a result of small problems with the previous upgrade that was ignored because they had “no complaints.”
As an IT specialist for the Federal Government, there were times when I had to predict when outages would occur and one time, from my data analysis, I found that it was during a particular holiday. I briefed the executives on this and that holiday happened and there were no outages. None. The executives came to me and challenged my analysis. They said I did not do a good job of analyzing the data. The senior executive came to my rescue and asked how many more people the executives had placed on call for that specific holiday. All the executives were silent. They had put more people on call to fix any problems before they became critical. The result: fewer outages and more continuous operations. So it was not the data that lied, it was the ability of people to recognize the problems BEFORE they occur and manipulate the data prior to the event.
That is what must happen, even when the numbers are favorable. A bit of skepticism is okay. Check out the numbers and check the call center. If there were truly a million pieces of hardware that did not have a problem, then start figuring out how to give that project team a raise. Also, make them understand that no complaints is now the new standard which they must attain every time. I guarantee they will WANT you to check the numbers in that case.
Nothing is perfect. Nothing. It flies in the face of human endeavors and certainly in human history. But it is proven and reliable and when something is perfect – it needs verification. Or at least validation. More on this in later articles. Learn, Offer, Value, and Educate (LOVE).