Big data has been taking the world of technology by storm. It is a word that has entered every meeting and memo promising magical insight into the working of the business. The reality however has exposed numerous pitfalls with regard to big data overriding expertise. Is the concept a hit even if it does not lead a company any closer to making the right decision?
Too little versus too much information
Before the debut of the concept of big data, decisions went wrong because they were based on too little information. Unfortunately decisions made today are based on too much information and they still are just as wrong as before. It has become what is known today as ‘stupid data’.
The data is referred to as ‘stupid’ as it is either misunderstood or misused. Data by itself cannot be stupid or anything else for that matter. It is the human involvement that gives it characteristics. The review process becomes overwhelming and therefore misunderstood or misconstrued.
Most industry experts seem to pin the unreasonable expectations behind big data as the main culprit for the birth of stupid data. The endless set of promises for magical understanding set the whole idea up for failure. Understanding big data for what it really is can be the only way to let it reach its full potential while circumnavigating the pitfalls.
Here are the major pitfalls that commonly sink the big data ship.
Paralyzing amount of information
A larger set of data does not necessarily translate into a better decision making process. Instead the opposite is often true. Larger sets of data can make accurate evaluation that much harder making good decisions difficult to come by.
Alvin Toffler in his book “Future Shock” written in the 1970s spoke about the effects information overload. The concept of the paralyzing effect of too much information, however was not grasped till the advent of big data and consequently stupid data.
The fact remains that raw data itself cannot answer any decision making questions. A careful evaluation of the facts surrounding the data and then the data itself is the magic equation to getting the right answers. Continuing in the same vein then, as the data grows in size, the evaluation process should also grow in terms of sophistication and clarity. The approach to the data set makes a world of difference. Drawing unnecessary conclusion is a major pitfall that should definitely be avoided. Trends, correlations and hard questions are the answers in many ways to avoiding the pitfall of drawing wrong decisions.
Wearing blinkers during the analysis
Pure data cannot be the only raw material in making decisions. Team experience is often discounted wholly by novices in favor of pure data. The experience of the team as a whole and as individuals will offer real world scenarios which can offer insights that data is simply incapable of providing.
While collective experience is important, the gut instinct also has value in business decision making. Both experience, data and instinct should be collected, measured and investigated in equal terms.
The leadership team in businesses often have experienced veterans and the answers for many business questions are already there in the minds of the team members. Effective discussions between the team members with the data set will lead to informed decisions based on both the data and the expertise. Data can be used to settle disputes but relying on it solely is not a good idea.
Uncertainty and the decision making process
Data alone does not eliminate uncertainty. While it does reduce significant amounts of uncertainty, it is not a magic wand that can be waved at any time to eliminate the unwanted stressors.
Most of the unrealistic expectations on big data stem from the fact that it can eliminate uncertainty completely. It can also lead to frustration, letdown and company paralysis that can be debilitating in a business world that is constantly on the move.
This frustration and debilitating paralysis was IBM’s experience before the turn of the century when the company turned its fortunes around to everyone’s surprise. Lou Gerstner spoke about the toxic culture at IBM in a Forbes article and was quoted as saying that IBM was struggling with “studying things to death” and “obsessive perfectionism”.
The human experience is not studied or understood as much as it should be. This becomes a problem in a scenario where experience is pitted against data. The business world is complex and often imperfect. Therefore all decisions will involve a measure of uncertainty which can never be discounted completely. The best decision makers or the ones who are most successful will recognize and understand the uncertainty in their decisions, reduce them as much as possible and move to the next decision. This is the only way to navigate around the pitfall of uncertainty.
Data is a means to finding the answer and not the answer itself. In order to make big data truly big, one has to use it in an effective combination with team experience, instincts and look toward data as a guide. The combination is one that will allow the business to control the data instead of letting the data control and run the business. Instead of shying away from big data or making it stupid data, businesses should use the data to their advantage and make informed decisions.