Big Data Antidotes: Tendency of Big Data Pitfalls

To no surpise, data continues to grow at a faster pace. According to a recent Gartner survey, 73% of organizations have invested, or plan to invest, in Big Data, and 15% of firms have already deployed Big Data projects in production. The Big Data Landscape 2017 by Matt Turck listed about 500 products, vendors, and sources, which demonstrates the overwhelming madness in the market. However, more than 65% of Big Data projects failed, according to research. Now the question is why.

There are many reasons for the initiatives' failure, ranging from strategic, tactic, operational, to process, people, technology, and cost. Here are the top-10 list of common pitfalls:

1. Failing to clarify the need for big data within the organization 
2. Islands of analytics in the “Excel culture”, where departmental thinking predominates over looking at the big picture
3. Not enough investigation on vendor products by blindly taking the path of least resistance 
4. Repeating an old approach without realizing it is outdated
5. Not establishing enterprise ontology and definitions for “single version of truth” culture 
6. Considering the effort as a one-time implementation rather than a living ecosystem 
7. Lack of vision and not having a strategy with a clear organizational communications plan 
8. Building silo charts to answer limited questions rather than holistic dashboards 
9. Inadequate upfront planning by overlooking the development of governance and program oversight 
10. Failure to reorganize for big data 

The Big Data antidotes are antipattern remedies to the aforementioned predicaments, as a common response to recurring problems. The antidotes offer a common vocabulary for the similar themes and prescribe proven resolutions for the common obstacles. They leverage expert knowledge and design tradeoffs, making the solutions sound and practical. As a result, they provide stress release in the form of shared misery and traps.

More details are available in a practitioner's guide, which defines an overarching structure to classify the issues in three dimensions. The symptoms, consequences, root causes, and solutions are discussed in great depth. 

For more information, please contact Tony Shan (blog@tonyshan.com) or leave your comments below. 

  ©Tony Shan. All rights reserved. All standard disclaimers apply here.