The Implementation Problem Of Big Data

What is “Big Data”

One of the fundamental problems with “Big Data” is the fact that there really isn’t one unanimous definition for it.  Ask 30 IT professionals to define it and you’ll get 30 different answers.  As you can see in this graphic by Intel, big data is not only involved in the amount of data but can incorporate complexity of the information and the speed at which that data is needed.[1]

For the simplicity’s sake lets use the definition that was attempted by Jonathan Ward and Adam Barker at the University of St. Andrews in Scotland who attempted to gather all of the various definitions of “big data” and put it under a collective roof.[2]  They define big data as

“Big data is a term describing the storage and analysis of large and or complex data sets using a series of techniques including, but not limited to: NoSQL, MapReduce and machine learning.”

So now what?  Even if we have a clearer picture of what big data really is (which one could easily make the case that we are lightly skimming over) what good is obtaining all of that information and using all the firm’s resources when you can’t properly utilize the information gathered? 

It is fairly evident that big data has the potential to transform firms, improve decision making and ultimately improve productivity within a firm.[3]  More and more companies are catching onto the notion that many of the problems from the “old school” of thinking, utilizing “hunches” and going off our “gut feel”, often leads them falling into the human biases and heuristics that have led to some of the biggest business mistakes in history.  

So it’s no surprise that companies are pouring millions of dollars into ramping up their IT and Analytic departments to gather vast amount of data in helping them make more informed decisions.  But what’s the point of this endless collection of data if the majority of the information never gets used or used in a meaningful way that leads to actionable decisions being followed through?  

Big Data Implementation Issues at its core, is a Personnel Issues

One of the major reason firms are finding it difficult to implement the data is due to the shortage of qualified individuals who can interpret said data.[4]  Without that, big data just becomes yet another data set that executives still end up using their “gut” or “intuition” as the basis for their decision making process.  Hiring data scientists who have a background both in statistics, computer science along with some functional expertise is hard to come by. It’s not a technology issue, it is in a large part a personnel issue. 

Data Silos

Most firms are set up to collect the data points and store them in their various departments.  Without proper planning, most data that becomes collected paints only a portion of the full picture.  This is yet another implementation issue that is rarely seen through.  Many companies will start the process of collection without having the end results in mind.  To fully utilize the impact of big data, it can’t be skewed due to the lack of availability or transparency between the functional silos.

Actionland

Many individuals who work with web analytics are familiar with Brent Dykes and his teaching of moving from Setupland to Actionland.[5]  Many of those same reasons why companies fail to move on web data can apply to big data in general.  Individuals who present data are a dime a dozen and in fact are merely a third party by which a software or other form of technology interacts with to get to the end user.

What is needed are more data scientist to who will produce actionable insights from the raw data collected into concise, coherent and actionable suggestions that executives can use to better make their decisions.  This is key in realizing the potential that big data brings to firms.  Again at its core, this is not a technological issue but rather a personnel one.  

References:

[1] http://www.intel.com/bigdata

[2] http://arxiv.org/abs/1309.5821  Undefined by Data: A Survey of Big Data Definitions

[3] http://www.kaushik.net/avinash/big-data-imperative-driving-big-action/ 

[4] David Mielach.  "Shortage of Analysts Challenges Big Data Implementation" Business News Daily, Jan 23, 2013

[5] http://www.analyticshero.com/2012/06/05/web-analytics-moving-from-setupland-to-actionland/