How Does Big Data Fit in Your Organization

Why Big Data?

We are witnesses to an ever evolving technical landscape. New buzzwords are popping up every day. One of the latest, Big Data, has some organizations excited and other organizations scratching their heads. At the end of the day, each new technological advancement attempts to replace what we, as critical thinkers, have been doing for millennia. The difference is in terms of scale. Big Data Analysis attempts to allow the corporate soul the ability to recognize patterns and react accordingly (critical thought). Individually, we do this on a daily basis. Some of us have a bit more bias in our pattern recognition, but we all have the ability.

Envision a small consulting firm with a portfolio of 10 clients. They most likely will not need to analyze Big Data to know how to best serve their clients. In the firm’s owner’s mind, he or she can remember each personal interaction he or she has had with clients. This memory enables the firm to tailor their service to each customer. They have the unstructured data all over the office in the form of paper files, invoices, bank statements, emails, texts, phone records etc. They are already doing unstructured data analysis on a very small amount of data.

Now, if they crank up their business 100 fold, the proprietor’s entire mind will be engaged in managing the business. The proprietor will have to break up his functions across many employees. "Psychological functions of knowledge and judgment once owned by the individual now are taken over by parts of the organization. Feeling and emotion are exiled. Modern psychology saw an active, self-making, language-creating individual. In the factory or bureaucracy, major functions of the psyche—mastery by ego and conscience by superego—are ripped out and transferred onto organizational structures." [1]  In other words, the ability to critically think within the confines of one mind disappear. This new business organization has increased their profits significantly, but they have stopped their unstructured data analysis. They have begun to treat each customer in the same manner and have stopped looking at patterns in the data. Instead, they have begun to "identify data points for reporting and retrospective analysis." [2] Their customers have noticed the loss in service, but they seem to be alright with it because there is some cost savings.

This opens a new market for the firm’s competitors. A small company can start at the beginning with tailored customer service. The small company can analyze and react to keep customers happy. Without Big Data Analysis, the large firm is helpless to react to moving customer trends.

What holds firms back from Big Data Analysis?

So, what is preventing larger firms from building enterprise level critical thought (Big Data Analysis) into their process?

First, the simple answer is fear. What are they fearing? Well, the layers of management have a prescribed process for their day to day operations. When the CMO comes to the CEO and says, "We need to start reacting to our data" the CEO know that is an operational issue. Mark A. Beyer of Gartner states, "Big data is more focused on process analysis and predictive analysis supported by pattern recognition and alternative interpretations. Yet, most big data efforts are starting with the CMO instead of the COO. Here we go again." [2] Does the large firm have flexible operations that can react to data quickly? Beyer also states, "Tracking that latency between data analysis and process change and true or bottom line effect on the business is where we have not done that great of a job."[2] Why would an organization even look into big data if they do not have the operational agility to react to the data?

Another fear that organizations have in implementing Big Data analysis is they will have to throw away all of their existing analytics investment. This fear seems to be subsiding as Gartner through its inquiries has shown that organizations that were going to completely replace their analytics environments over the years 2010, 2011, 2012, and 2013 have dropped from 17%, 11%, 7% and 3% respectively [2].

How do I bring Big Data into my company?

As a potential data scientist or big data analyst, how can you prepare your organization to be a big data consumer? The first item to tackle would be to ask the executive team, "How will you use the data?" If your data is not used at the C-Level, your project may look like the failure. If there is an appetite to have predictive data and to adjust operations accordingly, is the executive team willing to hire resources externally and internally to make sure the project succeeds?

According to Gartner, most companies prefer a mix of external and internal staff on the analytics team. Small companies and large companies tend to have an even mix of external and internal, while mid-size companies lean more on internal staff.

Source: "How Does Your BI Team Compare to Those of BIXY Award Nominees?"

However you decide to implement Big Data Analysis into your organization, it is key to remember that there is no central brain in a bureaucracy. "Corporate Critical Thought" can only occur if there is buy in from the executive level. It is a combination of stakeholders that you must persuade. It does you no good to sell the CMO and CEO on your ideas if there is a powerful COO and CIO in the way of your success. The same thing that made the bureaucracy profitable, tends to be the same thing that makes it less agile.

[1] The Oxford Handbook of American Bureaucracy (Page 333)
[2] The Big Data and Advanced Analytics Dream Team