Digital Analytics - University of Utah: Analytics

Tuesday, February 18, 2014

Improving Data Quality

The Problem

Data Quality is something that has been, and will continue to be, of utmost importance to organizations and their analysts because analyses are only as accurate as the data with which they were created. The problem is that data quality is currently getting worse. Worldwide, the amount of inaccurate data has risen from 17 to 22 percent. In the U.S. specifically, organizations believe that up to 25 percent[1] of their data is inaccurate. This means a quarter of the data companies use to make decisions could be misleading them. So why is this happening?

Three leading causes:

1. More Streams: Today companies are getting data from an ever increasing amount of sources. In addition to the traditional web traffic we now have access to mobile, video, social media and GPS, etc. Gathering and making sense of all this data has led to quality issues.

2. More People: As Internet access continues to expand globally we now have all the streams of data mentioned above, but now they are coming from hundreds of different countries. A simple example of this would be dates. Different countries store and display dates differently. So understanding which country the dates are coming from could make a huge difference in analyzing the data.

3. More Data: As more and more people begin to use more devices for more things there is inevitably going to be more data. Many companies don’t have the bandwidth or ability to make sure all of this new data is up to the necessary standard.

How are companies addressing this issue?

Here are a few of the many possible solutions for improving data strategy:

Fix the data at the point of capture: The first thing a company should do is make sure the data they have control over is always as clean as possible. The easiest way to get good customer data is to have the customer input it correctly when they create it. To accomplish this, the company needs to know in what format they want the data (i.e. Do they want addresses broken up into individual pieces or in a single entry?). Once the business need is understood, validations can be put into place to ensure the format is correct.

Conduct source system data assessment: Running an analysis of the current database/source system can give a company an idea of the current state of the data. If there are missing or invalid values there is a good chance that there is an issue with the data at the point of entry or somewhere else in the process.

Text Mining: For data where the input cannot be controlled (i.e. social media data) text mining can be a helpful tool. Text mining can parse through all of a company’s data and replace a set of variations of words with standardized terms. This can ensure that more of a company’s data is in the correct format.

Conclusion:

There are many data related challenges for online companies. As tools inprove, companies will be able to better handle the diverse streams of data. However, as more and more people begin to access a companies website it is vital that organizations are smart about how they are keeping the quality of their data high.

*for more insights on how to improve data quality go to http://dataqualitypro.com

_________________________________
[1]http://cdn.qas.com/us-marketing/whitepapers/unlock-the-power-of-data-2.pdf

What's the Big Effing Deal about Big Data?

Many of us working in technology have heard the rumblings and high and mighty boasts about using big data and how it can transform your business, create new opportunities, increase efficiency and usher in a new epoch of world history that will bring about world peace (alright, I made up the last one, but you get the point, big data has been mentioned as a prescription for many problems, but is it really the answer?) The question is, what is Big Data, what are its challenges and how can Big Data ultimately help you to better your business’s bottom line?

What is Big Data some of you may wonder?

Big data is “a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.”[1] Essentially, with the improvements in data storage devices, coupled with their decrease in price and the explosion of both structured and unstructured data during the Web 2.0 movement, new streams of data have been created that can now be analyzed for actionable insights into areas like customer behavior to allow businesses the ability to better market and target their core customers. Unstructured data has provided the biggest surge in the supply of data and is basically “data [that] is heterogeneous and variable in nature and comes in many formats, including text, document, image, video and more.”[2] The ability to use tools to make sense of all of the unstructured data has completely changed how we actually store the data.

Big data has proven to be a challenge to organizations wanting to implement a system to capture and make sense of all of it. Big Data is challenging for several reasons:

1) The three V’s of data: volume, velocity and variety[3]. Although this coinage has been adopted throughout Big Data discussions on the Web, it was first described by Doug Laney in a paper he wrote back in 2001 on how organizations must create information management systems to cope with these three aspects of data. Volume: the increase in depth and breadth of data that is collected. Velocity: the increase in speed of data that is being generated and hence needed to be stored. Variety: structured and the great increase in unstructured data that does not fit neatly into a typical relational database.

2) How to process the data into information. Much of the data being collected, now unstructured, does not fit neat into a relational database where fields must be defined for length and data type being stored. The rise of NoSQL databases, pioneered by many of the biggest names in technology such as Google and Facebook, has taken precedence as the best way right now to store unstructured data. Although no standard definition has been officially created for NoSQL, it is essentially a database that adheres to the following characteristics[4]:

i. Not using the relational model

ii. Open source

iii. Designed to run on large clusters

iv. Based on the needs of 21^st century web properties

v. No schema

This different type of database requires professionals with different a skill set than working with standard relational databases and the cost with hiring/training professionals to implement and maintain a NoSQL database must be considered when choosing whether or not to pursue a big data strategy for the business.

3) What to do with the data. This is the last and from my perspective, the most important challenge that must be addressed.

As the cartoon above illustrates, the ultimate goal of Big Data is to drive actionable insights which were not available before and help remove things like bias, from decision making. Most of the business value in respect to Big Data comes from the analysis and knowledge that is created, not from the stored data itself. It is not enough to just efficiently and effectively store the data that your business generates, but more importantly are the key decision makers in the business ready to listen to what the analysts tell them and prepared to take action based on this knowledge.

Big Data can truly be a game changer for your business. I believe it can benefit all industries, but some areas like computer and electronic, finance, insurance and government will see greater benefits.[5] I think much of the productivity gains to be realized by effectively using Big Data and acting on the insights gained from it has currently only reached the tip of the iceberg. This is where the right-brain people will get excited: what new creative insights can we gain from the data and where can we creatively apply Big Data analytics to solve problems? I think we are only limited by our own imagination and that is what excites me so much in this field—the realm of possibilities is endless.