Improving Data Quality

The Problem

Data Quality is something that has been, and will continue to be, of utmost importance to organizations and their analysts because analyses are only as accurate as the data with which they were created. The problem is that data quality is currently getting worse. Worldwide, the amount of inaccurate data has risen from 17 to 22 percent. In the U.S. specifically, organizations believe that up to 25 percent[1] of their data is inaccurate. This means a quarter of the data companies use to make decisions could be misleading them. So why is this happening?

Three leading causes:

1.       More Streams: Today companies are getting data from an ever increasing amount of sources. In addition to the traditional web traffic we now have access to mobile, video, social media and GPS, etc. Gathering and making sense of all this data has led to quality issues.  

2.       More People: As Internet access continues to expand globally we now have all the streams of data mentioned above, but now they are coming from hundreds of different countries. A simple example of this would be dates. Different countries store and display dates differently. So understanding which country the dates are coming from could make a huge difference in analyzing the data.

3.       More Data: As more and more people begin to use more devices for more things there is inevitably going to be more data. Many companies don’t have the bandwidth or ability to make sure all of this new data is up to the necessary standard.   

How are companies addressing this issue?

Here are a few of the many possible solutions for improving data strategy:

Fix the data at the point of capture: The first thing a company should do is make sure the data they have control over is always as clean as possible. The easiest way to get good customer data is to have the customer input it correctly when they create it. To accomplish this, the company needs to know in what format they want the data (i.e. Do they want addresses broken up into individual pieces or in a single entry?). Once the business need is understood, validations can be put into place to ensure the format is correct.

Conduct source system data assessment: Running an analysis of the current database/source system can give a company an idea of the current state of the data. If there are missing or invalid values there is a good chance that there is an issue with the data at the point of entry or somewhere else in the process.

Text Mining: For data where the input cannot be controlled (i.e. social media data) text mining can be a helpful tool. Text mining can parse through all of a company’s data and replace a set of variations of words with standardized terms. This can ensure that more of a company’s data is in the correct format.

Conclusion:

There are many data related challenges for online companies. As tools inprove, companies will be able to better handle the diverse streams of data. However, as more and more people begin to access a companies website it is vital that organizations are smart about how they are keeping the quality of their data high.      

*for more insights on how to improve data quality go to http://dataqualitypro.com

_________________________________
[1]http://cdn.qas.com/us-marketing/whitepapers/unlock-the-power-of-data-2.pdf

Sarcasm: A Challenge in Sentiment Analytics

We've all been there, right? Somebody misinterpreted the email we sent, Facebook status we secretly thought was hilarious, or Twitter feed that was meant to be witty and non offensive.  Recent exploration into sentiment analysis through social media websites has been huge in understanding the general population's "temperature" when examining issues such as the presidential election, product reviews, drug side effects, movie reviews, and stock market behavior.  Social mediums provide an ideal central hub of sentiment data waiting to be mined by companies interested in feedback of new products.  Many companies have started to explore user generated content on social media websites by using text analytics.  Emotion is so easy gauge online, right?

How are statements categorized as positive or negative?

One approach to categorizing statements is by using polarity, the simple, binary, black and white, good or bad, assignment of a statement.  A study by Peter Turney[1] identified three steps for an algorithm to classify an online review as recommended or not recommended.

1. Use a part-of-speech tagger to identify phrases  in the input text that contain adjectives or adverbs.
2. Estimate the semantic orientation of each extracted phrase by calculating its similarity with a positive word ("fantastic") or negative reference ("poor").
3. Assign the review to a class, recommended or not recommended.

According to Turney, 74% of reviews were classified correctly using these steps when compared to reviews evaluated on Epinions. As you have probably anticipated, the process becomes increasingly complicated.  The above mentioned process may work for a black and white world, but what about the 50 shades in between? Pang, Lee, and Vaithyanathan [2] used three machine learning algorithms, Naive Bayes, Maximum Entropy, and Support Vector Machines to categorize movie reviews into positive, negative, and and varying degrees of strength between the two categories.  They further identified challenges in sentiment analysis as some writers use words indicative of the opposite sentiment to describe their emotion (i.e. "The plot is such a mess that it's terrible, but I loved it.").  Another method of assigning meaning to opinions is through natural language processing in which words are assigned a varying numeric value to determine the strength of the emotion.  In a study by Thelwall et al.[3], a training algorithm optimized sentiment word strengths, assigned greater numeric values to subsequent words that "boosted" the emotion, factored in punctuation, and eliminated negative emotion in questions to better optimize a graded scale for determining levels of emotion.  Many statements consist of subjective and objective statements and many algorithms have been developed to filter out objective statements to gain insight on the context of subjective statements. In short, there are many factors that contribute to detecting the emotional sentiment of a phrase, and while many algorithms have been developed to recognize adjectives and adverbs contributing to the emotional temperature of a phrase, in many cases it is difficult for a human reader to decipher the sentiment. And I bet you just love a witty sarcastic joke at your expense.  (I know I do. For real.) 

Are computers humored by irony and sarcasm?

One of the challenges in a computer identifying sarcasm, is that people often disagree the boundaries of sarcasm and irony as some of the meaning is based on context and situational factors.  Filatova [4] attempts to computationally unravel sarcasm and irony by first understanding the terminology.  The literal meaning of an ironic statement "echoes" an expectation which has been violated.  Dews and Winner [5] define sarcasm as a special case of irony, "Ironic insults, where the positive literal meaning is subverted by the negative intended meaning, and will be perceived to be more positive than direct insults, where the literal meaning is negative."  Automatic irony identification focuses on the following features to signal irony: emoticons and onomatopoeic expressions for laughter, heavy punctuation marks, quotation marks, and positive interjections.  Filatova also mentions users using hashtags #sarcastic #sarcasm to identify sarcastic Twitter messages (I personally feel the sarcastic value declines with the addition of a #sarcastic).   Because sarcastic phrases are dependent upon the surrounding context, algorithms often focus on the document level as well as the text utterance of a sentence when evaluating sarcasm.  Often a sarcastic statement is ultimately classified as negative but must be carefully analyzed as it may contain positive sentiment to convey the message. 

Sarcasm will continue to be a challenge for text miners hoping to evaluate population sentiment because "computer programming follows strict rules, while natural language, particularly the inside-joke culture of the Web, doesn't." [6]  And who doesn't love a good inside joke?

References
[1] Peter Turney (2002). "Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews". Proceedings of the Association for Computational Linguistics. pp. 417–424.
[2] Bo Pang; Lillian Lee and Shivakumar Vaithyanathan (2002). "Thumbs up? Sentiment Classification using Machine Learning Techniques". Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 79–86.
[3] Thelwall, Mike; Buckley, Kevan; Paltoglou, Georgios; Cai, Di; Kappas, Arvid (2010). "Sentiment strength detection in short informal text". Journal of the American Society for Information Science and Technology 61 (12): 2544–2558.
[4] Filatova, Elena. "Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing."
[5] Dews, Shelly; Winner, Ellen (1995). Muting the meaning: A social function of irony. Metaphor and Symbolic Activity, 10(1):3–19.
[6] http://online.wsj.com/article/SB10001424052970203335504578088763796519732.html