Predicting the Future with Digital Analytics

Think of the enormous amount of data that we are generating and collecting from text and tweets to web traffic patterns and search trends. A lot of our conversations occur digitally and leave some sort of trail that if analyzed could lead to an understanding and prediction of the future.

We could predict things like potential disease outbreaks to the outcome of elections and the direction of the financial markets. We could be moving closer to the sci fi movie Minority Report where we could predict crime before it even occurs.

If all that data could be dropped into a black box and analysed using past data and real time data what could we learn and predict? In a discussion with BBC, Eyal Gever a digital visionary describes a crystal ball app that could change the world.

Potential Problems

One of the problems we have is the processing power needed for all that data, but that’s exponentially excelerating. The next problem is being able to get to all that data and the privacy concerns surrounding it. And what of all the power that someone would have that holds that data and can accurately predict the futue or if that data could be manipulated for the wrong reasons.

A few companies are already leveraging our data to better the world and predict future events. Lets look at some examples of the power that can be found within all that data.

Data Behind Texting Could Save More Lives Then Penicillin

Nancy of DoSomething.org describes the data behind texting and the lives that could be saved, policies that could be made, the effects of a hatefull speech or bad legislation and using trends and anlysis in that data to help better the lives of our children. She points out one example about noticing bad things occurring around 3pm in which a school could then change policies to fix this trend. She even goes on to say that Texting could save more lives than penicillin.

Disease and Health Prediction

A website called Sickweather uses social media to correctly predict and map disease outbreaks and in fact predicted six weeks earlier then the cdc in October 2012 about the early outbreak of this years flu season. Using Social media to predict disease outbreaks

Google Flu Trends uses flu related search terms as indicators of flu activity. There work is based on research described in an article in the Nature scientific journal titled ‘Detecting influenza epidemics using search engine query data’.

Another attempt at uncovering health trends and public health concerns using twitter was a study run by two researchers from John Hopkins. They uncovered interesting patterns that showed medications and home treatments being used by people to treat their  symptoms. One helpful trend they noticed was that people were trying to treat the flu with antibiotics, raising a public health concern that should be addressed.

Financial Market Prediction

Topsy Labs, a company that has access to twitters historical data and has a real time search engine for twitter mining finds that there is a statistically significant correlation between Twitter sentiment and Market prices.

Predicting Election Outcomes

In an article about Big Data in the 2012 election Forbes points out that were not quite there yet with using big data to accuratly predict ellection outcomes. One of the reasons they point out is that there are paralels to 1948 when Dewey was innacturatly predicted to win over Truman based on phone polling. The majority of people did not own phones and those that did tended to skew toward a particular demographic. Similar to today we dont have 80 year old Grandmas tweeting about there party favorites.

But Forbes concludes that it does add some useful information even though we still need to figure out how to process all that data effectively and filter out the signal from the noise.

Sources

• http://www.bbc.com/future/story/20121010-app-could-predict-the-future
• http://gigaom.com/2011/07/07/can-you-crowdsource-health-information-via-twitter/
• http://www.nature.com/nature/journal/v457/n7232/full/nature07634.html
• http://www.ted.com/talks/nancy_lublin_texting_that_saves_lives.html
• http://about.topsy.com/2012/01/16/predicting-stock-prices-using-topsy-social-sentiment/
• http://www.bbc.com/future/story/20121010-app-could-predict-the-future
• http://www.forbes.com/sites/netapp/2012/08/29/big-data-takes-center-stage-in-the-2012-presidential-election/
•  http://www.modernhealthcare.com/article/20130119/MAGAZINE/301199964