STRUCTURING BIG DATA: Approach to realize the value of Big Data

Big Data defined:

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, storage, search, sharing, transfer, analysis and visualization. Industry analysts articulated the now mainstream definition of big data as the four Vs and a C: Volume, Velocity, Variety, Variability and Complexity.

Big Data Analytics:

Big Data analytics is the process of examining large amounts of data of a variety of types to uncover hidden patterns, unknown correlations and other useful information. Such information can provide competitive advantages over rival organizations and result in business benefits, such as more effective marketing and increased revenue.

The primary goal of big data analytics is to help companies make better business decisions by enabling data scientists and other users to analyze huge volumes of transaction data as well as other data sources that may be left untapped by conventional business intelligence programs. These other data sources may include Web server logs and Internet clickstream data, social media activity reports, mobile-phone call detail records and information captured by sensors. 

Big Data Management:

Big data management is the organization, administration and governance of large volumes of both structured and unstructured data^kpsko. The goal of big data management is to ensure a high level of data quality and accessibility for business intelligence and big data analytics applications. Corporations, government agencies and other organizations employ big data management strategies to help them contend with fast-growing pools of data, typically involving many terabytes or even petabytes of information saved in a variety of file formats. Effective big data management helps companies locate valuable information in large sets of unstructured data and semi-structured data^kpsko from a variety of sources, including call detail records, system logs and social media sites. Tools used for Big Data Management are: Hadoop, MapReduce, NoSQL, Cassandra and Hive.

Important business questions will be:

• Why are our customers leaving us?
• What is the value of a 'tweet' or a 'like'?
• What products are our customers most likely to buy?
• What is the best way to communicate with our customers?
• Are our investments in customer service paying off?
• What is the optimal price for my product right now?

The value of data is only realized through insight. And insight is useless until it’s turned into action. To strike upon insight, you first need to know where to dig. Finding the right questions will lead you to the well.

Big Data as a Service (BDaaS):

Big data as a service (BDaaS) is the delivery of statistical analysis tools or information by an outside provider that helps organizations understand and use insights gained from large information sets in order to gain a competitive advantage. Given the immense amount of unstructured data generated on a regular basis, big data as a service is intended to free up organizational resources by taking advantage of the predictive analytics skills of an outside provider to manage and assess large data sets, rather than hiring in-house staff for those functions.

Big data as a service can take the form of software that assists with data processing or a contract for the services of a team of data scientists. BDaaS is a form of managed services, similar to Software as a Service or Infrastructure as a Service. Big data as a service often relies upon cloud storage to preserve continual data access for the organization that owns the information as well as the provider working with it.

The Challenges:

Many organizations are concerned that the amount of amassed data is becoming so large that it is difficult to find the most valuable pieces of information.

• What if your data volume gets so large and varied you don't know how to deal with it?
• Do you store all your data?
• Do you analyze it all?
• How can you find out which data points are really important?
• How can you use it to your best advantage?

Until recently, organizations have been limited to using subsets of their data, or they were constrained to simplistic analyses because the sheer volumes of data overwhelmed their processing platforms. But, what is the point of collecting and storing terabytes of data if you can't analyze it in full context, or if you have to wait hours or days to get results? On the other hand, not all business questions are better answered by bigger data. You now have two choices: Incorporate massive data volumes in analysis or determine upfront which data is relevant.

Final words:

Big data transforms the data management landscape by changing fundamental notions of data governance and IT delivery. Though big data is still at its early stage, the advantages of big data will feed the development of new capabilities in sensing, understanding, and playing an active role in the world for the next 20 years and will change all walks of life. However, the underlying analytics and interpretations of results will still require human cognition to connect the dots and see the big picture.

An organization needs a strategic plan to adopt the big data technologies. The ability to collect and analyze massive amounts of data will be a key competitive advantage across all industries, including government. Such analytics projects can be complicated, idiosyncratic, and disruptive—thus they require a strategic plan to be successful.

It takes time to change the culture of depending only on traditional data analytics. There will be occasions of unethical, abuse or misuse of big data applications as big-data analytics and technologies are implemented. Therefore, it is better to be cautious and start small and simple.

It takes the whole society to implement big data^kpsko technologies. Since big data will affect all of us in our life and collaboration and partnership are essential to make big data successful, we are all responsible to work together in dealing with the issues raised along with application of the technologies on this journey for the next decade or two.

"Now you can run hundreds and thousands of models at the product level - at the SKU level - because you have the big data and analytics to support those models at that level."

References :

[1] http://www.thoughtworks.com/big-data-analytics

[2] http://www.sas.com/en_us/insights/big-data/what-is-big-data.html

[3]http://searchbusinessanalytics.techtarget.com/essentialguide/Structuring-a-big-data-strategy

[4]http://www.meritalk.com/

[5]http://searchcio.techtarget.com/

[6] http://en.wikipedia.org/wiki/Big_data

Big Data Analytics

Big Data Boat

I have always been amazed with astrology and future prediction. Big data[1] is same, future prediction with a scientific approach to it. Big data is a collection of data which is so huge, that it is difficult to store and process them using conventional methods. Examples are RFID (Radio frequency ID), data from social networking site like Face book, Twitter, Internet search, Credit card payments, transaction of retail giants like wall mart, etc. 

So how do we define big data. How to know if the data can be considered "Big Data". What are the attributes of Big Data.

4 V's of Big Data.[2]

a) Volume - Volume refers to the size of the data. It's estimated that 2.5 quintillion bytes (2.3 trillion gigabytes) of data are created every day. 

b) Variety - Variety refers to the type of data. This case be server logs, click stream data, audio and video.

c) Velocity - Refers to the speed at which it is generated. We are generating data at a very fast pace. The New York Stock Exchange alone captures one terabyte of trade information during each session .

d) Veracity - Veracity refers to the uncertainty of data. The noise and abnormality in the data. Is the data which we are analyzing related to the problem at hand?

Big data analytics is already being used drastically in the retail and other industries. Companies have now started moving, from an approach of identifying customer transactions to understanding customer interactions. Now companies are not only interested in knowing what was bought, when it was bought but also why it was rejected. Based on the users online interaction, they try to find why a particular item was selected or rejected, and is there a pattern in that.

You would have experienced while doing online shopping that you select a pair of shoes and then remove it. Next time when you visit the site, the same shoes are displayed. They try to find a pattern. To get to a correct conclusion we need to analyze huge data. Each click that you make online is being stored on the servers and being analyzed. This is not only done online. There are companies which are now following this approach in physical stores. Using video cameras, they try to capture the expression of the customers. Try to analyze, which part of the store is being used more, which time of the day, which day of the week, etc. This analysis is more powerful than the analysis based on a few clicks made by customer online.

This development is going to make a huge impact on all the walks of life and not only retail.

1) This can be used by industries to analyze risks and find out ways to avoid them in future - With Big data we have all the past records. So before taking any major decision, the past trends can be observed to reach to a least risk conclusion.

2) It becomes easy to get to the root cause of an issue - Since we have all the relevant data of servers, logs and other details; it is possible to exactly pin-point the issue.

3) Understand the upcoming changes in fashion - Based on data from social media, it will be possible to identify the new trends. The company can start manufacturing the products based on these data.

4) Identify the voting pattern and the issue that matters the most. As per Bosmol dated 8th Feb, Big data analysis was one of the important factor's for Barack Obama's re-election in 2012.

5) Based on big data, the companies can predict when to launch the sales and how much. It can help to  provide the shopping pattern of a particular location. The sales offer can be done at one location and not the other.

6) It can be used by the telecom companies to identify the dead spots in the signal coverage and also to identify the calling pattern. This is already being used by most of the telecom operators. Based on the calling patterns various schemes are given to the customer.

7) Big data can be used in the fields of astronomy, medical science - With big data, we can identify why particular medicine affects one person more than the other. What is the reason for a particular allergy; is it related to location, climate or the individual. Lot of unanswered questions would be answered.

8) Understanding the mental status of an individual - In the current scenario, people are very active on social media. This can be used to check the mental status of a person. Using this, untoward incidents can be avoided.

Big Data Challenges

The three key challenges in making data analytics work are

1) What data you want to use – The challenge is which data to use, how to get it, how to integrate it and how to use it. There’s lot of data out there. Supply chain data, customer data, and performance data. Managing this data is a big challenge, but the real value comes when we integrate internal data with other external data like weather data, traffic pattern data and competitive data.

2) Analytics – The second challenge is to get the right people at the job. To get people who really know how to use the latest mathematical technique is difficult.

3) Transform the business – The third and the most difficult of all is to use this data to transform the business. There’s no point in getting insights and not using them. We need to change how the business operate, how the managers work on a day to day basis.

Some of the other technological challenges in big data implementation is integrating existing data ware house with Hadoop, getting professionals who know how to write Map-reduce code in Hadoop. Since these challenges are now being resolved by using technologies like Impala(which can be used to directly query using SQL, no need to convert to map-reduce), sqoop let’s hope that Big data will be used to its full potential in the near future.

References :

[1] http://en.wikipedia.org/wiki/Big_data
[2] http://dashburst.com/infographic/big-data-volume-variety-velocity/
[3]http://inside-bigdata.com/2013/09/12/beyond-volume-variety-velocity-issue-big-data-veracity/

[4] http://searchbusinessanalytics.techtarget.com/definition/big-data-analytics
[5]http://www.mckinsey.com/insights/business_technology/making_data_analytics_work
[6]http://www.justinholman.com/2012/12/19/geography-big-data/
[7] http://en.wikipedia.org/wiki/Cloudera_Impala

Moz: Moving Beyond Traditional SEO

Key players in online marketing are moving away from traditional SEO consulting and moving toward a more holistic view of marketing analytics to better understand customer behavior and increase revenue for their clients. Moz is a great example of a company who has realized this trend. Beginning in SEO, Moz has progressed into expanding their product functionality, creating a unique environment for all knowledge levels, and ultimately, innovating more comprehensive online marketing tools.

Who is Moz?

Started in 2004 as a search engine optimization (SEO) consulting company, Moz has undergone tremendous growth since inception. Within three years, Moz advanced into offering its own SEO tools, including Mozscape and a web app, earning itself a well-regarded name in the SEO community. In the last year, Moz has made another significant shift, transitioning from SEOmoz to Moz. Now a provider of inbound marketing analytics, Moz is evolving to give customers better access to actionable insights beyond traditional SEO. [1]

 [2]

How is Moz transitioning from traditional SEO? 

People are becoming increasingly interested in using SEO and other forms of digital analytics to drive growth and revenue. Customers are looking for tools that not only focus on data collection and reporting, but also provide real value to their marketing goals. This is a particular issue with firms who do not have any existing digital analytics infrastructure, and where companies like Moz step in. Known for being accessible to those who have no prior knowledge of SEO or analytics, Moz has altered its strategy to better meet the needs of a larger customer group. [3] Through a complete overhaul of its SEO tool offerings, Moz now provides users with Moz Analytics, an encompassing tool with “more of a focus on measuring your efforts rather than actually “doing” and managing your SEO work”. [4]

What are Moz’s current tactics?

With the introduction of Moz Analytics, Moz now offers a tool that contains all inbound marketing data. The following elements are included:

•      Content analysis
•       Link metrics
•       Search marketing
•       Social analytics
•       Brand mentions
•       Competitive analysis [1]

[5]

In offering this comprehensive tool, Moz is attempting to combine different analytics elements to provide a broader picture of how each works together. Now customers can get a better sense for how certain content may affect brand mentions. And, they get the benefit of receiving integrated information about multiple social media outlets. Even competitor tracking is included.

But something that makes Moz really stand out is the value it places on helping users truly understand analytics, which is one of their biggest competitive advantages. On the Moz website, many learning materials are available from “The Beginner’s Guide to SEO” to Moz Academy to Mozinars—all are provided to ensure that customers are able to take full advantage of Moz’s tools to improve marketing, which is somewhat unique and very helpful for newbies. [3]

Now Moz Analytics is only recently out of its launch phase, and as such, Moz will be focusing on adding additional important features, such as a content section. [1] Although some reviewers still see Moz Analytics as SEO at its core or prefer other tools such as Raven Tools or Web CEO Online, it is likely that Moz will continue to grow and respond accordingly. [4]

For a more detailed look at Moz’s tactics and plans for growth, here is a link to a summary post on The Moz Blog.

Signaling an Industry Shift

Analysts are in very high demand, and more and more companies are striving to make analytics an integral part of marketing as a whole. In fact, a Forbes article just named 2014 as “The Year of Digital Marketing Analytics”. [6] [7] In conjunction, analytics tool providers are trying to ensure the online marketing community has the best tools for the job.

With Moz’s rebranding, its actions coincide well with this year’s title. In making this shift, Moz intends to provide not only “the most robust, competitive set of data available” but also “data [that] helps you prioritize your next move.” [1]

And it is likely an industry trend that will continue. The marketing community wants comprehensive tools that provide value and insight aligned with business goals, and Moz Analytics took one big step in giving them just that.

[1] moz.com

[2] http://www.radiatedigital.com/seomoz-is-now-moz-is-seo-dead/

[3] http://www.examiner.com/article/an-honest-review-of-moz-analytics

[4] http://www.targetinternet.com/moz-analytics-review/

[5] http://marketingland.com/moz-analytics-launches-out-of-beta-61805

[6] http://www.cmo.com.au/article/535793/marketing_predictions_2014_5_analytics_trends_watch/

[7] http://www.forbes.com/sites/jaysondemers/2014/02/10/2014-is-the-year-of-digital-marketing-analytics-what-it-means-for-your-company/

Amazon’s Anticipatory Shipping – Collaborative Filtering, Predictive Analytics, and Supply Chain Mastery

Amazon is raising the bar once again in regards to the way in which it delivers its products. No, this isn’t about the super awesome drones that could possibly be delivering your next box of diapers to your front porch. This is potentially even bigger and it relies heavily on the ecommerce juggernaut’s ability to use predictive analytics in conjunction with its supply chain fulfillment.

Online retail is growing every day, it has a low overhead and is easily accessible. However, one disadvantage that Amazon, and other online retailers face, is the ability to get the product in the customer’s hands as quick as a traditional brick-and-mortar store would. If you’re online and need something shipped same day or next day you’ll have to pay an exorbitant amount of money which practically negates the savings found by going online. And that’s why Amazon is trying to solve this disadvantage with its patented ‘Anticipatory Shipping’.

Anticipatory Shipping

Amazon filed for a patent in August 2012 which was granted December 24th of last year (Merry Christmas, Amazon) for a shipping system designed to dramatically cut delivery time of its products by predicting what buyers are going to buy – even before they buy it. Amazon describes one of the methods of accomplishing this:

“…a method may include packaging one or more items as a package for eventual shipment to a delivery address, selecting a destination geographical area to which to ship the package, and shipping the package to the destination geographical area without completely specifying the delivery address at time of shipment. The method may further include completely specifying the delivery address for the package while the package is in transit.” [1]

Amazon’s method of selecting closest proximity speculatively shipped packages

Basically, Amazon plans on generating a predictive shipping model and speculatively shipping packages to corresponding destinations/geographical areas without completely specifying the delivery address. Then as soon as you or I go online and order that new 3TB hard drive or Call of Duty video game, Amazon will be able to select the order in closest proximity, convey the specific address, and deliver the items in record time to our front porch. An article on techcrunch.com said “…the language of the patent sounds as if Amazon is thinking of physical item delivery in the way a utility might approach supplying water or electricity to homes — by forecasting demand spikes and lulls, and tweaking its pipeline accordingly, but above all by keeping the stuff flowing (ergo having trucks constantly filled with packages in continuous perpetual motion).” [2] Either way, if successful, this new shipping method could revolutionize the way in which consumers interact with online retailers and receive their products.

Data / Communication flow of fulfillment system configured to support speculative shipping

So how does collaborative filtering play into all of this? 

In short, collaborative filtering is a method of making automatic predictions about the interests of a user by collecting preferences or taste information from similar other users. Have you ever wondered how Netflix recommends a new movie or how Pandora / Spotify can recommend unique artists based on your taste in music? There are algorithms in the background that predict what you would like based on what other users similar to you like. It is all about detecting the interests of the consumer and recommending items based on those interests. 

Likewise, the success of ‘Anticipatory Shipping’ revolves around Amazon’s ability to detect the potential interest in a product from a customer in that region and then weigh it against the costs of returning or re-routing the package. Amazon plans to detect the customer interest in a product based on many variables including: purchase history, browsing patterns, wish-lists, and demographic information of you and others similar to you. As soon as this level of interest is determined, a potential ‘cost to return or redirect’ is calculated and the package could be offered at a discounted price. [3]

Amazon’s method of speculative shipping with late address selection

Amazon has a history of paving the way with how online retail interacts with customers. They patented the ‘one-click’ buying mechanism back in 1999 and they’re hoping that anticipatory shipping will have similar success. Determined to revolutionize the industry, ‘anticipatory shipping’ could be the next ‘big thing’ in online retail.    

____________________________________

References

[1]  "United States Patent No. US 8,615,473 B2 | United States Patent and Trademark Office." 2013. 18 Feb. 2014 <http://pdfpiw.uspto.gov/.piw?PageNum=0&docid=08615473&IDKey=2809ACB12F05&HomeUrl=http%3A%2F%2Fpatft.uspto.gov%2Fnetacgi%2Fnph-Parser%3FSect1%3DPTO2%2526Sect2%3DHITOFF%2526p%3D1%2526u%3D%25252Fnetahtml%25252FPTO%25252Fsearch-bool.html%2526r%3D1%2526f%3DG%2526l%3D50%2526co1%3DAND%2526d%3DPTXT%2526s1%3D%252522anticipatory%252Bpackage%252522%2526OS%3D%252522anticipatory%252Bpackage%252522%2526RS%3D%252522anticipatory%252Bpackage%252522>

[2] Natasha Lomas. "Amazon Patents “Anticipatory” Shipping — To Start ... - TechCrunch." 2014. 18 Feb. 2014 <http://techcrunch.com/2014/01/18/amazon-pre-ships/>

[3] Taylor Soper. "'Anticipatory shipping': Amazon wants to ship purchases ... - GeekWire." 2014. 18 Feb. 2014 <http://www.geekwire.com/2014/anticipatory-shipping-amazon-wants-package-purchases-even-buy/>