Microsoft Gets Behind Predictive Analytics; Will You?

Mark Twain was foreshadowing when he said, "the past does not repeat itself, but it rhymes." Thanks to recent advances in predictive software, Big Data and cloud services, organizations now have real tools to predict the future with some degree of certainty. These technologies can (relatively) quickly mine terabytes of data on the Internet for clues and patterns to predict major events such as drought, flooding, and disease outbreaks, enabling organizations to improve their planning and risk mitigation efforts. Taken a step further, companies will soon be able to predict sales patterns and the likelihood of success for new products or promotions with a higher degree of confidence.

A variety of forecasting software tools have been available to help predict major events (with modest accuracy), as well as identify transactional risks and opportunities. Many are already in use in a variety of sectors including: retail, insurance, travel, and healthcare. What is different now is the availability of next-generation data mining software that leverages cloud-delivered processing power to parse vast amounts of current and historical data from numerous online sites and archives (think Big Data on steroids).

A good example of this forecasting software comes in the form of a research project from Microsoft Research’s managing director Eric Horvitz and former intern, Kira Radinsky, who worked together to develop a program that analyzes public news and archival websites for patterns and clues that have preceded outbreaks of disease, riots, and deaths. The algorithm then compares those patterns to current conditions to make predictions on the probability of those events happening again. It’s a unique and advanced form of data mining that digs deep into the Web and other data bases, enabling a sophisticated, causative analysis of seemingly unrelated incidents and seeing how and when they repeat themselves over time.

One example where their research made the right call was the occurrence of two cholera outbreaks: one in 2012 in Cuba and the other in Bangladesh in 2011. Given their different times and lack of proximity, one would have simply considered them random events. However, the program suggested that this was not the case. Searching 150 years of news reports and historical archives, the software identified a specific correlation in developing countries (with substandard or non-existent flood control) between a drought condition followed by major flooding, which subsequently led to a cholera outbreak — exactly what happened in both Bangladesh and Cuba.

Microsoft sees the potential. "When we look at trends in technology like cloud services, Big Data and business intelligence, and combine those with advanced machine learning, computer scientists will able to advance the use of the data to help predict catastrophic events more accurately in the future", says Grad Conn, senior director of marketing at Microsoft.

Their software is far from foolproof but has demonstrated an accuracy rate of between 70% and 80%, making it better than the more modest success rate of current tools. This improvement in accuracy is meaningful on a global level. Combined with preemption, enhanced planning and risk mitigation measures, better forecasting means that tens of thousands of lives plus billions of dollars could be saved.

Importantly, these same systems may be used to predict important trends and events in many other areas when combined with a companies’ own Big Data programs. For example, car companies can figure out which are the best years to introduce new convertibles by looking at weather patterns, levels of disposable income and competing vehicles over the past 50 years. Insurance firms can adjust their coverage, services and premiums based on the likelihood of different natural disasters occurring. Finally, travel firms that put together vacation packages can readjust their destinations, prices and itineraries (especially those to Cuba) to reflect the likelihood of other cholera outbreaks or hurricanes.

"What becomes critical is how the mounds of data collected are used to drive insights and make decisions", says Mr. Conn. "The ability to use sophisticated insights to develop innovative products and services, prioritize privacy, and reach and engage high-value customers is clearly a prized competitive advantage."

Even so, powerful data-mining software like this will not give every organization a clear crystal ball. All forecasting tools — no matter how sophisticated — should be used cautiously. There are enough gaps in the data and analytics to preclude most organizations from betting the farm on one predictive tool. You need a comprehensive approach to forecasting. Moreover, though powerful, these instruments may not be able to provide enough detail around timing or event severity to sensibly plan or take risk-reducing measures — or even overcome management inertia. Lastly, it is not clear how ‘Black Swan’ or unexpected events could be anticipated, especially in highly complex environments or where people’s fickle actions or attitudes can play a major role in shaping circumstances.

Mitchell Osak is managing director of Quanta Consulting Inc. Quanta has delivered a variety of strategy and organizational transformation consulting and educational solutions to global Fortune 1,000 organizations. Mitchell can be reached at mosak@quantaconsulting.com