Big Data: When statisticians become corporate heroes

Companies today face an overwhelming crush of available information, a situation referred to as ‘Big Data,’ and those best able to discriminate between relevant and irrelevant data will emerge as the most competitive.
In fact, statisticians may prove to be the corporate heroes of the future, delegates of an SAS Institute (Canada) Inc. seminar heard in Toronto on Nov 22. The seminar was entitled: Conquer Big Data with Big Analytics.
Speakers at the forum noted that social media have radically multiplied the amount of data available for analysis by companies to improve their business.
Bryan Harris, chief technology officer at Vision Systems & Technology Inc., observed that Twitter, for example, produces 6 million tweets per hour. “How quickly can you monitor that?” he said.
The key is using analytics judiciously to discern correlations and patterns in those tweets, Facebook status updates, etc. before a competitor does, Harris said.
“The majority of the data has zero value to you and your organization, and so we’re often trying to figure out how to [use that information],” he said
Harris added that new technology has allowed companies to see and use the information they have already stored in new ways.
Faced with this mountain of data points, the trick for companies is to identify and exploit relevant correlations between data points, said Merv Adrian, a research vice president of information management at Gartner. Analytics should be used to discover new insights, not simply to back up whatever is already known, he said.
He observed that while it may be irrelevant to a company if a person tweets about what he or she had for lunch that day, the point is to make connections with the people following that person on Twitter.
“Organizations use this information in a variety of ways,” Adrian said. “At Procter & Gamble, they used collected intelligence by allowing their customers to connect to them [using social media] and tell them about the experience they were having with their product.
“They cultivated ties with those consumers by giving them a place to go, giving them the motivation for going there. They reduced the R&D [research and development] associated with product development…by $600 million. Not only did they save money, but they stopped using some of their old methods of collecting information that were giving them some lousy ideas before.”
Given the new mountain of available data, it is no longer acceptable not to know things, IT futurist Thornton May said. “In 15 years, every molecule will be IP addressable,” he quipped.
And so the true problem-solvers in a company may well end up being heroes, May said. This bodes well for statisticians idolized in films such as Moneyball, in which a statistician helps a major league baseball team cull talented players lost to larger market teams.