Getting Value from Big Data

In the last decade, new concepts such as big data, expertise of so called data scientists, and distributed computational frameworks such as Hadoop/MapReduce have received a lot of share of mind. Further, substantial investments have been made in leveraging large quantities of data with the hope (and prayer) of improving short-term and long-term business performance. But the returns on such investments are either slow to realize or non-existent. Why? Here are some plausible reasons and suggestions:

1. There is no substitute to creativity and intuition: The hypothesis that collecting and analyzing large amounts of data moving at warp speed uncovers new insights is hyped. Patterns will emerge from such analyses, but the patterns need to be valuable to the business and timely actions need to be taken to change the status quo for realizing value. Human intuition, creativity, and interpretation are critical to convert big data analyses into recognizable patterns and data intelligence.
   
   
2. Executing is more important then just knowing: Identification of a pattern such as positive customer experiences (and quick resolution when there is a negative experience) improve loyalty of your most valuable customers is one thing. But how does the company change its core customer service processes, incentive structures, and leverage real-time data to manage and influence customer dialogues is another matter altogether. It requires ad-hoc and real-time decisioning tools and intelligence on top of big data and changes in human behavior to follow-through on the identified pattern to realize value. It is no different from the limited value realized if insights you have garnered from small data (market research, customer satisfaction surveys and the like) in yesteryears gather dust on the shelves. 
   
   
3. Don't ignore the art of story-telling: Ultimately, success of a business depends on sound strategies, anticipating the future, and impeccable execution. Decision-makers, managers and employees need simple and easily understandable stories emanating from big data and answer 
   “what-ifs” to change behavior and prioritize future investments and actions. 
   
   
4. Small data plus big data: Big data and patterns thereof need to be synthesized with small data for making strategic business decisions. Big data with supporting offline and real-time decision engines have natural strengths in tactical, granular, and operational aspects of running a business, such as recommendations, real-time bids for advertising, etc. But they need to be synthesized into coarser-grained patterns to support business strategy refinements and identification of new opportunities where small data typically have excelled.
   
   
5. Focus on data that (potentially) matter to your business: You don't need to gather and store every bit for eternity and then expect patterns to emerge for business decisions. One needs to prioritize types of bits based on potential value, invest technology and analytical resources in concert with the potential value and business strategy. For example, for managing customers in CRM systems, we don't need a 360-degree view of a customer but just the right view to help make smarter customer-level investment decisions.