One of the big lies about big data
I’m excited to attend the upcoming day-long New Frontiers and Opportunities in Customer Analytics workshop in Seattle (#wcaiconf), presented by the Wharton Customer Analytics Initiative (WCAI). Why? In part, because data analytics is central to the role chiefs of staff play in queuing up decisions for their principal executive and leadership teams (see my previous article on the role chiefs of staff play, here). Also, because everyone’s talking about data. Data-driven decisions. Big data. Predictive capabilities. It’s an exciting conversation if you’re a part of it. Finally, because I attended a WCAI workshop in July on customer analytics and voting behavior, at Wharton, and I know that this panel of analytics experts will close the gap between executive leaders and geeks, help me use the data I have more effectively, and help me apply data from seemingly unrelated areas to the situations I’m wrestling with the most.
Data, as a subject, has a lot of potential that hasn’t yet been realized. Its use comes with ethical questions that haven’t been fully answered or even explored. Yet many conversations in board rooms tend to leave you with the impression that decisions and behaviors are based either on numbers and evidence (i.e., they’re “data-driven”) or they’re based on intuition, gut, and emotion. In reality, to pit these against each other is a false dichotomy, and you embrace it at your peril, because these methods of arriving at decisions and behaviors are just different forms of intelligence.
For the purposes of this article, I use a definition of intelligence that falls somewhere between Merriam Webster’s definition1 and Gareth Morgan’s concept of it in Images of Organization (see the chapter “Learning and Self-Organization: Organizations as Brains”)2: the ability to learn or understand things initially but also to apply experience and learning to new or difficult situations through processes of continuous feedback.
The following is an example from a recent consulting conversation I had with a client of mine. He was convinced that getting customer feedback was the key to knowing how to effectively bring new products and services to market. True. But only partly true. You also need to layer on some intuition to the results, I advised. How do you know you’re measuring the right things? Do you just take the customer feedback at face value? Is it all self-reported data? Is that sufficient, or do you need some verification process? What do you do with conflicting feedback? What do you do with vague or ambiguous feedback? How do you compare your customer feedback with that of your suppliers, vendors, or national trends? Are there other data sets you already have that might show you what customers do, despite what they say? If you never look at that customer feedback (data) with intuition, which I will define here as the human brain’s subconscious calculation of data and experience, you might not even know to ask questions like these to get better explicit data. Is the original explicit data as likely to yield the right set of decisions, products, and experiences that your customers want without that intuitive layer?
I recently posed this false dichotomy question to big-data expert and founder at Big Nimble, Michael Kauffman. He says:
It’s false to say data and intuition are opposites. The truth is actually they can’t live without each other. My experience is that as people become more data driven, they start shifting the focus of their intuition away from the results and more towards the process. That is, if their intuition confirms the correctness of the various logical leaps they used in selecting and analyzing the data, then they learn to rely on what the data tells them regarding x or y or z result. And that’s when they can escape their own cognitive echo chamber and obtain game-changing insights: no longer are they dismissing data that refutes their preconceived notions. Now, they are seeing new insights because of their intuition, but now it’s intuition regarding process validity, so they know they can trust the even-surprising result. Basically, shifting intuition to process leads you to results you would have otherwise intuitively dismissed or ignored if just applying your intuition to the result.4
Big data is powerful and important, even if it’s not as much about having more kinds of data but just using the data you have more creatively. Equally powerful and important is how we apply it. How are you using it or helping others around you use it? How’s your current approach working?
I help C-suite executives and board members assess whether a corporate chief of staff can help them be more effective, find the right people, and manage the first 90–100 days with a chief of staff. I also help chiefs of staff be as effective as they can be.
1Merriam Webster, “Intelligence” http://www.merriam-webster.com/dictionary/intelligence.
2Gareth Morgan. Images of Organization. 2006; Sage Publications. pp. 71–112.
3Robert Burton. On Being Certain: Believing You Are Right Even When You’re Not. 2008. St. Martin’s Press.
4Interview with Michael Kauffman, May 25, 2016.