Getting thick and rich data right
Both human and artificial (AI) decisions are based on the data closest to us (1). At the moment we are very good at collecting, accessing and making use of quantitative data. But this data has limitations. It’s doesn’t represent a picture of the world, but a part of the world, the numeric part.
It represents the part that knows how many times you’ve been to the museum and how many hours you’ve spent there, but it doesn’t know which artwork made an impact, or how different artists made you feel or reflect. It also doesn’t know why you keep coming back and how the people you bring with you influence your experience.
For this we need thick data (qualitative, ethnographic, 2) and rich data (anecdotcal, 3).
But thick and rich data are not as available as quantitative data, and to some extent we seem to have accepted this instead of fixing it.
e.g. customer insights is often only collected in the beginning of a design thinking process, market research in the beginning of a product launch etc.. they are used as a springboard to give answers before we know which questions are most important to ask.
What the computer is especially good at is emulating humans at a scale of one to a million or more. With that scale it becomes much clearer what happens when you train your decisions only on quantitative data.
This customer-data-gap (4) was not as easy to spot with slower and few human-only decisions, but now it has become much more visible and critical through fast and scaled artificial (AI) decisions.
Spending time demonstrating this customer-data-gap I’ve never ran into anyone not agreeing to it nor the benefit of closing the gap. But I’ve also found that understandably we are so used to thick and rich data having their current properties that we think the issue is unsolvable.
Which is where I disagree.
So what are these properties?
What do we need from our thick and rich data to in order for it to be more available, usable and impactful? In order for it to help our organizations become what they want to become.
#Low cost:
Qualitative data is inherently expensive. We need to lower the cost of acquisition. We need new tools or thinking in regard to where and how we collect what data.
#Bias:
Too much qualitative data is collected in environments where the main actor (customer, user etc.) knows they are being observed. And so the likelihood of them changing their behavior is large. We need to avoid this Hawthorne-effect (5) by collecting data in new ways and environments.
#Continuous:
Any data gives you a picture of what the world looks like at the instance you captured it. We need a continuous view to capture changes in habits, reactions and anomalies.
#Closeness:
We make decisions based on the information closest to us. Thick and rich data need to be made available through formats accessible with the swipe of a finger on any device anywhere.
#Inclusive:
Every analyst has a bias, so we make everyone an analyst. Anyone can access, interpret and make assumptions about the data, it is not in the remit of some deity to tell the world what the data sees.
#At scale:
We need to scale the data. Having input from 10 or 35 people is good, but why not a 100 or a 100.000?
#Real-time:
Sometimes we need to see the data without delay (at least not a 3–6 week delay), we need answers as soon as we’ve asked the question. The faster we get answers the more questions we can ask, the more curious we can be and the more assumptions we can dig into discover or disregard.
#Everyone, anywhere, all-the-time:
A project is nothing more than a string of decisions made by everyone all the time. Every decision could potentially benefit from thicker and richer data, and so it has to be available for any type of question, anywhere in the process by anyone asking.
These are all properties, opportunities that can be solved with the right imagination, enthusiasm and eagerness to make a difference. Let’s go.
Sources:
- Helge Tennø, Keep your products close, but your customers closer, https://medium.com/design-bootcamp/keep-your-products-close-but-your-customers-closer-c991c600ef59
- Pratibha Kumari J., What is Thick Data?, https://www.linkedin.com/pulse/what-thick-data-pratibha-kumari-jha/
- Dave Snowden, Big, thick and rich (the data), https://thecynefin.co/big-thick-and-rich/
- Helge Tennø, The customer data gap, https://everythingnewisdangerous.medium.com/the-customer-data-gap-520cdf695d68
- Hawtorne-effect, https://en.wikipedia.org/wiki/Hawthorne_effect