Getting high quality customer data in real-time at speed and scale
Critical discussions on AI has moved on from it being a “black box” which we don’t understand how works (1)(2)(3) to what we put into it to make it work (4)(5)(6). We are moving from focusing on the algorithms to the quality of the data.
We might have given up understanding how AI’s work admitting that they just do. Focusing more on what makes them work: data and goals (7).
If we can ensure the quality of the data and sensibility of the goal (avoid destroying human kind(8)), we might be in a good place?
As we get more interested in the quality of the data it is timely to look at what data we use to influence our decisions.
And not necessarily inside big AI-engines like the ones from openAI, Microsoft or Google. But our much more relevant local engines, fed with our own data inside our own organizations to help us make better decisions related to our own customers.
The customer is a robot
We have a history of seeing the customer as a robot .. popular hierarchical decision making models like AIDA have been debunked for the last decades being too functional and not including the emotional part of decision making (9).
A customer journey can easily suffer the same fate (but doesn’t have to). It sometimes includes a smiley face as an emotional alibi, but its purpose is to pretend the customer is a train engine we can nudge algorithmically down a track towards the purchase (ugh).
Still we keep using them.
We have to stop the same happening with our data.
“Thin” and “thick” data
A decent AI-engine uses two types of data: “thin” data which is quantifiable, in abundance and used to look for patterns and to correlate probabilities.
And “thick” data. Which is the qualitative, subjective and contextual data adding nuance, reasoning, etc..
Simply put the “thin” data provides the “what” people are doing, while the “thick” data provides the “why” they are doing it.
The bulk of the data our AI-engines use is “thin”, while the “thick” data is there to make sure the models don’t become completely irrelevant by portraying the customer as a robot.
But adding and using the qualitative data is complicated and expensive. Because .. we think it has to be.
Let’s change that!
Qualitative data is not harder to get than any other data. If we design for it (10).
We have at our disposal the opportunity to both create and record any imaginable experience. We can learn anything we want from our customers continuously, in real-time and at scale as they are engaging with us (11)(12)(13).
We just need to ask the right questions first, and then adjust the design of the experience to learn the answer through the users behaviors and interactions.
Let’s break from our preconceived ideas of where and how we can get qualitative data. We can get good data from anywhere if we design for it.
“Nobody designed this website to learn anything” — Yuanyuan Zhao
Qualitative data is the response to a question or an experience that engages in peoples emotional reasoning and decision making. It explores what leads the customer to make their decision or action. Be that their own motivation (need) or external influence (the situation they are in).
There are no limits to where, how or who can collect qualitative data. Only our imagination should be stopping us.
We are on the verge of something big
As the AI-focus moves towards data quality we have an important opportunity to assess our own data. We have an abundance of “thin” data telling us how our products are selling and our channels are performing. What we have less of is “thick” data, which helps us include the customer, their needs, context, culture, nuances and complexities.
In one of our projects we asked two anthropologists to assess some of our models and dashboards that were supposed to include the consumer. Their feedback was clear: you have data about your own products and channels using them as proxies for the consumer. You don’t have any data about them, only about yourself.
“Thick” data is readily available, from every interaction with our customers, in abundance, real-time and continuos. If we choose to design for it.
And now with the increased focus on data quality let’s bring back the customer into our data. Hire anthropologists and ethnographers to assess what our data is telling us about the people we are engaging with, how we are collecting and making sense of the data to better understand them.
And to stop holding up our own outputs and channel performance indicators assuming they are windows into the customer — they are not, their are mirrors of ourselves.
If not now, when?
Sources:
(1). Janelle Shane, You look like a thing and I love you, https://www.janelleshane.com/book-you-look-like-a-thing
(2). Cathy O’Neil, Weapons of math destruction, https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
(3). Samuel R. Bowman, Eight Things to Know about Large Language Models, https://arxiv.org/abs/2304.00612
(4). Altay Ataman, Data Quality in AI: Challenges, Importance & Best Practices, https://research.aimultiple.com/data-quality-ai/
(5). Thomas C. Redman, Ensure High-Quality Data Powers Your AI, https://hbr.org/2024/08/ensure-high-quality-data-powers-your-ai
(6). Ronald Berry, Why AI Bias is a Big Deal — and How Quality Data Can Make a Difference, https://rberry19.medium.com/why-ai-bias-is-a-big-deal-and-how-quality-data-can-make-a-difference-a59370240315
(7). Cathy O’Neil, The truth about algorithms, https://youtu.be/heQzqX35c9A?feature=shared
(8). Stuart Russel, How will AI change the world?, https://youtu.be/RzkD_rTEBYs?feature=shared
(9). Wikipedia, AIDA (marketing), https://en.wikipedia.org/wiki/AIDA_(marketing)
(10). Helge Tennø, If we design for it, https://everythingnewisdangerous.medium.com/if-we-design-for-it-34e37d067746
(11). Helge Tennø, How we did it: Digitizing Design Thinking, https://medium.com/design-bootcamp/how-we-did-it-digitizing-design-thinking-c538f813943a
(12). Helge Tennø, Design for better data, https://medium.com/design-bootcamp/design-for-better-data-3dbc7fef28c5
(13). Helge Tennø, Designing for learning, https://everythingnewisdangerous.medium.com/design-for-better-data-6d2f780028d
