Experimentation is necessary in white water rapids environments. But it is important to apply the right type of experimentation. Image by Midjourney prompt by the author.

What does good experimentation look like?

Having designed, up-skilled and scaled business experimentation across tens of teams (and together with tons of brilliant talent) over the last four years, there are two key KPIs (maybe somewhat non-traditional) that I’ve found are the most important indicators of successful experimentation.

Surprise

A complex environment is an environment where there are too many moving parts, where making predictions are guesswork at best.

When I’ve asked colleagues and clients to describe their work everyone responds that they are operating in a complex environment, where they are not in control, but their job is to figure out what is happening and make the best possible decisions at the speed of their customers.

When asked which of the images most represent their work, everyone points to the white water river rafter. Work is unpredictable, it needs constant sensing of the environment around you due to exposure to unpredictable elements that you can not foresee. Images by Midjourney prompt by the author.

To do this they can use experimentation, but there are two types of experimentation:

You can use experimentation to charge or confirm a preset course making optimal decisions between option A and B along the way.

Or you can use experimentation to venture into the unknown (which is a vastly bigger space than what we know). To see if there are untapped opportunities that hadn’t been thought of or we didn’t even know existed.

Experimentation can be used to improve what we already know, or to learn entirely new things. Illustration by the author.

If we are in a complex environment where the job is not to know the answer, but to figure it out as fast as possible. Where the environment can make unanticipated changes all the time and where there are probably lots more opportunities in what we don’t know than what we do, then the latter form of experimentation is the better one.

And one of the best ways to measure if we are using experimentation to venture into the previously unknown is to check how often we are surprised by what we learn from our experiments.

By measuring if we are surprised or not, the frequency of our surprises and the significance of them we can get a good indication if we are using experimentation to learn something significant or just to make adjustments to what we already know.

Zero Marginal Cost

The cost of an experiment is critical.

Zero marginal cost refers to the additional cost of doing something after you did it once. How much does it cost to run additional experiments?

The higher the cost the more picky the organization will be. Making sure we only opt for experiments that are already the most valuable to us (which are usually the safer and more predictable experiments). If we can only bet on a few horses, we tend to choose the ones that most closely resemble what we are already doing.

Horse race. Image by Midjourney, prompt by the author.

The lower the cost the more experiments we can run. This opens the team to creativity, adventurism and more explorative assumptions to test. “What if”, “maybe”, and “why not” might be more frequent words used when making decisions about an experiment.

The most amazing teams I’ve seen have found ways to find, plan, run and learn from experiments at almost no cost. Leading them to run experiments on more and more unexpected and surprising assumptions.

And some of those assumptions have taught them things they didn’t even think was a thing and taking advantage of that learning led to great results.

The more efficient the system and culture of experimentation the organization can build, the more experiments they can run and the more surprises they will get from what they learn. For organizations finding themselves in white water rapids, this can be the game changer.