Experiment, Measure, Repeat
Improving your company with continuous experimentation
A few months ago, I read an article from Michael Simmons where he essentially said that
“Deliberate experimentation is more important than deliberate practice in a rapidly changing world.”
Even though the article focused more on experimentation for personal improvement, the first thing that came to my mind was something like: “this should be one of the core values of every company” (here’s a Forbes article that tells the same).
Here at buildo, we take experimentation and continuous improvement seriously: it’s our core value. The goal of this post is to share with you how we do experiments internally, also adding some advice that you could follow, hoping to be useful for some of you that do experimentation or that want to start (you should!).
Let’s start with a simple case study
In October of the last year, we started noticing that some of our Slack channels were noisy and difficult to follow, especially in high work-load weeks. People were speaking about different topics in the same channel (e.g., two issues of the same software project), with unrelated messages alternating between one topic and another.
We decided to solve such pain point like we usually do: with an internal experiment. Luca Cioria, who proposed the experiment, became its owner and did the following:
- defined a goal: “Make Slack easier and faster to read, considering the growing amount of messages.”
- defined how to try to reach the goal: “Use threads as much as possible, and create a culture of respecting channel’s quiet.”
- defined a metric to be measured to evaluate the success of the experiment: “The percentage of messages in threads.”
- defined success conditions for the experiment: “People like threads and accept to use them” and “there is a significant increase in the metric (set to 70% mostly as a psychological goal)”
- defined a date for the final evaluation (i.e., success or failure?): “2018, 10th week.”
Luca then presented all this to the rest of the team, which agreed to start; he then created a plot of the success metric and added it to our internal dashboard, to create awareness about the progress.
This is the result so far:
The experiment was considered successful, and the outcome was a new dedicated paragraph in our internal Slack guidelines.
Building a culture of experimentation
I think that having experimentation as one of the core processes in your company is extremely important, and I believe that creating such culture of experimentation is not as difficult as it looks.
There is no secret sauce to do it, but I will now share some essential guidelines that we follow and that you could try to use too:
- Set up recurring meetings to discuss pain points and ideas, and to come up with possible experiments to address them. Once you have a set of possible experiments, prioritize them and work only on few of them simultaneously. We prioritize based on value and effort of every experiment (e.g., we give priority to high-value-low-effort ones)
- Documentation is essential: every experiment shall have an owner, a goal, some success metrics to be collected, and a deadline; all this shall be clearly stated in a so-called “canvas” (we use DropBox Paper for that, and for almost all our internal processes. Francesco Negri spoke about it in another post)
- Use a data-driven approach: collect data and plot the experiment metrics; to create awareness, make them easily visible to everyone. For example, we just put a dashboard near the coffee machine, and it works. Quantitative metrics-based goals are good, but not reaching them should not automatically result in failure: there should be a final meeting for every experiment, to discuss its outcomes
- Embrace failure and avoid maintaining useless things: failure is a vital part of the process. If an experiment is not considered successful, ask yourself why: this could result in inspiration for future improvements. Most importantly, archive or delete everything related to it, to avoid having to maintain unneeded tools or processes
Although this works for us, we encourage you to take inspiration from what we wrote to build your own process of internal experimentation. The process itself shall be continuously improved, like basically everything in your company. At buildo, nothing can “escape” continuous improvement: the dashboard near the coffee machine started as an experiment, but also this blog, the process I described, and everything else we do and use every day.
At first, it seems a lot of work, but it’s easier than it looks: call the first meeting with your team, find a way to collect and display metrics, and there you are, ready to experiment. Happy improvement :)
Edit — A note about formality
I noticed that the post is on the first page of Hacker News: thank you, readers!
Some of you made me notice a mistake, and I decided to add this last note.
In the Slack threads experiment used as a case study, we performed pre and post-change measures without having a control group or a baseline, which means that formally speaking it is not an experiment. A better way to conduct the same test could be, for example, using A/B testing: we could capture metrics from two different channels, with only one of them following the new guidelines about threads, and perform a comparison after a certain time. However, in my opinion, not everything needs rigorous tests: for cheap changes that affect small and easy-to-change processes, the fact that the team perceives things as better after the experiment could be enough. In this case, we asked all the team members if they saw an improvement in the Slack situation, and the majority said yes.
Keep that in mind for your experiments, especially when the changes involved are critical: sometimes it could be worth doing more rigorous statistical tests.
Acknowledgments
- To build our experimentation process, we took inspiration from the well-known PopcornFlow. For example, here’s the PopcornFlow experiment canvas template
- Special thanks to Jacopo Romei and to Avanscoperta, who help us continuously improving our continuous improvement processes over time
We plan to write something more about our internal processes in the next posts: if you liked this, stay tuned!
—
If you want to work in a place where we take experimentation and improvement seriously, and where we care about the quality of our development workflow, take a look at https://buildo.io/careers
