How to Make Data Experiments Powerful

The power of experiments comes from the insight they provide into causal relationships. Simply put, by randomly manipulating only a single focal variable, we can assume that any observed changes are likely due to the manipulated variable, and not to something else.

Experiments based on data can be particularly powerful for organizations, especially if they easily augment managerial intuition and exploit unique data.

Data-based experiments are not new. Harrah’s Casinos provided early examples of how organizations could add confidence to organizational decisions, but at that time, many companies resisted aggressive testing. Change is difficult, particularly for experiments where some level of underperformance is inherent in the design. By definition, an experiment that shows differences between an indicator of performance will mean one group underperforms the another. For example, it can be difficult for managers to test a change that their intuition tells them will increase customer satisfaction or sales, because it means not following their intuition and reducing satisfaction or sales for a control group.

Now, there may be less resistance as organizations see the successes of others. Data-savvy organizations are setting up platforms to promote experimentation throughout their organizations. Experimentation can be “virtual research centers” that allow scale in R&D.

For example, Edmunds.com is a car shopping website with 20 million unique visitors every month. They built their own testing platform called the Website Testing Framework, on top of their extensive cloud-based data infrastructure. Phil Potloff, chief digital officer of Edmunds.com, says it allows all of their managers to ask “WTF (Website Testing Framework)? What’s going on with this test?” The framework supports dozens of simultaneous tests against three key performance indicators: user engagement, ad impression delivery rate, and lead volume. Questions can be big, such as evaluating an entirely new vehicle appraisal tool. Or tiny, such as the color of a single button on the website. Potloff reports that the framework has “really changed the way that we interface with analytics and how we make changes.”

WASH Laundry is another example of a company scaling experimentation. WASH Laundry manages laundry facilities embedded in 75,000 locations. More than 7 million people per week use laundry facilities under their management, generating a massive amount of data in addition to a lot of clean clothes. They’ve also just built an experimentation platform to support decision making. John Buccola, CIO of WASH Laundry, describes their approach: “Through experimentation and through data science, we’re trying to come up with the ideal setting to optimize the experience for our customers, both from a revenue standpoint for our customer partners, and from a laundry room experience for our residents, using IoT, using data science, and using experimentation, so we can get to that end result. Because there’s not a lot of research on this, we’re going to have to just try and figure it out.”

What makes experimentation such a powerful tool for these organizations?

Experimentation is powerful when it deepens managerial intuition. At Edmunds, the testing culture is pervasive. As Potloff describes: “People are actually more savvy now because they think more deeply about the metrics when they can pull them themselves than when they’re just asking a business analyst to pull some reports for them on a product. So, they have to actually think through the touchpoints, the task completion, metrics, more deeply than have they had in the past. And I think we have a much better product team as a result of that testing culture.” Managers can quickly test their insights, either validating their thinking or sending them back to think more, then swiftly bring changes to scale.

Experimentation is powerful when the organization has unique data. WASH Laundry has a repository of data that no individual location can match. Buccola describes the advantage it offers: “Are we getting incremental revenue because we’re actually adding loads, or are we getting the revenue because they’re just adding more cycles, or more cycle time in the case of a dry? Was it from the change we instituted or did we see the same in another cohort of locations that were similarly-situated? We can actually isolate and determine whether or not an experiment was successful.” In this case, WASH Laundry’s ability to compare locations creates an advantage that no individual location can match.

In both cases, the experimentation platform is embedded in routine managerial processes. The organizations have made it easy, both technically and culturally, for managers to test ideas and learn from them.

I don’t think we’ll see sci-fi movies depict data scientists the way that that they have portrayed other types of scientists in the past. It is too much to expect that data-based experiments would provide visual sparks from a Hadoop cluster or maniacal laughter from data analysts. But experimental platforms can spark change and at least a small smile when they allow organizations to continuously make small improvements.