Top Challenges from the first Practical Online Controlled

Experiments Summit

Somit Gupta (Microsoft)

1

, Ronny Kohavi (Microsoft)

2

, Diane Tang (Google)

Eytan Bakshy (Facebook), Niall Cardin (Google), Sumitha Chandran (Lyft), Nanyu Chen (LinkedIn), Dominic Coey

(Facebook), Mike Curtis (Google), Alex Deng (Microsoft), Weitao Duan (LinkedIn), Peter Forbes (Netflix), Brian Frasca

(Microsoft), Tommy Guy (Microsoft), Guido W. Imbens (Stanford), Guillaume Saint Jacques (LinkedIn), Pranav Kantawala

Online controlled experiments (OCEs), also known as A/B tests,

have become ubiquitous in evaluating the impact of changes made

to software products and services. While the concept of online

controlled experiments is simple, there are many practical

challenges in running OCEs at scale. To understand the top

practical challenges in running OCEs at scale and encourage further

academic and industrial exploration, representatives with

experience in large-scale experimentation from thirteen different

organizations (Airbnb, Amazon, Booking.com, Facebook, Google,

LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and

While there are papers from individual organizations on some of

the challenges and pitfalls in running OCEs at scale, this is the first

paper to provide the top challenges faced across the industry for

running OCEs at scale and some common solutions.

1. INTRODUCTION

The Internet provides developers of connected software, including

web sites, applications, and devices, an unprecedented opportunity

to accelerate innovation by evaluating ideas quickly and accurately

using OCEs. At companies that run OCEs at scale, the tests have

LinkedIn, Twitter) to travel services (e.g., Lyft, Uber, Airbnb,

Booking.com), OCEs now help make data-driven decisions [7, 10,

61, 76, 12, 27, 30, 40, 41, 44, 51, 58].

1

Organizers email: [email protected]

2

[email protected]

3

[email protected]

experimentation from thirteen different organizations (Airbnb,

Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft,

Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford

University) were invited to the first Practical Online Controlled

Experiments Summit. All thirteen organizations sent

representatives. Together these organizations have tested more than

one hundred thousand experiment treatments last year. Thirty-four

experts from these organizations participated in the summit in

Sunnyvale, CA, USA on December 13-14, 2018. The summit was

chaired by Ronny Kohavi (Microsoft), Diane Tang (Google), and

participate in at least two breakout sessions. Each breakout group

presented a summary of their session to all summit participants and

further discussed topics with them. This paper highlights top

challenges in the field of OCEs and common solutions based on

discussions leading up to the summit, during the summit, and

afterwards.

1.2 Online Controlled Experiments

Online Controlled Experiments, A/B tests or simply experiments,

are widely used by data-driven companies to evaluate the impact of

software changes (e.g. new features). In the simplest OCE, users

feature X. External factors such as seasonality, impact of other

feature launches, or moves by the competition, are evenly

distributed between Control and Treatment, which means that we

can hypothesize that any difference in metrics between the two

groups can be attributed to either feature X or due to chance

resulting from the random assignment to the variants. The latter

hypothesis is ruled out (probabilistically) using statistical tests such

as a t-test [21]. This establishes a causal relationship (with high

probability) between the feature change and changes in user

behavior, which is a key reason for the widespread use of controlled

simple, deployment and evaluation of OCEs at scale (100s of

concurrently running experiments) across variety of web sites,

mobile apps, and desktop applications presents many pitfalls and

research challenges. Over the years some of these challenges and

pitfalls have been described by authors from different companies

along with novel methods to address some of those challenges [16,

23, 43, 48, 50, 61]. This is the first time that OCE experts from

thirteen organizations with some of the largest scale

experimentation platforms have come together to identify the top

challenges facing the industry and the common methods for

companies across the industry. Later sections discuss specific

problems in more detail, how these problems have an impact on

running OCEs in different products, and common solutions for

dealing with them.

2. TOP CHALLENGES

Software applications and services run at a very large scale and

serve tens to hundreds of millions of users. It is relatively low cost

to update software to try out new ideas. Hundreds of changes are

made in a software product during a short period of time. This

provides OCEs a unique set of opportunities and challenges to help

the same across these sources. These top challenges reflect the high

level of maturity and scale of OCE operations in these companies.

There is literature out there on the challenges and pitfalls some of

these companies faced and solved during the early years of their

operations [32]. The challenges mentioned here are at the frontier

of research in academia and industry. While there are some

solutions to existing challenges, there is a lot of scope for further

advancement in these areas.

1. Analysis: There are many interesting and challenging

open questions for OCE results analysis. While most

term factors without having to wait a long time in every

case? What should be the overall evaluation criterion

(OEC) for an experiment? How can we make sure that

the OEC penalizes things like clickbaits that increase user

dissatisfaction? While there are methods to test an OEC

based on a set of labelled experiments [24], how to best

collect such set of experiments for evaluation of the OEC

experiments, most of the low-hanging fruit get picked

quickly and many Treatments may then cause a very

metric may imply a change of millions of dollars in

product revenue. How can we best increase the

between good and bad Treatments [18, 42, 75]? If we are

treatment effect on the entire population is not enough.

How can we best identify heterogenous Treatment effects

what is the best method to obtain a good estimate of the

4. Data quality: Trustworthiness of the results of an OCE

cookies and we are left with tracking a biased sample of all users

exposed to the variants [23]. Further, a user may access the same

service from multiple devices, and the user’s friends and family

may access the same service. As time goes on, a user or their friends

or family may be exposed to both the treatment and control

experience during an experiment, which dilutes the impact of the

suspect that there is a short-term novelty or user learning effect. At

Microsoft, while most experiments do not run for more than two

weeks, it is recommended to run an experiment longer if novelty

effects are suspected and use data from the last week to estimate

the long-term treatment effect [23]. At Twitter, a similar practice is

followed. An experiment at Twitter may run for 4 weeks and data

from last two weeks is analyzed. If a user exposed in the first two

Good proxies that are predictive of the long-term outcome of

interest are commonly used to estimate the long-term impact. For

correlation may not imply causation, and such proxies could be

susceptible to misuse, where a treatment may cause an increase in

the proxy metric, but ends up having no effect or regression in the

long-term outcome. It may be better to develop a mental causal

structure model to find good proxies. Bing and Google have found

proxies for user satisfaction and retention by having a mental causal

structure model that estimates the utility of an experience to users

[36, 49].

Another approach followed by Google is to explicitly model the

group is exposed to the treatment provides an estimate of user

learning from the treatment. Google also used cookie-cookie day

randomization to get an estimate of user learning for any duration

(in days) since the experiment started. In these experiments and in

the subsequent analysis, the authors carefully designed the

experiments and did careful analysis to ensure that they were not

seeing many confounding effects (e.g., other system changes,

system learning, concept drift, as well as selection bias issues due

to cookie churn/short cookie lifetimes). They took this information

and modeled user learning as an exponential curve, which allowed

them to predict the long-term outcome of a treatment using the

between the treatment and the long-term outcome. It satisfies the

condition that treatment and outcome are independent conditional

on the statistical surrogate. You can use observational data and

experiment data to find good surrogates. Even if no individual

proxy satisfies the statistical surrogacy criterion, a high-

dimensional vector of proxies may collectively satisfy the

surrogacy assumption [8]. Having a rich set of surrogates reduces

the risk of affecting only a few surrogates and not the long-term

outcome. Facebook used this approach with some success to find

good surrogates of the 7-day outcome of an experiment by just

4.1 Problem

can streamline the decision-making process and make it more

objective. Without understanding the causal impact of an idea on

regarding the change only relying on their own experience, recall,

and interpretation of certain business reports and user comments.

Person’s Opinion) [37] and is fraught with many cognitive biases

[2]. To help change HiPPO decisions to more objective, data-driven

experiments on your product.

Not all metrics computed to analyze the results of an experiment

are part of the OEC. To analyze experiment results, we require

different types of metrics [22]. First, we need to know if the results

feature or local metrics. These metrics help you understand the

source of OEC movement (or the lack of).

It is hard to find a good OEC. Here are a few key properties [19,

24, 55] to consider. First, a good OEC must be indicative of the

long-term gain in key product indicators (KPIs). At the very least

make it directionally accurate in estimating the impact on the long-

term outcome. Second, OEC must be hard to game and it should

incentivize the right set of actions in the product team. It should not

be easy to satisfy the OEC by doing the wrong thing. For instance,

if the OEC is limited to a part or feature of the product, you may be

the long-term outcome should also have a statistically significant

movement in OEC metrics so it is practical to use the OEC to

distinguish between good and bad changes to the product. Fourth,

the cost of computing OEC metrics cannot be too expensive. OEC

metrics must be computed for 100s of the experiments and be run

on millions of users each and every week. Methods that involve

costly computation or costly procedures like human surveys or

been a research subject for a long time in academia and for many

products including Bing, Google and Yandex. It is well established

that metrics, like queries per user, cannot be a good OEC because

queries per user may go up when search ranking degrades. Sessions

per user or visits per user are considered better OEC metrics [49].

In general there is an appreciation of focusing on HEART

(Happiness, Engagement, Adoption, Retention, and Task success)

metrics for the OEC and use PULSE (Page views, Uptime, Latency,

Seven-day active users [i.e., the number of unique users who used

the product at least once in the last week], and Earnings) metrics as

homepage, or Yandex homepage. The challenge is to understand

user intent. Sometimes users will come with a goal-oriented intent

and would like to quickly find what they are looking for. Other

times users may have a more browsing or discovering-new-

information intent where they are not looking for something

specific but just exploring a topic. In this case it is not clear if lack

of a click on an article link with a summary snippet can be viewed

as a negative experience or positive because users got the gist of

the article and did not have to click further. Further the two intents

(goal-oriented and browsing) can compete. If a user came with a

have clarity on product goals and strategy alignment across the

entire team. This includes decisions like defining who the customer

is and how best to serve them. Further, your team must also create

a monetization strategy for the product. In absence of such clarity,

each sub team in the product group may set their own goals that

may not align with other teams’ or corporate goals.

Even after the product goals are clear, in most companies you end

up with a handful of key metrics of interest. It is challenging how

to weigh these metrics relative to each other. For instance, a product

may have goals around revenue and user happiness. If a feature

desirable to ship this feature? This decision is often made on a case-

by-case basis by leadership. Such an approach is susceptible to a

lot of cognitive biases, and also may result in an incoherent

application of the overall strategy. Some product teams like at Bing,

tried to come up with weights on different goals to make this

tradeoff align with the overall product strategy and help ensure that

consistent decision-making processes are used across multiple

OEC and ensure that it retains the properties of a good OEC. To

past experiments widely seen to have a positive, negative or neutral

impact. Changes to the OEC were evaluated on this corpus to

ensure sensitivity and directional correctness [24]. Microsoft and

Yandex successfully used this approach to update OEC metrics.

The challenge here is to create a scalable corpus of experiments

with trustworthy labels.

Other approaches include doing degradation experiments, where

metrics to ensure that they are not gameable.

Some product teams tried to incorporate machine learning models

methods are more commonly used in mature product areas, like

gamed by optimizing for the underlying model that created these

Without loss of generality, we consider the case that there is only

platforms. Using device and platform segments in A/B

testing is essential to flag potential bugs using live traffic.

For example, in a recent experiment, a feature of the

Outlook mobile app was moving key metrics on all

Android devices except a few versions, which indicated

further investigation was needed. Device and platforms

also represent different demographics. Many studies

4. Time and day of week: Another common segment used

is time. Plotting the effects delta or percent delta by day

can show interesting patterns, such as the weekday and

weekend effect, reveal a novelty effect [13], and help flag

hundreds or thousands of metrics on millions of

experiment units (users), the resources and time spent on

an automatically scheduled analysis cannot be too much

to ensure that results are not delayed and are not too

expensive to generate. There is a desire to use a simple

algorithm directly formulated using sufficient statistics,

instead of using individual-unit level data.

2. Low Signal Noise Ratio (SNR): A/B testing is already

dealing with low power to estimate the average treatment

effect. Learning HTE is even harder than learning ATE

because of the reduced sample sizes in each

4. Interpretable and memorable results: Most

experimenters are not experts in statistics or machine

learning. You must have concise and memorable result

summaries to facilitate experimenters to act.

5. Absolute vs. Relative: While determining the HTE, you

must decide whether you will use absolute CATE or

relative CATE (as a percentage of average value of the

metric in control). In many cases it makes sense to use

the relative CATE as the baseline or the average value of

a control metric can be very different for different

segments, like different countries. Use a relative CATE

one-off analysis, linear regression with sparsity (L1 and

elastic net) and tree-based algorithms, like causal tree, are

very popular. Double ML also gained a lot of attention

recently [14].

2. Because of the challenge of low SNR and multiple

testing, sparse modeling is a must. Even if the ground

truth is not sparse, there are limited resources that

experimenters can spend on learning and taking actions

based on HTE. Sparse modeling forces concise results.

3. To make results memorable, when certain segment has

and the rest of the world, and report only two different

effect estimates. Algorithms that can perform regression

and clustering is preferred in these cases, including Fused

Lasso [69] and Total Variation Regularization.

Another difficulty in acting based on HTE results is more

fundamental: HTE results are not causal, only correlational. HTE

is a regression to predict individual treatment effect based on

devices by other factors would be more useful. Historical patterns

information.

6. DEVELOPING EXPERIMENTATION

CULTURE

6.1 Problem

norms define what is encouraged, discouraged, accepted, or

rejected within a group [35]. There is a big challenge in creating

an experiment-driven product development culture in an

organization.

intuition gets challenged. Finally, a culture develops where there is

humility about our value judgement of different ideas, and better

understanding of the product and customers [3].

experiment results feel like someone telling you that your baby is

ugly. You must enact a paradigm shift to put your customers and

business in focus and listen to customer responses. At that point,

negative experiment results are celebrated as saving customers and

your business from harm. Note that not only bad ideas (including

bloodletting [11]) appear as great ideas to a human mind, we are

also likely to discount the value of great ideas (including good hand

hygiene for doctors [65]). There are cases where an idea that

languished in the product backlog for months as no one thought it

was valuable turns out to be one of the best ideas for the product in

trustworthy analysis of this experiment quickly, and interpret the

results to take the next step: ship the feature, iterate, or discard the

idea. A strong experimentation culture ensures that all changes to

the product are tested using OCEs and teams benefit from OCEs

discovering valuable improvements while not degrading product

quality. It allows you to streamline product development

different ideas with minimum viable improvements without having

to sell the entire team on the idea beforehand. And allows the team

to make future decisions to invest in a product area based on

changes to the OEC metric due to features seen in that area.

6.2 Common Solutions and Challenges

test the hypothesis of interest to the product team. To do that one of

the of the most important things required is a set of trustworthy and

easily interpretable metrics to evaluate a change made to the

product. In addition, it’s useful if there are easy tools to manage

multiple experiments and clearly communicate results from these

policies, and capabilities to encourage teams to test every change

made to their product using OCEs. The following are strategies that

different companies use to achieve this goal.

handpicks a few business-critical teams, prioritizes these teams,

and then works closely with them on their needs. At the end of the

quarter the team agrees they’ll use that experiment platform going

forward, and the experimentation team continues to monitor them.

Over several years a data-driven culture is built. Managers and

directors now rely on development teams running experiments

The downside of the High Touch approach is the large overhead in

having a deep engagement with every team, and it may become a

bottleneck for scaling.

experimentation by leadership and they expect every change tested

in a controlled experiment. Further they can set team goals based

on moving a metric in controlled experiments. This creates a culture

where all ship decisions are talked about in terms of their impact on

key metrics. The product teams celebrate shipping changes that

improve key metrics, and equally importantly, celebrate not

shipping changes that would cause a regression in key metrics. It is

experimenters, product managers, and leadership teams before an

experiment is “rolled out”.

negative results where a feature that is widely acclaimed as a

positive causes a large regression in key metrics, or a surprising

positive incident where a small change no one believed would be

of consequence causes a large improvement in a metric were great

drivers for cultural change. These cases drive home a humbling

point that our intuition is not a good judge of the value of ideas.

There are some documented examples the best OCEs with

surprising outcomes [4]. For instance, an engineer at Bing had the

in the idea. When it was finally tested, it caused one of the biggest

don’t have the feature turned on. During this controlled feature

rollout, the feature’s impact estimate on key reliability and user-

behavior metrics helped find bugs.

level [31]. This report card gives the team a way to think about the