A/B Test Like a Pro #3: Understanding Experiment Results Video Lecture | Introduction A/B Testing: From Experiment to Result - Software Testing

hey there app developers have you been
running an a/b test with their fancy new
AV testing feature and are dying to find
out what all those numbers and charts
actually mean well you've come to the
right place we're gonna take a closer
look at how to analyze the results of
your a/b test in the firebase console
are you excited I am you should be too
come on so if you've been following
along with this video series you've
probably gotten your app ready to
perform AV tests by adding a remote
config and analytics to your app then
you've run your first AV test by
creating an experiment in the firebase
console and now you're probably
wondering okay so how can I get my
results so you'll probably start seeing
results trickle in after about half a
day but here's the thing while these
results will probably be super
interesting to see and I totally
recommend you check them out don't go
making any changes to your app until the
experiment is complete and a/b testing
has told you that it's found a winner
see whenever you make a user visible
change your users are going to notice it
and they're going to act on it not
because it's necessarily better but
because it's different and it's grabbed
their attention but that doesn't mean
it's going to be the right move in the
long run so looking at my experiment if
I changed my background color from like
orange to green it's totally possible
that in the long run this doesn't work
out maybe we find that people just
prefer that orange color better but it's
also quite likely that in an experiment
where I tried changing the background
color to green people will notice the
new panel color and click on that
sign-in button again not because it's a
better change but because it's different
so to avoid giving you misleading
results like this firebase has generally
determined that it needs to wait about
two weeks for this short term bump to
level off
it also helps make sure we account for
things like variances within the week as
far as how people interact with your app
and that's generally why even if your
experiment results seem to dramatically
favor one variant over the others a B
testing might tell you it's not ready to
declare a leader just yet
you really shouldn't wait until a B
testing has told you it's found a clear
winner in your experiment before you
start making changes now luckily it's
been at least two weeks since I first
started my a/b test so I think at this
point it's safe to check in and see how
things are going so I'll jump into the
fire
console select my app and go to the
remote config panel and then I'll select
the a/b testing tab now I can select my
experiment from the running experiment
section and well I basically have zero
results here and that makes sense
because I was testing this on a sample
app that's got a monthly active user
count of one but even if I were to try
this with like a dozen test devices that
still wouldn't be enough for a
meaningful test see it's not enough for
firebase a be testing to tell you hey
this group got better average results
than this other group it's got to tell
you yes this group got better results
than this other group and I'm also
reasonably sure that these differences
are actually due to the changes you made
in your app and not just random chance
now it does this through a lot of fancy
math called Bayesian statistics that
maybe I think I understood at one point
back in college and have totally
forgotten about by now but trust me when
I say it's very fancy and one big factor
driving these fancy statistics is making
sure that it's had the opportunity to
measure these changes in front of a
large group of people the larger the
group of people the more random chance
gets factored out of the equation so if
you're gonna try and run an experiment
with like a few dozen QA folks or in my
case a handful of test devices that's
not really going to be enough to get you
meaningful results you're gonna need to
try this with an actual app in
production within actual audience and I
don't want to shock you but it turns out
that my app it isn't real I know
surprising right I'm gonna need to turn
to an app that has actual users out
there in the real world incidentally if
you are testing this with a real app in
the real world and you're still not
seeing any results there's a good chance
you forgot to update your remote config
libraries in your pod file or your
Gradle file like I was telling you to do
in the last two videos remember you got
to do that so that remote config knows
enough to tell analytics about the
experiments that it's in anyway I'm
lucky enough to have access to a real
production app where we were able to run
some experiments this is a game called
bingo laughs that gets a few thousand
active users per month which should be
enough for them to run a real AV test
and get some meaningful results so in
fact the bingo blast folks recently
tried an experiment where they want to
find out if they disabled ads in their
game would that improve the retention
and keep people coming back into their
app well that seems like a worthwhile
experiment to try so let's take a look
at the results in the firebase
console and find out what some actual
real-world results look like
so I'm now looking at the remote config
panel for bingo blast and over here in
the experiment section we've got this
experiment called ads impact on churn
this is the one we're interested in
checking out so let's take a look at the
results so right up here this is the
most important line of the entire
experiment it basically gives you the
summary of your experiment in plain
English and this is telling me that as
far as it could tell stopping ads didn't
seem to affect retention one way or
another and it's run this experiment
enough that it's pretty confident it
won't find a meaningful difference in
the future if it were still running and
not yet ready to present results you'd
probably see the word yet in there
somewhere like we haven't found a clear
leader yet but in our case it's saying
there's no clear leader in achieving our
goal which one is actually kind of
normal a lot of experiments you run will
probably end up without a be testing
being able to declare a definitive
winner but two is actually still kind of
interesting when you think about it it
lets me know that my interstitial ads
aren't driving away customers which is
valuable info you don't have to have a
clear winner for you to learn something
from your experiment but let's take a
closer look at our data because I bet
you there are still some interesting
insights we can get out of this
experiment that we run so first if I
click on this details drop-down I can
see more details about the experiment
this really just gives us a summary of
what our variants consisted of and what
changes we made for each one and this
basically covers everything we talked
about in the previous video and that's
great and all but I'm more interested in
our actual results so let's take a look
at the next section now this improvement
overview section gives us a summary on
how each of our experiment variants did
compared to our control group based on
whatever factors we chose to measure in
our experiment now this first column
will always be the primary goal of our
experiment and in bingo blasters case it
was their apps one-day retention so
right here we can see the range of
improvement this variation has over our
control group now you might ask yourself
why is this a range and not like a
single number telling us how well this
variant did that's because it turns out
that predicting the future it's a little
tricky I mean just because some variants
saw a 5 percent improvement in retention
on average during the experiment that
doesn't mean that's what you're going to
get in the future that depends a lot on
the sample size the consistency of the
results that were observed
throughout the experiment and a whole
bunch of other factors that go into that
whole fancy mathematics thing I was
talking about earlier so instead a be
testing generally prefers to give us a
range of values that it's fairly
confident you will see in the future
based on what it's calculated so far
sometimes these ranges can be very wide
other times they can be quite narrow
like it's one thing to tell you hey I
think these results will get you between
a four and six percent improvement and I
think these results will get you between
a thirty-five and negative twenty five
percent improvement like sure they both
average out to a five percent
improvement but only one of these seems
certain enough that I'd be okay changed
my app based on those results so going
back to the results in our bingo blast
app what a B testing is telling me is
I'm reasonably confident that based on
the results I've seen so far this
variant will give you somewhere between
a 7% decrease and a 13% increase in your
user retention well that's a pretty big
range which is really why it's telling
me there's no clear leader here in cases
where it was more confident I would
expect this range to narrow considerably
that's also why our results are in this
unimpressive looking gray if we had
results where a B testing was more
confident they were going to be better
or worse you'd see these values in green
or red along with some kind of arrow now
I can hover over this range and get some
more details like the actual median
value in some other ranges but honestly
I tend not to use these too much and
then in these columns here I've got
improvement stats for all the other
metrics I chose to measure you can see
I'm also measuring user engagement
crashes purchase revenue and so on and
it's important to look at these numbers
in addition to the primary goal of your
experiment as I mentioned in the last
video if I end up improving the main
goal of my experiment but like totally
tanking my retention or something that's
not really a successful experiment so
you should make sure you view all these
values together to get sort of a more
holistic view of all your results you
might notice as I click on these metrics
by the way I can see more details about
them in the section below so let's take
a closer look at the details for our one
day retention metric now on the very
left you can see each of my variants
starting with my control group along
with the number of users that were
placed in each one and if everything's
working as planned these should be
roughly the same size in the next column
over I have the same status above the
range of improvement that a B testing is
pretty sure my action
results will fall into in the future
next to it the probability to beat
baseline column is basically how likely
a be testing thinks this variant will
perform better than our control group
for this particular measurement now when
you first start this experiment and it
knows nothing else about the results
this will be 50% because you know it's
basically a coin flip right and now that
it's run a while it's saying that my
test case has roughly a 71% chance of
being the better option and honestly
that's generally not enough of an
indicator to be something I would
consider actionable I like to see this
value either in the high 90s or the low
single digits before I'm confident
enough to really rely on these results
now probability to be the best variation
is essentially a way of saying hey out
of all the variants in your experiment
which of these is likely to do best now
in our case we have only one variant and
the control group so these numbers will
be exactly the same as the probability
to be baseline numbers but if I had
multiple variants it wouldn't be so like
take a look at the snippet from another
experiment here notice that both these
variants seem to perform better than our
control group but this variant here is
the most likely to be the best out of
all of them
now this retention rate column here is
looking at the absolute percentages
instead of the percent improvement
numbers so for instance here at saying
my control group will have a 1-day
retention rate of roughly 27 to 31
percent while in my variant it's roughly
28 to 32 percent next to it this last
column is essentially the accumulative
value of the thing we're measuring for
attention it's the total number of users
in our experiment who came back the next
day for a particular event it would
probably be the number of times an event
was triggered and for user engagement if
the total time spent within the app
across all of our users I'll be honest
though I usually don't look at these
last two columns very much they're more
just kind of interesting background info
than anything else and in some cases
you'll also see a graph that shows off
the same data if you happen to be more
of a visual person because you know
graphs are always pretty so these
results are kind of interesting and it
does help us answer the question of does
removing ads improve retention and our
case a be testing is telling us not
really in a way that I could measure so
does this mean there's no difference at
all between the versions not necessarily
there is a subtle but important
distinction between I can confidently
tell you that both vary
performed similarly and I think some
variant might perform better but I'm not
confident enough to stake my reputation
on it and in our case honestly it looks
to be a little of both if there is a
change it's going to be minor and
because it's minor it's hard for a be
testing to really tell me confidently
that one variant will be better but
let's take a closer look at some of
these other measurements to see if any
of those can give us more information
over here and daily user engagement it
looks like people are generally spending
about the same amount of time in my app
whether or not they're seeing ads I
guess maybe they're spending about 30
seconds more in my control group which
honestly might just be the amount of
time they're watching these
interstitials who knows over in our
crash 4 users report it looks like
there's pretty much no difference
between the two groups which is kind of
what you'd expect unless it turns out
that like AdMob was crashing or
something also it looks like we got a
pretty solid app nice work there
engineers for purchase revenue it looks
like it's nearly impossible to tell
whether or not there's a meaningful
difference between the two groups
and if you're wondering why this range
is so large it's probably due to the
fact that only a small segment of my
population is a paying user so we're not
going to have nearly as much data as we
would for other stats and that's
something you should keep in mind if the
goal you're measuring is something that
doesn't happen very frequently within
your app you may need to target a larger
percentage of your audience than usual
to get the results you need so in my
case while it's showing that our ad free
version might have performed better when
it comes to in-app purchases that's a
really big range it has there and this
percentage to be the best variant still
isn't at the point where I'd be
confident making decisions based on that
like I said I really like to see this in
the high 90s
it looks like my longer term retention
isn't that different either between the
two variants again there's nothing here
to suggest that one version is
definitively better than the other so
when you're all done with your
experiment you have some options up here
around what action to perform next you
can stop the experiment which means
you'll no longer be running this a/b
test or collecting ending data you can
also duplicate the experiment if you
want to run it again or run it again
with some changes maybe try some more
dramatic changes and see how that
affects your results you'll often see an
option to increase the distribution
which will push your experiment out to a
wider audience if your current
experiment is targeting only 5% of the
population and you feel like you need
more data you could alter it to target
15 or 20% something like this your last
option is to
all out to 100% now rolling out to 100%
doesn't mean you're putting more people
into your experiment it's not the same
as increasing your distribution it
basically means you're declaring a
winner that is you're going to stop this
experiment take the settings from one of
your variants and make those the
official new values in remote config for
all your users moving forward if you
have a winning variant in your
experiment you'll probably see it
selected by default in this drop-down
list but you know you can go ahead and
select any variant you want so should
the makers of bingo blast remove their
interstitial ads from their game well
I'm leaning towards no yeah there are
some indications that removing these ads
has a minor positive impact on their
attention and you know if the bingo
blast people didn't have to give up all
their ad revenue to get these
improvements they might want to make
this change anyway even though a/b
testing couldn't definitively say that
it was better but you know the fact of
the matter is that a B testing isn't
confident enough in these results that I
would be okay saying y-yes I'm happy to
give up all my in-app ad revenue in
favor of these changes honestly I think
I'd want to see a corresponding uptick
in my purchase revenue before I really
turned off all those ads so if I were
them I'd probably stop the experiment
and not make any permanent changes to
the app as far as advertising goes and
again I think an important takeaway here
is that sometimes having no clear leader
and experiment is just as valuable as
having a winning variant so hopefully
this gives you a better idea of what
kind of data you can get from an a/b
test in firebase and how you can best
interpret these results to make the
smartest decisions within your app so as
usual if you enjoyed this video be sure
to subscribe to the firebase YouTube
channel for more like it and if you've
learned something interesting or
surprising about your own app by
conducting an a/b test tell us in the
comments below in the next episode we'll
describe how you can use a be testing to
experiment with your notifications to
should be pretty useful so stick around
[Music]