A/B Test Like a Pro #2: Creating an Experiment Video Lecture | Introduction A/B Testing: From Experiment to Result - Software Testing

oh hey their app developers didn't see
you there so if you've been following
along with our series you've already
learned about how to wire up your app to
use remote config so you can create an
a/b test as well as how to add Google
Analytics for firebase into your app so
that firebase can measure the results of
that test so now we've done all our prep
work why don't we create an a/b test for
real now as you recall from our last
video I've got this great little sample
app where I have this little panel that
slides up and asks our user whether or
not they'd like to sign in and I've set
up my apps that it can change a whole
bunch of values through remote config
everything from the content of this text
to the color of these labels to the
speed of the slide in animation and so I
want to run an experiment to see what's
going to be most effective for getting
my users to sign in so let's head on
over to the firebase console and make
this happen also I mentioned this in the
last video but I'm gonna want to make
sure I have a recent version of remote
config installed so that it knows to
tell analytics about the experiments it
might be running and that means running
pod update on iOS and updating your
Gradle file on Android and you can refer
to the documentation to find out the
exact minimum versions if you need them
so from the firebase console I'm gonna
select my project and head on over to
the remote config panel now keep in mind
our UI is always being updated and these
screens might look a little different
than what you're seeing today but they
should be similar enough that you get
the general idea anyway you might notice
here on the right there's a new a/b
testing button here so I'm gonna click
on it and this panel slides in and it'll
tell me what experiments I currently
have running which ones are still in
draft form and which ones I've completed
well right now I don't have any
experiments running anywhere so I have
this nice little create experiment
button at the bottom so let's click that
and start creating our very first a B
test okay so these first few fields
they're pretty straightforward I'm going
to call this our sign-in panel test and
give it a short but relevant description
there we go and now for our target users
I'm going to need to select a specific
app note that it can't run the same
experiment on say my iOS and Android app
at the same time but if you think about
it that kind of makes sense right these
platforms have like different use
patterns your app probably differs
between platforms and what works well on
one
platform won't necessarily work well on
another so I'm going to select my iOS
app here just to run the experiment on
iOS devices and now I'm going to click
the + button to refine my audience
further I might want to limit my
experiment to only certain users like
those belonging to a certain analytics
audience or those who have a particular
version of my app installed in my
specific case since I'm going to be
trying out some text changes I'm going
to limit this to users who have their
device language set to English like you
don't want say your french-speaking
users to start seeing these English
labels that you're running AV tests with
that would just kind of mess up your
results now it just so happens that my
app is already using rural config to
power every variable that I want to test
but my guess is that there will be times
when you're going to be creating
brand-new values in remote config for
the specific purpose of running an a/b
test and in those cases you'll probably
want to limit your audience only to
users who have the app version where
you've added in these new values and so
you'd be doing that by selecting version
here from the drop-down list next I'm
being asked what percentage of my
potential audience to put into this
experiment now if I made this a hundred
percent that would basically put my
entire app population into the
experiment but it's usually best to keep
these kind of small if my experiment
were to introduce like major bugs or
mess up my in-app economy or trigger
some major backlash for my community I'd
want to make sure it's only affecting a
small portion of my user base so I'd
recommend keeping this value in like the
five to ten percent range depending on
how risky you think your changes are
okay now we get into the real meat of
the experiment and that's the variance
this is where I decide for each of the
variants in my experiment what different
remote config dal use they're going to
receive so to start I'll click the add
parameter button and now it's asking me
what parameter I want to change in our
experiment I think the first thing I
want to mess with is this call to action
label maybe this save your data across
multiple devices text isn't that
compelling this might not be an
important use case to my users or they
might not even understand what it means
now the name of this parameter in my
remote config setup is sign-in request
text so I'm going to copy and paste that
directly from the code just to make sure
I get it right note that if you have
used other parameters with rural config
in the past they'll be shown here in a
drop-down list as suggestions but you
can also go ahead and type in new values
that you haven't used
four and that's fine too so I'll select
create parameter and now I have these
fields to fill in so for our control
group I'm gonna leave this as no value
now this doesn't mean I'm going to be
passing an empty string like don't worry
that all of a sudden my text is gonna be
blank this is just a way of telling a
row config go ahead and use whatever
value you'd use as if you weren't in
this experiment and so for us that's
going to be the default value that we've
supplied in the code and I do generally
recommend going with this no value
setting for your control group along
those lines I also recommend not mucking
with this value much in the normal
remote config console while your
experiment is going on because like if
your results are changing in your
control group while the experiment is
happening that could definitely mess up
your results as for a first variant
let's change our text to maybe emphasize
that by signing in you can still get
access to your data in the event that
you lose your device maybe I'll say
something like keep your data safe and
secure by signing in let's type that in
okay we also get a chance to name our
variant this is really just for our
benefit when we're looking at the
results later so we can remember what we
changed in each variant and so I will
call this safe and secure so this right
here this is a basic a/b test I've got
two test groups the first group is
seeing the original call to action text
and my test group is seeing a modified
version of that text and honestly this
would be enough to start an experiment
but I could go a little deeper than this
for starters I don't need just one
variant if I want to test out several
different versions of this call to
action text at once I could just by
adding new variations kind of like this
but I could also change more than one
variable at once for example what if I
decide to change both the text and the
background to color of the panel well I
could do that by adding a slight eview
color parameter and changing that as
well and while this could work you need
to be kind of careful here if this
variant ends up being more successful
well I still don't really know why right
is it the new text or is the background
color I honestly don't know and in fact
maybe this change would have worked
better if I hadn't changed that
background color so if you're going to
change multiple parameters at once my
recommendation is to do this when the
variables are related and you can't
really change one without also changing
the other for instance if I were to
change the background color to something
dark well I'd probably need to change my
text color to something that's kind of
whitish that the text is still legible
and in this case it would definitely
make sense to group these two changes
together in like a dark color scheme
alternately I can just make sure that I
have my bases covered
by making sure I've got all combinations
covered here in my variants for example
in this experiment you can see that I'm
changing the button text in this first
variation but then leaving my slide
eview color unchanged then I'm changing
the slighty view color parameter in my
second variation while keeping the text
unchanged and then finally I'm changing
both the background color and the text
in the third variation for those of you
who aren't hip to your AV testing lingo
this is known as multivariate testing
and I think this might be a fun
experiment to try so you know what I'm
gonna go with this once setting our
variants is done our last step is to
tell a be testing what our goal is for
this experiment and you can see here at
the top of this drop-down list it's got
a bunch of high-level goals things like
increasing engagement the amount of time
people spend with my app or in-app
purchase revenue but if I were to scroll
down further I could start to see some
of the events that I've been recording
in Google Analytics for firebase and
right here is my custom user signed in
event since the point of my experiment
is to see if I can get more people to
sign in I think I'm going to select that
you can see that a be testing also by
default includes some other metrics to
track looks like right now it's
including engagement retention that is
how many people come back to our app
after several days and minimizing
crashes and this is all good information
to include in my experiment because I
want to make sure I don't mess up any of
these other metrics when I'm optimizing
for my sign-in I mean I'm sure I could
get like super in-your-face and
aggressive with my sign in messaging and
that would probably give me a boost and
sign in but if that change also ends up
driving people away from my app well
that probably wouldn't be considered a
successful experiment right so it's good
to make sure I'm not hurting my apps
fundamentals when I initiate a test like
this okay so I'm gonna click the review
button and I can see a summary of my
experiment right here everything's
looking pretty good but how do I know
for sure I should probably make sure my
text fits the space I have reserved for
this label and that this green color
isn't too garish
luckily a/b testing makes testing these
variants easy what we need to do is grab
the instance ID token of our application
that is the unique identifier assigned
to an instance of our app running on a
particular device and then we
can tell remote config to deliver a
specific variation to that specific app
instance so then we can try out each
variation before we publish our
experiment so getting our instance ID
token requires a tiny bit of code it's
basically this one line on iOS and this
one line and Android and so I'm going to
jump into my app for a second and I'm
going to stick this line of code into my
app delegate and then run it and now I
can grab my instance ID token right here
from the console output it should be a
giant string that looks a little
something like this next I'll go back to
my experiment in the firebase console
and click the manage test devices link
here at the bottom now in this dialogue
I'm going to take this giant string I
copied and I'm gonna paste it into the
instance ID token field and then it can
decide which variant I want to push to
this device maybe I will try out myseif
and green variations so I can test both
the new text and the new green color at
once so click Add and then I do also
need to make sure I click Save okay now
I'll run into my device and mmm well
depending on how things are set up it
might look exactly the same as before
and that's because if you remember from
our earlier remote config the remote
config library will cache values that
downloads from the cloud for about 12
hours by default before it fetches new
ones now in a typical production app
this is just fine but for testing
purposes this can sometimes make things
a little more difficult so I'm going to
go ahead and make sure I have developer
mode enabled and my cache expiration
time set to zero and I'm also going to
add a giant warning here to remind
myself that I shouldn't ship any
production app with this code in it or
my app will get throttled by the remote
config service in fact I will probably
get reset all of this as soon as I'm
done testing anyway let me give this
another try this time my app is grabbing
new values from the cloud remote
configures giving it our specific
experiment variation and there we go
look at that there's our new background
and our new test message nice how
persuasive I think I do want to sign it
and if I wanted I could go ahead and
test out all my other variations through
the same manage test devices dialog I
can just select a different variation
from this drop-down right here hit save
and just like that I'm testing my
variant with the new message on the old
orange background look at that and I'm
free to add more devices to that manage
test devices dialog so I can make sure
things look good on all
sorts of other types of devices or you
know I can use that to see the different
variations side by side all at once
by the way do keep in mind that this
instance ID token can in theory change
from time to time but it generally only
happens every few months or if you
reinstall your app but if you suddenly
define that none of your test devices
are getting the variations you're
expecting just double-check that their
token hasn't changed so I've gone ahead
and created my experiment I've added in
my variants and determined what remote
config file use to change for each one
of those variants we've defined our
success metric for this experiment and
then I verified that all my variants
look good through this managed test
devices dialog I think at this point I'm
ready to run my experiment should we go
for it let's go for it
I should probably note that once we
start an experiment we can't change any
of the values for any of the variants we
also can't push out specific variants to
specific test devices anymore so just
make sure you've done enough testing
with your variants that you're happy
with how they look but you know what at
this point I'm pretty happy so I'm gonna
start my experiment and once I've done
that I get this page where it shows me
my results so far and as you might
expect I don't have a whole lot of data
yet so at this point my next step is to
sit back and wait we're going to need
some time for this experiment to run
both so that it can reach the right
amount of users and also they have some
time to get used to this change I'll
explain all that in our next video where
we talk about analyzing the results of
an experiment but for now go ahead and
make your own even if it's something
silly or frivolous it'll be a nice way
to get your feet wet then sit back have
a cup of coffee and I will see you in a
week or two
[Music]