A/B Test Like a Pro #1: Preparing for A/B Testing Video Lecture | Introduction A/B Testing: From Experiment to Result - Software Testing

quick which one of these buttons do you
think would get more people to browse
your in-app store are you sure
would you bet your entire company's
future on it cuz you know you're kind of
doing that everyday when you make
decisions like this so let's find out
how to make sure you're making the most
informed decisions with a little help
from a be testing from firebase okay so
what is a B testing and why should you
care I may have been a little
overdramatic in our intro but you do
make lots of decisions every day about
your app whether it's the look and feel
of a button the text that's designed to
get people to try out a new feature how
often you show ads or how challenging
you make level 12 in your game and while
individually these changes might seem
small together they can have a big
impact on your app success often meaning
the difference between a killer app that
feels just right versus a mediocre app
that never really gains traction but how
do you know you're making the right
changes I mean you're probably a little
too vested in your own app to see it
from the perspective of a typical user
and sure you could ask your friends your
family or various focus groups but that
can be kind of inefficient and you're
usually dealing with small and probably
biased sample sizes sorry mom also
frankly sometimes what people say
they're going to do within your app
doesn't actually match up with how they
really behave so another option is to
make a change in your app publish it to
the world wait a couple of weeks and see
if your users behave any differently so
this can work but it's also time
consuming if you wanted to say test
three or four variations I mean assuming
you want a couple of weeks to try out
each change that could be like two
months spent just trying out those
variations but more importantly how do
you know the change in behavior you're
seeing really resulted from the change
you made in your app I mean maybe you
ran a different ad campaign that week
that brought in a different set of users
with different characteristics maybe
your app got a mention in a blog post
that brought in a whole new set of users
or maybe the weather was just super nice
that week and people decide to spend
time outdoors and not on their phones it
can happen and so that's where a B tests
come in the idea behind an a/b test is
that you take a subset of your users
usually not all of them and break them
up into randomly assigned groups then
each one of these groups sees a slightly
different version of your app and you
can measure how your users interact with
your app in each one of these different
group
since these different groups are all
using your app at the same time any
external change like a particularly
sunny day will affect all of your users
equally so if there's a noticeable
change between the groups you can be
pretty confident the change in the
behavior that you're seeing is due to
the change within your app and not like
some external factor now firebase
recently made some major improvements to
their a be testing framework first they
added a whole bunch of enhancements up
front to make it easier for you to get
started with an a/b test and then they
added a whole lot of fancy math under
the hood to help you measure the results
and let you know that if these results
you're seeing are truly meaningful and
we're going to show you how to get your
app up and running with a B tests on
this new system over the course of the
next few videos now the first two touch
points that the team has opened up to
a/b testing are fire based notifications
and firebase remote config notifications
are great for testing well notifications
and remote config is good for testing
just about anything else that happens
within your app so we're gonna focus on
remote config in Ischl e in our video
series but we will touch on
notifications in a future video so the
first thing you're going to need to do
if you want to run a good ami test is
wire up your app to use remote config
now if you're not familiar with ro
config you can kind of think of it like
a key value store that lives in the
cloud your app can download this data
and change its behavior based on the
values that you've specified here now
we've already got some pretty good
screencasts on how to get started with
remote config for Android and iOS so if
you've never used a wrote config before
I recommend you check these out and you
can find links to them in the
description below if you are somewhat
familiar with remote config you might
remember that the process works a little
like this first you supply remote config
with a bunch of default values then
usually when your user starts up your
app the remote config service will go
out onto the internet grab any new
values that you've defined from the
firebase console and apply these new
values on top of your old defaults then
when you query remote config for
particular values and will either give
you an updated value that you've
specified from the cloud if one exists
or a default value if it doesn't and so
the strategy we generally recommend is
to take any value you might want to
change right like every bit of text
every color every game balancing
variable and wire them up to use remote
config remember because you're only
downloading the values that you've
changed the actual network calls
we'll still be nice and small but you
still have the freedom to change any
value you might want later this does
mean that for maximum a be testing
effectiveness you'll probably want to
spend a little more time setting your UI
elements by code instead of relying on
what you said within Xcode storyboards
or your Android layout files it does
seem like many larger organizations
prefer to do this anyway because dealing
with git merges is usually easier with
like code than with a generated XML file
but it turns out it also makes
everything a little more remote config
friendly too and as far as localization
goes you should still be fine remember
you're defining your default values
through code or through a cue list or
resource file so you can go ahead and
continue to use your localization
friendly code to set these default
strings or make sure that this external
file is among the resources that gets
sent off for translation either way
you'll end up with default strings that
are properly localized at runtime for
your current location incidentally since
you are gonna end up with a slew of
different keys for like every constant
you're defining within your app I do
recommend making sure you use something
like an enum or a bunch of static
constants to reference these keys
don't rely on like hand type strings cuz
one typo could really throw you off and
also you want to make sure you can use
your IDs Auto completion to help you out
here
but you probably already knew that so
let's take a look at my sample app here
this here is my front page isn't it
lovely and I've added in this little
slide up panel thingy that's going to
ask users to sign in if they haven't yet
and this is probably something I want to
experiment with to see what works best
for driving sign-ins so let's think
about how we would want to wire up
remote config within this app for
maximum a be testing effectiveness well
it looks like there's a bunch of values
I might want to tweak later obviously
I'll want the ability to change the text
of this call to action label as well as
maybe the text of these buttons this is
probably less critical but I might also
want the ability to change colors both
the background color of my view the
color of this label and maybe the color
of these buttons and then just to be
complete I've also got the content of
these two text labels going through
remote config as well as a general label
color variable that I'm using to set the
colors of all of my other labels and
finally I'm setting the content of this
image to the remote config too to do
that I'm creating a tiny little object
in JSON that specifies if this image is
local and if so what the local image
name is or remote and if so
URL to use keep in mind that remote
config doesn't support JSON objects
natively but I can certainly send them
across as strings and decode it on the
client and in practice that is what a
lot of developers do when they're using
remote config now obviously I could go
even further if I wanted right like in
theory I could probably get every aspect
of my app to use remote config
everything from like the font type and
size to the exact spacing and layout but
I think for my app this probably isn't
necessary but you should decide what
makes sense in your particular situation
now I don't want to derail this video by
getting into all the code that I put
into getting the sample app up and
running but honestly it's nothing too
fancy however if you are interested in
the implementation details I'll be
making a follow-up video where I walk
you through what I did and you will
eventually find that link in the
description below so I've got my app
wired up for remote config which gives
me the ability to change all sorts of
aspects about my app later but I'm not
quite ready to run an a/b test yet I
also need to determine whether or not my
changes are successful which means I
need to measure what's going on within
my app and for that I will need some
analytics now if you don't know anything
about Google Analytics for firebase the
product formerly known as firebase
analytics we also have some videos for
you to check out and if you're only
going to watch one I recommend checking
out our video on events because that's
primarily what you're going to be
dealing with and yes I know the video
title for it says iOS but like 90% of
the video is applicable to you Android
folks too so go ahead and watch so
Google Analytics for firebase works by
recording events that happen within your
app a number of these are recorded
automatically by the SDK things like
in-app purchases or add clicks which it
can use to determine revenue as well as
events like your user bringing your app
into the foreground which can be used to
determine retention and engagement but
you're probably also going to want to
record custom events that happen within
your app partly because well by
recording these events you can get a
better sense overall of how people are
interacting with your app but also
because you might want to use some of
these custom events as a way of
measuring success within your a/b test
see when you run AV tests on firebase
part of running that experiment is
telling firebase what you consider to be
a successful outcome
in some cases it might be earning more
money within your app in other cases
might be your users spending more time
within your app or coming back to it
more often but in other cases it might
be maximizing the occurrence of a
specific event
for instance if you want to see whether
a change in your app does a better job
of getting your users to complete your
in-app tutorial you probably want to add
a tutorial start and a tutorial
completed event and then see if you can
maximize the occurrence of that latter
event if you're messing around with the
look and feel of a sieve to my favorites
button you'll probably want to create an
added to favourites event so you can
maximize that so going back to my sample
app I think I'm going to add a signing
panel revealed event to record that I am
presenting the user with this little
panel but then I will also record a
sign-in event when my user is signed in
because that is an event I will probably
be interested in maximizing within my
experiments finally I'm going to want to
make sure I have a fairly recent version
of remote config installed so that it
knows to tell analytics about the
experiments it might be running that
means running pod update on iOS and
updating your Gradle file on Android and
you can refer to the documentation for
the exact minimum version numbers that
you need so I am now at the point where
I can create an experiment by changing
up a whole bunch of parameters within my
app through remote config I'm also now
able to measure the results of an
experiment by going to Google Analytics
for firebase either by looking at some
of the values that's recording
automatically or looking at some of my
specific events so I think I'm finally
ready to run an experiment for real and
we will look at doing that in the next
video so stay tuned
[Music]
you
[Music]