A/B Test Like a Pro #4: A/B Test Your Notifications; Too! Video Lecture | Introduction A/B Testing: From Experiment to Result - Software Testing

so up until now we've really only
focused on using a/b testing in
conjunction with firebase remote config
to help you a B test things that are
happening inside your app but it turns
out there are also things outside your
app that you might want to a b test as
well for example how about those
notifications that you're sending would
it be nice if you could a be test those
too not just to see which ones are
getting people to tap on them but also
persuading them to do important things
like spend quality time within your app
or make an in-app purchase well it turns
out you can use a be testing for that
too which is a good thing because
otherwise this would have been a
terrible intro let's find out how on
this episode of a B test like a pro so a
B testing works in conjunction with
firebase Cloud messaging to help you run
experiments with your notifications and
firebase Cloud messaging or fcm as the
kids like to say consists of a few
different pieces that all work together
first you have the back-end service
itself now this records which device
token corresponds to what kind of device
so it knows whether or not to route your
messages through APNs it's also got a
few other nifty features that I'm not
going to get into now it's a secret then
there's the client library which helps
register your device with cloud
messaging and sends information about
the notifications you received analytics
without your needing to do any extra
coding then you've got the notification
section of the firebase console this
basically is a tool that makes it easy
for anybody in your organization to send
a notification through FCM without your
needing to write any code to talk to the
service so you can compose your whole
message in there maybe determine what
subset of your users you want to send
this message to and even do nice things
like schedule your notification to be
delivered at some point in the future it
also provides you with some basic
analytics around your notifications now
for ordinary notifications you don't
have to use the firebase console in fact
many developers have written their own
tools to send messages over fcm
but for a/b testing you do so to get
started with a/b testing with
notifications you're going to need two
libraries on your client first not
surprisingly is firebase cloud messaging
which provides the foundation necessary
to make sure that firebase can talk to
your device through the FCM service as
well as enabling some of the other
analytics you can use to see your
results now we already have some helpful
videos on how to get started with FCM on
Android and iOS so go ahead and check
those out if you need help adding those
libraries to your app but the good news
here is that there's very little client
code you need to get the basic FCM
service up and running which is all you
really need for a/b testing now even if
you already have Cloud messaging
installed I would recommend doing a pod
update or putting the latest version
into your Gradle file because you are
going to need a fairly recent version of
the library installed on your app this
will include the logic needed for Cloud
messaging to tell analytics about any
a/b tests that we're running on our
notifications which leads us to the
other library you're going to need and
that's analytics now analytics is part
of the core firebase library which means
that if you've got FCM installed you
already have at least a simple
implementation of analytics running
within your app but you might not be
using it to its fullest as I noted in
our first video you can use events
recorded by analytics as goals for your
experiments now analytics automatically
records a few events like making an
in-app purchase and opening a
notification but if there are any app
specific events that you might want to
record his goal such as like completing
a level in your game using whatever new
feature you're promoting or visiting
your in-app store you should make sure
those are also recorded as analytics
events
so with these prerequisites in mind
let's take a look at how we'd set up a
be tests for a notification now this is
going to cover a lot of the same ground
that we covered in our remote config so
I'm not going to spend too much time
going over content that I've already
covered in a previous video if you do
want more information about any of this
stuff be sure to check out our other
videos in the description below or
probably linked in one of them fancy
YouTube info cards so let's say I've
added a brand new rate that houseplant
feature to my app to help us better
understand what houseplants you might
like well I'm excited to get people to
try it out and I think a notification
might be the way to do it now using
firebase analytics I'd probably want to
add something like a houseplant rated
event if I didn't have one already
because that's going to be the main goal
of my notification I want people to get
into my app try out the new feature and
rate at least one houseplant so it turns
out I already have the FCM library
installed in my app so really just
adding this one analytics event is all I
need to get going from a code standpoint
now to start an experiment we're going
to want to go to the notifications panel
of the firebase console from there we
can click on
this tab to start a new experiment or
see the status of all our existing
experiments or we can click this create
experiment button to jump right into
creating a new one now you can see here
we have a lot of the same set up fields
as creating experiments in remote config
we can give our experiment a name and a
description decide which app we want to
target and perhaps pick a subset of our
users to target with this notification
note that we can't use FCM topics here
but we can target segments like user
audiences app version which would be
important if only the recent version of
my app contained this new feature and
language which will probably be
important since I'm going to be sending
this notification in English we can also
define what percentage of our target
users we want to send this notification
to like with other experiments I might
recommend keeping this small if you're
at all worried that you might turn off
users or create some kind of backlash
with a poorly worded notification don't
worry you'll still be able to roll out
your notification later to the rest of
your users once this experiment is done
next we get to our variants now this
time there's only one variable we're
allowed to change here and that's the
notification message itself so we can
try a couple of variations here and see
what works best one note here is that up
here in the control group you have the
option to leave this field blank this
will basically not send a notification
at all and so that's if you want a be
test the difference between sending a
notification and just not sending one
which can also be useful okay now for a
goal we want to determine what is the
main point of sending this notification
now it could just be getting our users
to tap on the notification in which case
I might want to add notification open as
her main goal but is that the real goal
you're trying to get to here I mean it's
probably not most likely you want our
user to do something else within your
app whether that's spend more time with
it or perform a specific task and you
have to remember that even if a user
doesn't tap on your notification they've
seen it and that might be enough to get
them to respond to your app differently
so in my case since we want to direct
users to check out our new rate that
houseplant feature we'll want to make
our goal the houseplant rated analytics
event and as you can see down here a be
testing by default will also include
notification open among the secondary
goals of my experiment so I can still
see what version of my notification got
more people to open it without making
that the primary goal okay in this next
section here you can decide when you
want to send
notification maybe it's immediately but
maybe it's the next day or at a specific
time and just like with other
notifications you send through firebase
you can choose to send this at a time
that's local to your recipients time
zone so you can send notifications all
over the world without waking up your
users in Indonesia at like 3:00 in the
morning down here are some other
advanced features that might look
familiar if you've used firebase
notifications in the past you've got
things like adding custom data turning
on a badge count enabling sound and so
on now just note that none of these
attributes can be a be tested only the
notification message itself but they
will be included with every variant oh
and remember that if you're on iOS you
probably want to keep this priority
field set too high after that you can
click review and we are back in our
experiment review panel again this
should look pretty familiar to you if
you've watched our previous videos and
just like our remote config experiments
you can also test any of your variants
on any individual device by using this
manage test devices panel as you recall
I do this by grabbing my instance ID
token by using this line on iOS and this
line on Android
I would then copy whatever giant string
I get into here select the notification
very and I want to see click add click
save and send and the notification will
show up on my device now once you've
tried out all your test devices and like
the way the notification looks go ahead
and click start experiment and that will
run the experiment now once an
experiment is started you'll start
seeing results trickling after a few
hours but it usually takes about a week
for a/b testing to officially declare a
winner now this is about half as long as
the time it takes to declare a winner in
a remote config test since you don't
have the same kind of hey I'm gonna pay
more attention to the things that are
different by us as you do with remote
config right it's a it's a notification
it's always different so I'm not gonna
get any meaningful results from my
little test app but let's take a look at
some experiment results for an actual
experiment in bingo blast they ran an
experiment a week ago to see if they
could increase their daily user
engagement so let's see what they found
out so I think most of these results
here should look pretty similar to what
we had in our remote config and you can
always check out that video if you want
more detailed information about any of
the stuff we're seeing here so up here
you have a brief summary of your
experiment if it found a clear winner or
not or if it still needs more data and
you'll have all your detail
results here in this expandable section
now below that you can see your most
likely improvement range for all the
goals you're measuring that'll include
your primary goal as well as the
secondary goals things like the rate at
which the notification was open your
app's retention stats like that
now the bingo blast folks tried three
different notification variants in the
control group they didn't send a
notification at all they kept that
message field blank then for the other
two variants they tried two different
notifications one was a back-to-school
kind of notification and the other was a
level up and unlock new content kind of
notification and you can see here that
both of these notifications did a
significantly better job of increasing
our user engagement than not sending a
notification at all see how these
improvement goals are in a nice
confident green with this little upward
arrow that's a B testings way of telling
us it's confident that these variants
will do better than the control group
they also did a significantly better job
of causing a notification open event but
I guess that's to be expected since this
control group didn't send a notification
at all for most of these other
measurements in-app revenue causing a
level up event and so on there's a
general sense that they may have
performed better but a B testing isn't
confident enough to be sure we've got
some pretty wide ranges here and most of
these things and just like before if you
click on any one of these results you
can see more detailed information about
them here we've got the overall
improvement range the likelihood that
this variant is better than the control
group the likelihood that this variant
is the best one out of all of our
variants our raw value ranges and in
some cases you'll get some pretty
nice-looking bar graphs below if you're
more of a visual results kind of person
now you'll notice that in our experiment
a B testing is very confident that both
of these variants will do better than
our control group of not sending a
notification at all and while it seems
like our level up notification did
better than our back-to-school one it's
not quite as confident about picking
that as the absolute best variant and
that's why it's telling us up on top
here that there isn't a leader in our
experiment I mean sure it can't pick a
winner out of these two but I think we
can say with confidence that sending
either of these notifications will work
better than not sending one at all so if
I were the bingo glass folks I'd be
looking to send out that level up
notification to the rest of the group
and how would they do that
Oh glad you asked so back up here at the
results summary section you'll have more
options depending on the state of your
experiment if you're in the middle of an
experiment
you're still gathering information
you'll be able to increase your
distribution to a larger percentage of
your potential audience when you do
you'll be given another chance to
schedule these extra notifications since
you might have scheduled your first
batch to have already happened at some
specific time in the past you also have
your choice of stopping the experiment
which will primarily just stop gathering
information and you can duplicate the
experiment if you want to try running
something similar but probably the
option you're most interested in is
having the ability to push out a variant
to a hundred percent of your population
as a real notification if you're in an
experiment with a clear winner it'll
probably show up here is a big button
but even if you're not an experiment
with a clear winner it should be
available through a menu option here and
you know just because it's not in a big
button doesn't mean you shouldn't do it
I mean you probably just ended up in a
situation like bingo blast where you had
several versions that worked equally
well so you can just pick one even if we
haven't been able to statistically prove
it's the best one now when you do roll
out a message you'll get a dialog like
this one where you have a chance to pick
a variant and if your experiment has a
winning variant it'll already be
selected by default you can make any
last-minute edits scheduled time to push
this out and this will basically go out
to the rest of your audience that wasn't
in your original experiment hit Send and
there you go
it's sent now once you've rolled out an
experiment like this it'll graduate to
official notification status and it'll
show up here in the firebase
notifications panel just like any other
notification and you'll have your usual
stats here notification sent and
notifications opened although we don't
record anything else here as a
conversion event also keep in mind that
these stats aren't retroactive meaning
that if you run an experiment on 10% of
your population then roll it out to the
remaining 90 percent I would kind of
expect these absolute numbers here to be
90% of your typical results so like
don't freak out if they seem a little
lower than what you're expecting that's
normal and as a nice little bonus
feature if you're ever looking at a
notification in the console that began
its life as an experiment you'll see a
nice little button here that will take
you to the original experiment which i
think is kind of neat so there you go
everything you've always wanted to know
about a/b testing your notifications
with firebase now if you're already
using the FCM library in your app it
should be pretty easy to get one of
these up and running so give it a try
and I will see you soon in a future
episode of a B test like a pro