Coffee with a Googler: Chat with Paige Dunn-Rankin - Coffee with a Googler Video Lecture | Coffee with a Googler - IT & Software

today I'm meeting with Paige from Google
search to talk about the coolness that
is now untapped so Paige you're a
product manager for Google now on tap
right so tell me all about Google now on
tap what is it it's a new feature for
Android marshmallow and it helps bring
me related information in actions
whatever I'm doing on my phone I can
just use now on top to get information
he's gonna show me so here is a chat
with my friend Brian so you know he says
hey thanks for picking me up at the
airport gives me his flight number and
then I'm asking her about dinner so what
I would typically do in this situation
is copy and paste some of these things
or to copy with you yeah and you might
check out a couple of different apps to
see about the restaurant he's suggested
so with now on tap
it makes all that super easy you tap and
hold on the home button there and then
you get all this information and actions
related to what you were doing so here's
the restaurant that he's suggested
here's the play he's delayed I can see
it's delayed right so that's useful I
can create a calendar event for this
dinner that we're having and it actually
used natural language to predict this
calendar event which i think is pretty
cool because we had mentioned dinner
tonight and the place all in different
parts of the conversation and it
combined all of those into one in one
action and so it's trying to predict
what I want to do and also what
information I need okay cool and now is
there some beyond just the information
in a chat like this kind of also like
when you're in a particular location it
can predict the kind of things that you
do based on time it's that right so
that's more of like in Google now use
case so now on tap is actually building
on top of what Google now has already
done so Google now predicts what's going
on in the real world and what
information you need that's happening
around you now on top is about what's
happening on your phone when we were
thinking about this this was one of
these crazy ideas we had like what if
you could get now cards for what's on
your screen at any moment anything that
you're on when you were working on this
kind of thing like what it must have
been really challenging and exciting to
like think about the context of what
people doing their phones and how was
that yeah I mean it was a completely new
idea it was something that's never
really been done before and so yeah we
looked at a ton
screens you know like we were all
donating our screens as data and we had
people going through their top apps and
making sure now on top worked well and
yeah it was a really a totally new
project that started out as just a small
team doing some prototypes and then grew
into this product that we launched in
less than a year right and it's in
Android marshmallow right so cool so if
I'm an app developer and I want to take
advantage of something like this what do
I need to do so the great thing about
now into app is that developers don't
really have to do anything and so it
just works automatically on most apps
there are a couple of things that we
really recommend you do so one of those
is getting your app into app indexing
basically if you have information that
should appear in search so for example
you have Yelp with a bunch of Yelp
reviews or IMDB with a bunch of pages
about actors we actually show those apps
and search if you've gone through and
participated in app indexing and that's
where we know exactly where to go with
it now on top so if you notice we have
all those out by cons on the cards and
then when you click on those it goes to
exactly the right place ok and the way
that we do that is we actually know from
search where to point to one of the
other issues that we have is when people
do crazy things with views and so I mean
that's that's a little more rare but as
long as your app is basically working
well with accessibility API isn't such
like that it should did you work well
with now on top what kind of stuff did
you guys encounter while trying to build
something into Android one of the things
that was really exciting for me was
seeing how the API I got design so now
on top is actually built on top of an
open API so it's an API that anyone
could build an assistant that takes
advantage of this this information about
the view is underlying on the screen and
also this layer that comes up on top of
the page ok and so that was really cool
to actually design that that API and
work with the Android team and then we
also had some pretty funny challenges
you know it was actually pretty hard to
tell from context what you should be
showing I mean I guess it sounds alright
I'm sure some things went wrong along
the lines yeah yeah we had funny you
know how there's a band called yes like
we would trigger for things like that it
was it was kind of a little bit dumb at
the beginning but we started seeing
these these things and then and
fixing them what is sort of tuning to
the context around the words so what if
I was like in a messenger chat with a
friend and the word yes was in there and
I did I loved it but that I'm sure lots
of other things could go interestingly
wrong with that yeah yeah one of the
things that we have actually struggled
with is making sure to always find
something useful in every screen because
if you can imagine there's just not
entities you know perfectly spell it out
on every screen so one thing that we
introduced that I think is really cool
is basically ways to find like trending
searches on your screen we can actually
know a lot of people are searching for
right now and so you can actually
incorporate that into now on tap that's
pretty cool and I mean obviously we're
just talking about English right now so
it's also available in other languages
yeah at this point we're launched in I
think eight or more languages and it was
a challenge you know you can imagine
this all relies on natural language
processing and so you really have to
know all the nuances and the quirks of
the different languages in order to not
make those types of yes examples right
those bad over triggering is what we
call it and so over triggering I love to
remember anyway whenever I try to speak
French and insult somebody I'll just say
I'm over triggering yeah so luckily we
you know we had a team in Japan
dedicated to this we also had a lot of
Engineers in Europe as that's the where
the Cornell and have team is based and
so they all speak like seven languages
and it works just as well so you know
now on top is actually really useful
some engineers in Japan have been
telling me it's really useful for them
because it's like hard to type out all
the character copy and paste is even
more useful in Japanese and so now on
top is incredibly helpful this so back
to developers you did mention app
indexing and app indexing as part of it
but I hear you got some great resources
for developers for everything that they
need to get into now on time yeah if
you're really interested in making your
app work perfectly with now and to
happen learning about these assistant
api's I suggest taking a look at our
developer Docs on this we have a great
article that's all about the assist
api's one articles all they need yeah
it's really pretty simple nice well
thank you page and thanks for coming
today thank you so much this is great if
you've got any questions for me or if
you have any questions for page just
please
leave them in the comments below and
don't forget to subscribe to the Google
developers channel on YouTube thank you