How I Use BigQuery to visualize streaming data Video Lecture - "How I"

Hi.
My name's Johann du Toit.
I am the co-founder
of Get Me Known,
which is a national microchip
database featuring everything
from dogs to
horses, and where we
look at patterns on that data
to then better understand
how these records are spread
out across the country.
Today, I'm going to be talking
about how we use BigQuery
to get insights from data that's
constantly being streamed in
from across the country on
records from pigs to farmers.
So Get Me Known was founded
about three years ago.
Coming from a family that's
passionate about animals,
we were registering when we had
to fill in [? free ?] forms.
So we decided that's
not how we want it,
and we built a company where
you can go to websites,
fill in all your details.
And it assigns a certain
number to your information.
So if a pig gets lost,
stolen, is found somewhere,
you've got a record
that links it to you,
and they can contact you and
get your little guy home.
So users use the service by
actually going to the websites,
signing up, filling in the data.
That data is then stored
in the back end which
we stream to BigQuery,
where we look
at data to get some insights.
So I see this being used in
almost every organization.
And this day and
age, we've got data
coming in from every
side, every angle.
And every piece of
your business is
going to be using some
metric, which you can look at.
And BigQuery is
essentially where
we're aggregating all this
and looking for patterns.
So what sales are doing good,
what records are not coming in,
where are you actually getting
data from, and just looking
for patterns to
grow your business.
So we use BigQuery
to essentially
look at all our sales data,
where we look for patterns,
where they're coming
from, and how much.
We tend to lay this
out in a nice heat map,
so you can actually,
by one glance,
see where your
business is taking off
and where you need to put
more pressure on to push sales
in a certain region.
So Get Me Known in itself is
just a website where anyone can
go to to get data about a
microchip on a dog, or a cat,
or a horse that they found and
get the contact information
to speak to the breeders.
We also partner with all
the local databases that
are also running.
So you can get information
if we might not have it.
So the idea is you'd log in.
And here's your account that
you have set up with us.
And you'd have a
list of microchips.
So these are all
registered to your name.
And they pipe their
data into a back end,
where we keep a big log of
all the changes happening
on all of these records.
So we can actually
go back and look
at when the chip was registered,
when the name was changed,
when an owner
might have changed,
how long the owner
has been the owner,
and print the certificate.
So that would be the fountain
to what we call the back end
service, which we actually use.
And that pipes into
this big record
set where we keep
all this information
and look for patterns.
So using BigQuery, we've got
a schema of all these records
where we can actually go run
popular queries on this thing.
So we might want to look at
breed, color, and popularity.
So if we're on that, you'll
see this here Rottweiler
and black and tan combined
with the most popular.
And then if you go a step back
and look at maybe flag records.
Flag records are
records that you
give an extra piece of
information about [INAUDIBLE],
so who to contact
if the pet is found.
So these will essentially be
stolen cattle, dogs, cats.
And we can actually go look
at where they were stolen,
what city, when.
So using this information
in the back end,
we can then look at hotspots.
So we know which areas of
say, Johannesburg and Pretoria
are known for getting
pets lost or stolen.
And we've got a
certain error rate.
But when that hits, we call
and notify external services
about these locations and give
them the top 10 records that
actually just tell
them something's
going on then to have a look.
And we do this nation-wide.
This data I got by looking
at our main record set
and pulling out a limited
amount from BigQuery,
and then just rendering
it using Google Maps.
So this is a smaller
data set that I took just
to show you
distribution and how we
use BigQuery to pull this out.
And we'd internally any maybe
look at only South Africa
and pull out details
from South Africa
and look at certain cities.
So say we want to look
at anything in Cape Town,
or Johannesburg, or Pretoria.
Using this and small little
utilities that we write,
we can map out our data and
see where our customer base is,
and where we need to look at
to help ease local problems.
Thanks.
And that's how we use
BigQuery and the insights
we get from it to
get pets home safe.