A quick look into the tensorflow lite demo Video Lecture | Machine Learning with Java - AI & ML

hello explorers and welcome to another
video today I'm gonna talk about
tensorflow lite and how to use
tensorflow in the constrained
environment and in this case I'm talking
about Android and how to run your models
on a phone and I'm not talking about
training models or anything like that we
need to have a pre trained model that we
have run on some good GPUs or some TPU
CPUs and so on so we have the data ready
or already but we want to have a use
case on our phone where we will predict
something or we will try to understand
the world around us using tensorflow
models so there's a very nifty and
interesting demo that I have in the
tensorflow framework that you just can
download you can do a git clone or
download the zip file from github and
install it wherever you have your
workspace and when you've done that you
need to install the latest Android
studio and if you do any Android
development you should already have this
installed and then inside of your
Android studio you just import this
project and it will be inside the
tensorflow github repository and the
contrib Lite Java demo and you import
that project into Android studio
and after that you will download the
tensorflow Lite mobile net model here
and after you have downloaded that you
will put that into your tensorflow
contrib Lite Java demo and in the
application that you have imported you
go under app source main and assets and
there you put this TF light model so
this is the trained model
that you download and they didn't put it
in the git repository because it's a
little bit larger and checking in this
resource would make no sense and then
you just build it in Android studio they
say well that's not completely true when
I imported in it under studio had to do
some small changes to my Gradle file so
I open the Gradle file and here it says
26.0.1 and I didn't have that build
version I had 27.0.3
and whatever build version you have you
just change the build version for your
build tools so the current version is in
in your under studio either that or you
install this 26.0.1 not
something that I wanted to do what I
didn't want to have different build
tools it's I'm doing Android development
at the moment and I'm quite satisfied
with having one of those and what you
have imported this you will get a
project and there's a few views and it
will add a bunch of libraries and so on
that does the actual tensorflow
recognition so if we look in this
Gradle file again we have all the usual
suspects to build a GUI and then you
have this tensorflow org tensorflow
lite and just the latest version so it
will compile that into your project as
well in order to run this and if we look
at the code we have first one file here
called auto fit texture view and it's
just something that will change the
aspect ratio so it will fit your phone
the best so you get
the correct height and width and the correct
aspect ratio for the picture that
you want to gather and do your
predictions on so this is not that
interesting then we have this camera
activity it's also very small activity
frame here just adding the basic frame
for the camera so nothing interesting
here and the basic frame here we have
the actual logic and this is a lot of
code so I will not dive into every piece
of this code because that will be quite
boring it has some texture surface
texture listener
it sets up here and just overwrite some
functions it has some state callback for
the camera device where it will actually
keep the state of the camera in a good
order so that's just some camera
fiddling and if you have worked with
cameras and streaming camera data before
this is nothing new for you and then we
have some capture progression and
completion and these are just override
them and not used at all and so this is
just something that you need in order to
capture data from the camera and then we
have this choose optimal size and it will
figure out the optimal size of your view
port for your recognition and yeah
there's a lot of functions here set up
camera outputs yeah just look into the
code if you want some more information
about all all the different things that
are based for the camera but a bit
further down here need to find it again
we have
little thing for the actual prediction
so down here we have the classifies
frame for a preview stream so this will
actually take some bitmap from the
texture view so the textview is where
you present the data from the camera so
the camera will show something and put
it inside of and textview and you get
the bitmap of that texture view and you
send that bitmap to the classify frame
and then you recycle the bitmap and show
a toast with the text that you got from
the classification and if we look at the
classify frame it's inside of a runnable
that is run periodically and it's this
background handler that is set up to
handle when to run this so this is a
looper that will run at a handle thread
name see here if I can find the actual
periodically how often it will run there
was a constant for that somewhere here
it's not that important will run quite
often and as there is new data to handle
in the in the camera it will classify a
new frame and it will send it to the
image classifier so if we look at the
image classifier the last class that we
will go through and also the last class
in this package it's a very simple class
I would say it has this tensorflow lite
this interpreter from the tensorflow
lite library and this
interpreter will load the model file so
it has that and then we have another
function to load the actual label
list
so we have a list of labels and that's
just a bunch of strings in an ordered
manner that we know that if we got
something for the number 4 for instance
it's this label that actually got the
prediction right and how much it was
predicted for that label and so
when we have loaded all this that
classifier is ready so we will classify
a frame and in this we will convert the
bitmap into byte buffer because the byte
buffer is something that we can send to
tensorflow
and then it will just take the bitmap
read all the image data into a byte
buffer that they call image data and
then it will send the image data and the
label probArray so that's the
probability of the actual label for the
tensorflow lite responsible this will
be filled with the response from the
tensorflow lite and then it prints how
much time it actually spent to run
inference and then it will apply some
filter for the results so it actually
takes the probabilities and does some
mangling with the data so it doesn't the
result isn't that sharp and then we will
do some print K top levels so this is a
number of labels that it predicted and
how good of a prediction it actually did
for those so if we look at this printer
K labels it will go through the labels
and then so add a sorted list of these
labels with the probability of the
actual result and then it will fetch a
few of those and put that into a text
show string so this is pretty
everything you need to do so if you have
an already good implemented model or
already a model that have predicted a
bunch of stuff you only need to read in
the data and add that data to your tensorflow
lite model or the tensorflow
lite library run your tensorflow lite
over a byte buffer from your image and
you will get some probabilities back and
then you have to interpret your result
that these probabilities are actually
for these labels and the probability is
somewhere between 1 and 0 so if it's a 1
it's totally sure that it has found this
object and if it's a 0 it's not sure at
all if it has found anything so let's
say that it sees a ball and it says that
it's point 9 then it's pretty sure that
this is a ball and I will show you a
little demo now how did this app looks
when you actually have compiled and run
it on your phone well last I just want
to show this little app in action so
when you have compiled the application
you will get something like this and it
will look at your surrounding in the
background and you might not see it down
here but you can actually see some
numbers what it thinks things are and
also how likely that it have found the
right thing so if it's 1.00 it has found
a perfect match but if it's less than
that it's not that confident of the
match
so this is the application thank you for
looking at this short video introducing
you into the tensorflow lite library
there's a lot more to say on this
subject but I hope that you learn
something today and that you got
inspired to look further into this code
and figure out how you can apply this in
your workload and your workday and if
you like this video give it a like share
it with your friends and colleagues and
if you haven't subscribed yet please do
that and if you have any comments or if
you have any questions please ask them
down below and I really hope to see you
in the next video
you