Object detection Part 5 - Evaluation and Tensorboard [Tensorflow] Video

hello explorers and welcome to another
video today we're gonna talk about
evaluation so we're gonna evaluate what
we have done so far how good are we how
close to the truth are we what is the
actual status of our trained model and
so to evaluate our model we actually
need to install some more stuff when we
run the model and so on
you just need tensorflow Python and a
few pip packages but when you evaluate
the model for some reason you actually
need a library that is a graphical
library if I can understand it correctly
active TCL in Windows and you need TCL
support or tk support in your
Python so in Windows you need to install
active TCL and this could be found at
the download site of the active state so
you download this version here and it
they ask for quote but you can download
this free version and when you have
install this you need to install Python
again so you download Python and install
that and if you have Python installed
you just run the Installer and then say
that you want to change some features
and the thing that you need to check is
that off to install TCL in Windows that
you have this checkbox checked here
tcl/tk and idle so that you
install this tkinter because tkinter
will be used by the evaluation script so
that's important for you to install so
you don't get any failures during the
run of this script and I teased the
other day how to run the evaluation
process is quite easy just run in the
model research directory object detection
evaluation and it's after you have set
up your system there so you run that
script and then you log to standard
error you have the pipeline
configuration that will look that many
times you run a checkpoint directory and
evaluation directory so that's where to
put the result and it runs and it
doesn't give you that much of an output
it actually just says that it's starting
to process stuff and then it ends it
doesn't give you any pop-ups any extra
information it just creates a directory
of it with the evaluation result so in
order to look at this you need to use
tensorboard and tensorboard can be run
like this you have tensorboard the log
directory and then the training out the
directory or the evaluation directory so
if we look at some of these that I set
up here so here we have the evaluation
directory so I've run tensorboard Exe
this could be found in your Python
scripts directory under windows log
directory and then I run this under port
6006 that's the standard port and I also
have one running that runs under
6005 and there I run my training
directory so we can look at both of
these so let's go to the browser and
take a look at what you can see in
these overviews so this is the training
directory and here we see that we get
some scalars and these I can understand
you see how the learning rate actually
was performed during the run here so we
used the same learning rate over the
whole set and if you have a training or
a learning rate that is actually scaling
and it's
moving down during the running you can
see this curve here but it's pretty flat
for me and then we have this
classification loss so this is how good
of a classification we actually got and
we see that it actually stays pretty
stable all over here it doesn't really
go that low it starts really high and
then it keeps around a value of 0.7 and
it's down to 0.1 in some instances but
it goes up and down and then we have
localization loss this is how close to
the actually where the box should be
located in the image and this is all
over the place and we have the same on
the RPN loss so this is the suggestion
for boxes and localizations these are
related and then we have objectness
loss is the actual what kind of objects
you found and how far from the result it
is and then we have total loss and total
clones so these values I can understand
a little bit about and I can also see
that our training wasn't that successful
when you look at these values so that
could be improvement either in the model
in the setup and in other things to
actually improve this and then you have
the graph so here you can look at the
actual graph of the model that we ran
and this is quite a complicated model
and if you like to be really
professional or really go into the field
more you can go in and change this model
and figure out how a model should be
efficiently built in order to actually
make it perform better but one thing
that you can do if you don't want to
dive that
deep is that you can actually see here
you have some colors on this side that
there we go
so device here if you turn that on you
can actually see different colors so you
have some bluish color here that's run
on the CPU so you know that this
specific box runs on the CPU and then
you have some stuff over here that's
green and these things are run on the
GPU so you can actually see how much of
the workload is run on the CPU and the
GPU can actually figure out how to set
up your environment for the best
performance and you can also you also
have a little bit of an interesting
check here you can check if you are TPU
compliant so capability with TPU
and here you see that this model is
fully compatible with the TPU so you can
run this on the tensorflow processing
unit so if you have your own model that
you have built and you open it up in
tensorboard you can actually look in
the graph and see if you can run this
more efficiently on the Google cloud
this is good to know that you can
actually both look what kind of things
in my graph have a setup that will run
on a GPU or a CPU and can I run it in
the cloud so these kind of questions you
can find in the tensorboard
utility and then we come to
distributions and this is also a little
more low-level if you want to debug your
specific model it goes into a lot of the
different stages in this model and they
are quite advanced and I can say that
I'm not really sure what all these
values actually say so I will not talk
about these the same goes for histograms
here you can also debug your model
and see how well it was
training in the different stages but as
you see here it's 70 pages of values
to go through so you need to have
some know-how of the actual model in
order to debug it and the projectors
slide here I was actually thinking that
I would see something that would tell me
what kind of a distribution I have and
or how well it actually trained and how
far from the actual classification that
it got but it only shows me values that
is box encoding predictive weights and
momentum and the same goes for class
predictor weights and momentum I'm not
really sure if these values actually
tells me some information that is useful
if you are more advanced you can perhaps
use this to actually see how the weights
are distributed and if that correlates
to the actual result that you expect but
I'm not that well-versed in this
specific models I can't really tell if
these weights are reasonable or not and
because this is a pretty pre setup model
and I'm not looked into it specifically
and then we look at the evaluation
directory and here we have a tensorboard it shows that kind of data here we
have some scalars and these tell me
nothing pretty much because they are
just one value so this is on the frozen
model so they will not show you that
much information so it's not that
interesting you can see that you have a
classification loss that is under or
it's like actually 0.3669 so that
where was what it actually ended up but
without seeing the actual progression up
to this point I think these specific
points in time is not that interesting
and in this you also have the graph so
you have the same kind of look at this
graph you can see that this graph is a
lot simpler because this is a frozen
state and is also a frozen model so if
you freeze your model and run it you can
see that it's a lot less and it's also
in this case we have a lot of things
that are not compatible with the TPU so
their specific evaluation model is not
compatible with TPU that's good to know
but you will not run on this specific
model it's not the model that you
actually trained on and it could be
interesting to see when you freeze a
model later on if that model can be run
on the on the TPU or if you but then
again why would you like to run a model
that is trained on a TPU to train it you
wouldn't run inference on the TPU right
and so and I think that's okay that it's
not compatible with TPU and but an a
thing that is really interesting here is
that you can actually look at the images
and look at the specific prediction and
this is very interesting to see how good
it actually went here we have the first
image and it doesn't know at all what
that is I think it's a TV monitor or
perhaps oven or something showing the
Sun perhaps and then it's things that
this railroad car or this train is a
bus not really sure that that was a
great one and then we have this little
harbor crane I think
that person not that good either
and here it doesn't find anything in
this image and here we have a TV monitor
that's pretty good and it's fifty-five
percent sure that it's a TV monitor so
that's at least right and here we have
an aeroplane well done you found an
aeroplane by 65 percent so that's a
correct one so I wonder if these also
its actual Oh a person right on found a
good one there too and then we have a
person and a car so I wonder if it's
actually person in the car there I don't
think so so this was not a good match
for person but a pretty good match for a
car but I think it should actually take
the full image for a car that's what's
not a good one either here it's very
confused it could either be a car or a
bus or both well it's a vehicle so it's
at least right there but it should be a
bus everything and last but not least
here's a lot of persons and it's
actually right it's found too many of
them but the bounding boxes some of the
bounding boxes are pretty good at
finding persons but as you see here the
training of this data set was not good
so that that's an interesting thing that
you can actually go in look at specific
image set and see how well did your data
actually train so this is a good set a
way to evaluate did I do a good training
or do I need to train this model again
um so this was pretty much what I wanted
to talk about today and I hope that you
found this video interesting
to check out this tool tensorboard and
use it and that you have a little bit
more insight in what you can do with
tensorboard and how you evaluate your
models I hope that you learned something
today if you like this video please give
it a like share it with your friends and
colleagues if you haven't subscribed yet
and want to follow along in this little
tutorial or this little series then
please do that and I really hope to see
you in the next video