Object detection Part 1 - Create custom dataset [Tensorflow] Video Lecture

hello explorers and welcome it's another
video today I'm gonna talk a little bit
about machine learning object detection
or object recognition the way for you to
find a thing in an image and recognize
what it is it's not just figuring out
what an image is actually off we are
trying to recognize where in this image
is this object and there are a couple of
really interesting examples and some
research going on in the tensorflow git
repository so I picked up the object
detection or object recognition
repository and I have worked with it a
bit and I actually found some ways to
help me to both create data run the
different training schedules then freeze
the data and also do inference so I
think there is a lot of things to talk
about here so this is part one of this
series first off I would like to talk a
little bit about a way to do your
annotations because when you need to
actually have a lot of training data you
need to label the training data and
doing all these annotations can be quite
some work I'm injecting this into the
normal video because I think it's I
found a good tool that you can use to do
this and as I did it before with
creating a JSON and actually creating
start and stop values or min and max
values in the JSON from an image was
quite cumbersome and as I came
across this tooling here I thought it
could be really interesting to show you
how to use this tooling so you can label
your images in an easy way
so this label image is done by tzuta lin
perhaps I'm not good at
pronouncing Asian names so hopefully not
butchering it too much and he made this
library label image and he's quite
interesting because he's worked at
Amazon ASUS Nvidia and HTC on these
kinds of things computer vision and
framework and kernel work in the Android
case so it's really interesting here
that he has so much knowledge and
actually built this label framework so
it's something that you should look into
and use and if we look at these images
here I have the program running so I can
show it later on but we have a simple
way to just annotate an image and I'm
using Windows at the moment but you can
do within Mac OS and Linux of different
kinds you need the Python Qt library and
you need lxml installed
and if you have these installed on
Windows you have to do one more thing
you need to run your pyrcc so that's
a compiler I have number 5 and you might
have number 4 depending on which
platform you have in in your binary or
script folder you will find this tooling
and that's to compile your Python file
into a qrc so this is the resource file
that you need to compile before you can
run this and then you just run it as
usual with Python you can tell it what
image path to open if you like I didn't
do that because in the program you can
actually open a path so here I have
opened up this program as I see in the
middle here
you have some images and could go to the
next image and the previous image if I
like and you can easily annotate so
let's just annotate one image so you can
see what it is in my example here I'm
trying to find Waldo so I'm trying to
chjeat at the game finding Waldo
or can you find Waldo or where is Waldo
and you need to look at an image and
figure out where this character that is
very characteristic is in the image and
in this case I know where he is
so we zoom in here and we go down can
you see him
zoom in some more go down and there is
this is the characteristic Waldo figure
so I take this create rect box and I
draw it over him like this and I say
that this is Waldo and I press ok so now
I've created a rectangle and I say that
and then I will save it as an XML and in
this case I had already labeled him but
I will replace that so now you can go
between images and look at the different
images where you had labeled him and
where he is and I have labeled 25 images
so far and try to train that and I think
that's a lot of images to label but it's
still too few labeled so I think I will
need to label even more images and if
you want to try this process out you can
actually help me out by finding Waldo
images or where is Waldo images and find
Waldo and label it my wife was actually
I showed her these kind of images and
said that I had to label them and she
actually took up the task to label them
so many of these images was labeled by
her because she thought it was such a
fun thing to go through images
find Waldo it was a fun challenge for
her so maybe you find it fun as well
and if you are labeling these images and
you finding Waldo in places and
saving these XML files send them my way
with the image and I will put them into
my training set and we could do this
collaboratory if you like so this is the
tooling I wanted to show you now back to
my code I will talk about the way to
actually create TFRecords with labeled
information in JSON but I have also
implemented a version in that will take
the labeled examples in XML format and
this is the VOC format that they use
in in the larger example from Google
or I think it's University that have
taken out this image set so it's a known
format it's this PascalVOC format you
can also save it in JOLO format if you
like that's two different formats so you
can read and save data into so it's good
to use these formats but I have code for
that and we will look at more code right
now I hope you find it interesting and
so let's start by creating some training
data so we can actually have something
to train on and I want to do this in
Java so there is an API for this and
there is some code but I ran around on
the internet and actually found a class
that could write TF records for me and
also some definition of how a TF record
is structured for this specific training
case so let's go into this TF record
writing class so this is pretty much
mostly copied code from other places and
some things that I've written a little
bit myself and first off we look at the
actual format so a TFRecord
a bunch of data that comes after each
other and when they write down these
data they write the length and then they
write a cyclic redundancy check for that
length so they actually check that the
length is correctly written then rewrite
some data and they have some error check
on that data as well so that's the
format of the TF record and you can save
any amount of byte data in this kind of
a record set usually use save some
examples that you want to train on and
so let's continue here the TF record
writer we start by initializing that and
in the cyclic redundancy check we will
use a mask delta and this mask delta is
something that we just will add to the
check so this is a constant that this
format is using so we don't go into that
more in detail it just good to know that
this is the actual number they chose to
use for this format and then we have
this little right here we see that is
following the same kind of structure we
take the length and this length is
in int 64 in bytes so they read this and
it's lower endian and then they write a
length and then they write the int 32
lower endian masked cyclic redundancy
check for that length and then they
write the actual records the actual data
from offset to length and then they
write this cyclic redundancy check for
that as well and so this is just an
example of one of these two functions so
we have here a buffer of 8 bytes because
it's 64 bits in length and then you have
a byte buffer where you wrap that byte
buffer and we set the little
endian order and then put a long into its
bytebuffer and return the bytes and this
is the cyclic redundancy check
function so we use PureJavaCrc32C and
that's in Apache Commons so you we have
an implementation of this function there
and these kind of cryptographic
functions usually use an update function
to add data into the message and then
when it's done you do some kind of
digest to get the value out in this case
they chose to actually call that
function get value so that's the last
thing you do to actually get the CRC
value and then we do some bit switching
of the value and adding the Delta to get
this CRC back so this code here is copy
pasted and it's just to get the right
numbers written in the file so not
something that we really need to know
that much about but it's good to have
seen these kind of codes just so you can
when you're reading code and when you
are trying to figure out what does this
actually do you can see some patterns
and at least understand why a certain
code is written in with a certain way
and when you copy/paste some code you
can actually be sure that okay I don't
understand exactly what this does but I
understand the function of it in its
entirety so you know why I need this
function so let's go into the writer
application the writer application
starts by actually creating in the file
output stream so we can put our TF
records somewhere and it's a data output
stream and then we create our TF record
writer
and here we have a little bit of code I
don't want to go in too much - because I
actually did some some preparing of the
data by hand I went into the data and
the images and actually created my
bounding boxes for each hit by myself
and I saved those values into JSON
files this is just reading the JSON file
something to be a note of here is that I
get the ID and actually check if the
idea is lower than 20 and then it's a
training record otherwise it's if it's
above 20 it's an eval record and in my
case I had 25 examples that's very few
and yeah let's see if I can actually get
that trained later on and actually get a
good result from it and so let's look at
this run file so this is the function we
saw in this example here so this start
off by reading the actual ID getting a
JPEG image there we put that image into
a buffered image we use image IO to read
images it's a very good part of java
at that you have a very simple interface
to read pretty much any image into a
buffered image you can play around with
them and we take the width and height of
that image now we create some labels and
some labeled text and these are just
lists of things and we also have min max
of the x and y for the bounding boxes and
these are also lists so if we have one
hit in the image we have one label one
label text and one min max of each of
the coordinates but if we have multiple
hits who have found multiple things in
the image that we want to
actually highlight these lists holds all
the values of the things we found in
the image and next up we get the min/max
from our JSON so this is the actual
values of the minimal x value and the
maximum x value and then the minimal Y
value and the maximum Y value so we
actually get each axis is there then we
add some labels here and I as I use just
one label and it's a 1 and Waldo and then
I have actually taken these values and
normalized them because in the actual
training then you normalize the values
so the image can have any size but the
actual placement of the bounding box is
somewhere between 0 and 1 so the
function actually knows where to look at
for it in the image and then we have
this large set of features that we want
to create here so we build a set of
features and we put in the width of
height of the image of course we put in
the filename of the image if we have a
filename this is not strictly required
it's just if you actually have a
filename they want it you can add that
into your record file that makes things
easier to find and then we get the
actual image feature and add that as an
encoded and then we tell it which format
this was encoded in in this case I use
JPEG images then we are the actual box
data so this is the bounding box data
min max X Y data and last but not least
we are the lists of text and label
features into this and we build this
features list we take these features
that we have created
we add them to an examples each image
will become one example with many
bounding boxes and then we build that
and the built version of that we do an
byte array of that and write that out so
example feature and feature list is
something that is actually inside of
tensorflow libraries it's just things
that we reuse and things that we can
include in our java code so let's see
how we can create an INT feature as you
remember we had the long list of things
that we added and some of them was int
features some things was length and
some things were lists and so on and int
features we used for width and height
for instance and to create these we just
created a list int list builder we add
one value to that int list and then we
create a feature and return that and
then we have an int list and the
interesting thing here is when you add a
list of int or long's in this case so we
used long for the labels
so when we want to create this list we
need to add all the values in the int 64
list and build it and we create a
feature so that's the difference there
just at helper function if we want to
add strings we need to do a byte string
of these strings and we need to do it
from UTF-8 because UTF-8 value is
all Java strings are UTF-8 and we also
need to actually have UTF-8 values
inside of our TFRecord because that's
what's used inside of the those files so
we create a bunch of bytes here a byte
string we create a byte list we add our
byte string
that and we build a feature and that's
the same we did before and then we have
this string list feature and that's for
the label text that we saw because we
sent in a bunch of the a list of all
these values again we create a byte
lists builder and then we loop through
and take each of these strings and add
them separately to the lists and to the
byte builder and create a feature and
return that feature of byte list
so all these functions are helper
functions to take our pure Java values
and creating something that we can put
into a tensorflow
feature list and last but not least we
have the little bit more complicated one
where we want to extract image feature
and first thing that we need to look at
is that we actually need an RGB image
and we are not actually required to give
it an RGB image it could be any kind of
image with alpha values or the color
values can actually be switched around
and so on so we want to actually ensure
that we have an RGB image so that's
red green blue and so we create the new
one with a width and height and then we
create an graphics object of that and
draw the original image to that
buffered image and that ensures us we
have the correct image that gets out of
this we get a buffered image that has
those values we take this buffer the
image create a byte array output stream
and write it out to as a JPEG format to
that byte array output stream and flush
that out so we actually know that all
the bytes have been written before we go
to
part the next part is to actually create
byte string of this byte array output
stream and then we close it then we
create a byte list and add the value of
this byte string to that and then we
create a new feature of this and return
that so the last part was pretty much
similar to handling string that we did
earlier so this is all that I wanted to
show you today it's a lot to take in I
know and there's a lot of interesting
things here but the code is not that
complicated if you look at it and for a
while and go through it and but it took
a while for me to actually figure out
how to write the data correctly so the I
get a good file that I then can run
training on and so I hope that
you found this little tutorial
interesting if you have any questions
comments something that I did wrong
something that I did right anything
like that if you like this concept if
you want to see more of these then tell
me as well and I think I will do a
series of this as I said a few different
parts and if you don't want to miss
those then please subscribe to the
channel if you like this video give it a
like share it with your friends and
colleagues I really hope that you
learned something today and I really
hope to see you in the next video