Object detection Part 4 - Training in the cloud [Tensorflow] Video Lecture | Machine Learning with Java - AI & ML

hello explorers and welcome to another
video today I'm gonna talk a little bit
about cloud training so we have in the
previous episode talked
about how to prepare your data how to
configure your training how to run your
training locally and today we're gonna
talk about running it in the cloud and
why should you run your training in the
cloud well maybe you have a computer at
home or at work that doesn't really
isn't really made for running these kind
of tasks or perhaps you are in a
time-sensitive windows you need
something done a little bit quicker than
you can do locally because in the cloud
you can put as much compute power to the
problem as you like but on your local
machine you are restricted to the actual
power that you have locally so first off
before you can actually run something in
the cloud you need to actually package
your object detection library offline so
if we look at this command here we will
actually build the py tools so we have
those built and then we run the Python
setup sdist which will
actually create a package that we can
deploy online to the cloud and run this
object detection library online and we
also need to package the slim package
because it's also required by this
object detection library so we will
build that as well with this command we
go into the slim package we run the
setup there and run sdist on that as
well and that will package it down and
create tar gzip bundle that we can
upload to
the cloud server next up we need to set
up server in the cloud so we create
project online we initialize that
project either by downloading the GCloud SDK which we are required to have
and installing that and that will prompt
you to actually initialize it and set it
up with your online project or if you
already have it installed and want to
change project you can run gcloud init
and actually initialize it again offline
and you do that from that directory you
want to run your instance from in this
case the tensorflow models
research another thing that we actually
need is we need to setup a bucket where
we can place both our data and the
configuration that we talked about in
two episodes previously I will link that
episode as well so you can see how you
set up a configuration for object detection
and we go to the cloud here we can
create a bucket we will get that bucket
we will put our training data so the
images that we want to train on and also
the label file and then we want to add
the configuration file but we want to
make a few small changes because the
configuration file needs to have these
kind of paths so it finds it in the gcloud storage under a subfolder
somewhere so this is my bucket name and
then I have a subfolder in there where I
have both my label file and my training
records so this configuration needs to
be changed to have these kind of paths
in it and you also need to actually
point out where this configuration is
but we come to that and so when we have
set that up we can run our compute
engine job by running gcloud mlengine
so this is the machine learning engine
we want to create a job
we want to submit this new job it's a training
job and we want to give it a name so we
give it object detection with a date so
this is just to get some nice naming
you can easily find it and you can see
when you actually started this job and
I ran about five jobs which was quite
nice to see that okay here's my new job
it's started now so I know which was the
latest job and then I say which one
runtime version I want to run on here I
put 1.7 I think the last one is 1.8 so
you put it in here what can runtime
version you want you can put that in
your configuration file as well then I
say where the job dir is so this is
where it actually works and where it
will save the model or the check points
when it's training I also need some
distribution packages so I will need to
run with these this command so I
actually upload my current build object
detection library and also the current
slim package in my local build
directories those I need to also send to
the cloud and I want to say which model
name I want to run in this case it's
object detection dot train we give
it a region we want to run it in I run
it in on the US central one region and
depending on what you need for your job
you might want to run it in there EU or
you want might want to look into other
region I run it at the US central
because in the US they had later
versions of the runtime so I can
actually run tensorflow
1.8 and not an earlier version and this
is my configuration this is my local
configuration so on my local computer
where is my configuration file and then
I need to
say where my training dir is and this
is on the gstorage cloud so this is
where it will the first job dir is
where we'll save all the job related
files this is log files and so on and
training dir is actually where it will
save the checkpoint data but I put it in
the same directory I didn't want to have
a large structure of different
directories it's not that important you
will find your training data there as
well and it could be good to have them
contained so you can zip everything and
download it later and then I want this
path to my pipeline configuration path
and that's our model path so this is the
configuration that we talked about
earlier so lets look at this training
file what we can actually define in the
training file to set up our cluster and
in my case I wanted to run it on custom
tier you can run it on a basic tier
that's just a computer with no GPU then
you can run it on the basic GPUs you get
Tesla k80 GPU you can run it on a basic
TPU then you get one computer with the
tensorflow processing unit or you can
run it in custom and when you're running in
custom you get a few more parameters
that you can set up first off is the
master type this is the master that will
actually sync and keep everything else
running and actually make your little
network of workers set it up and tear it
down so this is just a standard GPU
machine you can run this in a lot of
different setups a standard GPU is this
Tesla k80 again and this will probably
change depending on when you run this
because
the hardware will be updated so you will
look into the documentation and see at
what you actually get for it if you are
interested or you just say I want to
standard GPU and that's fine you can have a
premium GPU you can go all the way up to
say that I want the cloud TPU and then
you will get the best processing power
for the money and then you say what you
want your worker type to be so
this is your worker servers and you can
set how many of the worker servers you
want to run I run nine worker servers
and then you need some parameter servers
and this was synced your servers with
the actual biases and weights so it's
good to have a few of those I run three
of those because I have nine workers so
third was my count there and this is
actually a setup that they are
suggesting on the wiki pages or the
repository of the tensorflow
research so I should just copy these
parameters you can supply the runtime
version again you can also supply Python
version I set it to 2.7 and if you put
it to 3.6 which could be a good thing
you need to actually do the fix where
you go in and change it to use list
instead of the just simple range we
talked about that in the local training
the same applies here if you run 3.6 I
put it to 2.7 at the moment but it's not
that important which version you run and
I think 2.7 is supported two more years
so it should be fine for now if you can
run it and so I run this setup of the
with the training input for
think I run it five or six times to
actually test it out and I actually ran
out of memory for my model so it crashed
number of times and during this I think
I if I just sumize how much it actually
trained it trained a fifth of the actual
training data so I had to run it at
least five times as long and that cost
me twenty dollars so I guess a standard
model with images and two hundred two
hundred thousand iterations will cost me
$100 and I as I had a pretty good
computer at home it wasn't that much
faster for me to run it on standard GPUs
online I think it was double the
speed of offline so I wasn't
really that keen on burning money on
this case and especially as it actually
threw a memory out of out of memory
exception and just crashed so I think
you have to set up this and actually try
out different configuration until you
actually find something that works for
you and perhaps you should start with a
basic GPU so you just run one computer
until you have actually tried your model
out and then run it with a little bit
more compute power if you want to try
this out yourself and I hope that you
found this video interesting I hope that
you learned something today if you have
any questions about training your models
online and so on then please send them
my way in the comment section below and
if you like this video give it a like
share it with your friends and
colleagues and if you haven't subscribed
yet please
do that and I really hope to see you in
the next video