TFLearn - Deep Learning with Neural Networks and TensorFlow p. 14 Video

what is going on everybody and welcome
to part 14 of our deep learning with
Python tensor flow and a neural networks
tutorial series in this tutorial what
we're going to talking about is the
high-level abstraction layer in library
on top of tensor flow that is TF learn
so you actually have quite a few options
when it comes to abstraction layers
there's Kerris which is for tensor flow
and theano
but then there's also TF learn which is
what we're covering now TF slim and SK
flow okay so we've got really four apps
for pretty popular abstraction layers on
top of tensor flow they're all
contenders which is pretty interesting
because tensor flow is actually still in
beta so the fact that we already have
this many is kind of strange and why
people keep coming out with new ones is
beyond me especially when the other ones
are relatively ok so from my looking
through them I decided to go with TF
learn Carris is Kerris was they learn
umber two contender but TF learn makes a
little more sense if you're sure going
to be working with tensor flow you might
as well be using TF learn because it's a
it's right on top and so it's a lot
easier to use some of the more
interesting functionalities of tensor
flow with TF learn
I'm sure Karis can it can be done it's
just not as simple I don't think at
least for me so okay why don't you
actually want to have abstraction why
would you even want to do this so
there's like two major reasons the one
reason is just for simplicity sake so
this code is actually fairly short you
know for making a neural network
especially like a convolutional neural
network this is actually not very much
code okay there's there's not too much
here but the code isn't super simple
it's fairly verbose and and it could be
a lot simpler because really all we're
all we need to do convolution layer
actually we do have an input layer
convolution one convolution two and then
the fully connected layer output layer
and we're done right that's that's all
we really need so we could definitely
simplify this so so that's one thing is
just simplifying the code and kind of in
line with that but for a whole other
reason is you're a lot less error-prone
if you use a higher level framework
that's going to just not have the
availability there for you to make that
and so tensorflow you know I don't claim
to be the best programmer in the world
but it's easy at least for me to make
mistakes I think it's fairly easy for
anybody to make mistakes now it couldn't
come at a better time because I actually
made a pretty important or pretty large
mistake in the previous tutorial but it
went unnoticed because it didn't impact
it had a huge impact relatively but in
terms of percentage and all that I just
chalked it up to oh we don't have enough
data okay but that wasn't actually truly
the answer so let's talk about that
really quickly and then we will get back
into TF learn so what I forgot to do to
two things one thing is the biases in
this case it didn't actually matter
it I guess this data set is good enough
it doesn't have biases and it has no
zero input so biases really don't matter
that much in this example but we should
have had them so normally it would be
you know x times weights plus biases
right it's always that so and I keep
trying to like screw you know bang that
into y'all's heads but for some reason I
guess I just don't have it yet but
anyways always input times the weights
push the biases don't forget that stuff
come on guys
so anyways actually I was one I forgot
it and it was brought up pretty quickly
by other people so anyway maybe just me
but it's an easy mistake to make same
thing with the Matt mole that we that I
made initially really easy mistake to
make as you're just kind of going
through it so like if you're just
quickly prototyping something or testing
something or even just you just want to
use a really simple convolution neural
network or recurrent neural network it's
easy as you're going through to just
whoops make a pretty simple mistake so I
was mistake number one mistake number
two was we didn't have an activation
function and we're not running the
rectified linear operation here
that is what accounted for basically
about two percent of accuracy because
really we just have this single
activation function on the fully
connected layer again whoops so anyway
to properly fix that you would do TF nut
n dot ray Lu and just in case this
entire thing so I'm just going to just
copy this
copy paste in case that is how the
convolutional neural network should look
until someone corrects me so so anyway
so that's what it should be and let's go
ahead and just run this really quickly
I'm actually going to run it the
background as we work on TF learn but I
just want to show you the actual output
the real output that was supposed to be
so let's just do Python 3 CNN example I
let's make sure it runs and then as soon
as we get one loss I will move this item
will get into TF learn
cool so uh moving this over well I'll
show that before we run the TF learn
model so let's make this full size let's
make it a little bigger and then let's
do file save as and we're going to save
this as CNN ta plan cool and we're gonna
delete it all crazy I know okay so with
TF learn basically you're going to use
only TF learn you don't need to import
tensorflow in fact if you did on port
tensorflow you'd have to it would be
treated as a completely separate package
and it's going to do the whole loading
the GPU libraries and all that kind of
stuff if you're on the GPU version
otherwise I guess you wouldn't really
see that but it's redundant to import
them both in most cases anyways import
TF learn then we're going to say from TF
o learn layers comm we're going to
import car under square to D and max
pool to D so these are two functions
that you should already know what they
do we've already written them they are I
haven't looked to confirm this but I
would wager they are identical to the
ones in that convolutional neural
network I think it was the amnesty for
experts data tutorial on tensor flow I'm
going to wager those in the same but
anyway they're going to do the same
thing also while we're talking about
this and making these imports notice
that we're importing these functions
we're not writing these functions these
functions I hate to say they don't exist
in tensor flux they actually might but I
don't think they do right but in a
higher level abstraction layer or
library they will exist and chances and
if you're going to use an abstraction
library the goal is to write a hundred
percent of your code in that abstraction
layer and not try to write custom stuff
around the abstraction when you need to
chances are if you if you find yourself
in a situation we like room I really
need to do this and you're thinking
you're going to do something fancy to
get around it chances are it's already
been done and it's there you just don't
know about it so I highly highly highly
recommend you go to TF learn org and go
through the things here you don't
necessarily have to read everything but
what you should do is go through the
examples we'll pull up
at least one example later on because
it'll be important hopefully I won't
forget but anyway I might but anyway
hope I will pull up example later but
look at the examples you can go to the
models you've got a deep neural network
and generative neural networks those are
the two major model types here we
haven't really talked too much about
generative models that's those are but
interest me the most but they have like
the lowest use case right now they're
the most like kind of research intently
they're in more research as far as
application is concerned in the spectrum
of research and application anyway
layers
okay you got core layers convolutional
layers recurrent layers normalization
layers and bedding layers merge layers
and estimator layers got lots of layers
here some of these we you probably don't
even know what they are
feel free to look at them and learn more
and if you don't if you want to learn
more you can always google these things
but we're going to be using convolution
layers okay
then we've got operations that are
built-in you've got some data management
prior really useful one to look at is
data utils and pre-processing and then
maybe the data flow and augmentation but
those first two probably have a lot of
really useful things in them that
otherwise you'd be writing yourself so
think about it anyway I'm gonna set that
aside for now and we're going to keep
importing stuff from TF learn but just
know if you're ever you know if you want
to learn more about these things you can
always go to TF learn org or you can
just straight up google TF learned liars
comic-con 2d and you'll get to the
documentation and you can read more
about what it is you're actually
employing and you really need to
otherwise you're not going to know what
the heck you're doing so next from TF
learn actually mostly you just want to
learn what the parameters are right like
the function names are actually pretty
good right you know what the function
names are but it's pretty useful to know
what your options are with that function
and what your parameters are that's just
essential okay so anyways because a lot
of the parameters are required anyway so
from TF learn layers dot core we're
going to import input data so this will
be your input layer basically drop out
and then the fully connected oops
fully
connected layer so input layer here drop
out just to have drop out fully
connected layer functionality their
convolution leg or convolution here max
pooling there so I've got the
convolution layers fully connected
layers input and in theory output with
fully connected so those are just all of
our kind of definitions almost right out
of the gate then from TF learned layers
estimator we're going to import
regression since we're going to use
regression at the very end and then
finally import TF learn datasets dot and
mist as M that's just TF 1 into M mist
now we're going to have X comma Y con
test X comma test Y is going to equal M
miss dot load actually which load data
input yet load data I'm praying just
wanted to turn out there
one hot equals true okay so here we're
just loading in our data really nothing
crazy now we're going to have a reshape
for X x equals x dot reshape and the
reshape we want is negative 128 by 2801
and then we actually want to do the same
thing to test X so I'm just copying and
pasting test X let's test X and test X
okay now ComNet is going to equal first
we've got the input layer so input data
and the shape of that input data is none
none by 28 by 28 by 1 and then name here
for now this isn't totally essential but
later on if you want to use fancy tools
that test flow offers you like tensor
board I'm pretty sure it's mostly why
we're using name here also if you never
mind we'll just have it there
now we're going to do is begin that's
our input layer now we have two of the
kind of convolution and pooling layer so
Kahn's net equals pond actually this
underscore 2d this takes a bunch of
parameters we're only going to use a few
of them first of all we're going to
input comp net it's going to be we use
32 for the size so we use 32 windows -
and then the activation function just
activation equals we use rectified
linear so Rayleigh then what we're going
to do is continent V equals now we're
going to do the pooling max pool to D
again we pass in here confident and then
the window is to okay so we're going to
take that copy paste and we just do the
same thing again basically only this one
is 64 since that's what we used before
and hopefully everyone can run a network
of this size so or most people can so
those are 2 convolution layers what's
next
fully connected layer so and then
followed by one dropout okay confident
equals and this is fully connected what
are we going to pass content that's the
input is whatever's coming at this point
we use ten hundred ten hundred 1,024 for
the input there and then the activation
again is really rectified linear and
finally and basically just for the
record that there is a default I think
the default is just linear
oh look let's see um TF learn dot layers
comp dot
come to D let's look it up guys so bring
it over and we're looking for calm 2d
where's activation yeah so the default
is linear so I'm not very good
all right but but you couldn't forget an
activation function because it's just it
has a default okay it would be nice if
the default was better than the linear
but okay moving right along rectified
linear fully connected now what do we
have to do we have to drop out so we're
going to complement equals drop out and
what do we apply and drop out to
continent and then what's the dropout
rate well we used point eight so we'll
stick with that next what we're going to
do is we have the output layer so the
output layer itself is also a fully
connected layer so continent equals
fully connected input continent ten this
time and the activation function is not
rectified linear it is softmax
and let us do confident in equals and
then finally we run that regression and
in this sense it's almost kind of a
little more verbose than then tensorflow
was if you ask me but anyway anyway
we're out of regression on the continent
and the optimizer optimizer is Adam
learning rate I honestly don't I think
we use the default and I think the
default is zero zeros will go with zero
point zero one that should be fine the
loss was cross-entropy in this case
that's referred to as CAD you run into
my face categorical category
Ross and TRO P hopefully we didn't typo
that next the name here we're going to
call this a these are the targets so
you're going to basically in this this
is your you're calculating the loss
basically so from the input after all
the weights and all that how close are
we to our targets so that's the
regression layer now we're pretty much
almost done so model equals TF learn dot
and of the two models available to us
we've got deep neural network and
generative neural network which one do
you think it is it's probably DN n so so
we're going to do DN n so TF learn dot
capital DN n continent kinda cool now
once you've done this and defined all of
this you can do all the quick commands
for a model now we don't have anything
trained at this point so we're going to
do model dot bit and we're going to fit
the following in normal cases you can
just pass x and y but you can also give
them labels and we're going to go ahead
and do that so we're gonna say input
that's your ex your targets
that's your y the number of epochs even
though it's an epoch is 10
pretty sure epoch is not one of those
weird words that doesn't have like s as
the plural but okay
someone let me know if I'm wrong
validation set is equal to and basically
the same thing I'm going to take this
copy and list pasted rather than just s
at X it's test X and then this is test Y
test why I'm getting lost let's hit
enter here this is for the validation
cool some validation set and then we
have snapshot snap
i underscore step we're going to say 500
this is just how often do we want it to
report to us like what the final value
for this is you'll see exactly what I'm
talking about in a moment
show metric we'll say equals true and
finally run ID we'll call this M NIST
great one final look over because that
was running off the screen and I think
we're good looks good okay so we fit now
just to throw all this all in one basket
this is the exact tutorial from before
so it's at 30 lines right now 30 lines
compared to out of my way and an example
I think it was like 80 lines possibly
now 73 lines okay so um a little less
than half but even more importantly like
I don't have any of this nonsense we
don't like we're not this like we don't
really have like annoying looking code
like this code is super clean it's super
easy to like look at it and be like okay
I see exactly what's happening like
immediately right people who like Python
like that so that's pretty nice and in a
moment I'll show you another example of
a much more complex model that yet again
is very simple like this whereas like in
this case it's kind of like you know
whether and use a for loop or not to
iterate through something for like a
small neural network like what we have
here it's not as visible as to how much
this is saving us but on a much larger
network I'll show you huh what it looks
like actually I don't even have full
code for the network I'll show but
anyway so what we're going to do now is
model dot save and so whenever you're
done with fitting a model you can do you
can save the model so we'll just save
this as TF learn CNN model cool so
that's our model file okay and all that
does is saves your weights it doesn't
save like your model it's not like we're
pickling a mAh
or something here it's just saving
weight values so when we load it back in
the only thing we get to not have is
like these lines here we still have to
keep confident and later we'll low end
the definition of model and then what we
do is load back in the model because
again all this contains is just settings
for your weights that's it there's
nothing more to it so if you try to put
this on a neural network that has three
convolutional layers and then five fully
connected layers it's going to be like
oh I don't know what to do okay because
it doesn't have the same values okay so
you just have to you have to make sure
you're matching but anyway we'll save so
let's go ahead and save this and let's
run it hopefully we don't have any
issues let's see Python 3 C ntf learn
dot pi you okay downloads the data and
starts the training so you can see it's
got like this really nice UI almost
right I do wish that maybe accuracy was
more like maybe Boldin orange or
something like that like floss is really
important but accuracy is also important
so I'm not sure why we don't do that and
really all these values are pretty
important but like you obviously
wouldn't need to capitalize a demand
epoch and so anyway that's it they give
a crap they're not watching my videos so
so anyway we can already kind of see
actually like after like the first kind
of batch um it was done but you the step
remember we did that like snapshot step
500 you can see you that's like here
2000 2500 3000 34 40 3500 4000 there
somewhere it doesn't honor that like
2500 then it does 25 80 and then this
3000 is 34 40 I don't know for sure
maybe that's the end of the epic or
something like that and it's like okay
we're done anyway so here's here's what
we have right now okay I'm not positive
on why does that that's just my guess so
while we're waiting on that we're almost
done right up eight um so I'm going to
move this over now what we could do
later is like once that's trained we can
just
and this out we already have the
definition of model so now what we could
do is we could say model dots load I'm
pretty sure I like the load data I'm
pretty sure it's load will try load and
we'll see what happens
so load TF learned model so copy this
paste that alright and let's see if
we're done yet yes here they are
compared both about 97 98 percent
accurate so one is ninety seven point
five then one's ninety seven point eight
if you kept rerunning them you might get
slightly different numbers but yeah
somewhere in that range just wanted to
show their same model same things
happening so yeah okay so okay so we've
done that let's save this will load in
TF learn and then let's like do
something some so one thing you could do
is like you can predict so you can print
a model dot predict let's do test X the
first if the first to test X and let's
just get a quick prediction
hmm listen Dex out of range
really
Oh weird all right I'm not sure how that
mistake was made but all right test X
that all right let's try again test X 1
nope
test oh I see what I did okay test X it
needs to be a list there that was way
too difficult so what was the initial
one I can't remember how that it was
almost like part of the thing or
something okay so as we can see 0 1 2 so
for literally the character 2 but also
the second index is a perfect 1.0 if we
ran like a tard max or something on here
this would be the max panel so so that's
a prediction that you could do also like
let's say you want it like you've
trained it and then you're using it then
you've got another new batch of data ok
fine
model dot fit right run model dot fit
again and then again save it of course
and you know you can keep going on this
way so it's not totally so like it's
online in the sense that it's constantly
training but you could do something like
that now I'll save that and the final
thing I want to show you is going to the
examples I'm going to pull up Alex net
here this is an example of how to apply
Alex net so Alex net is a network that
was proposed and actually seems to be
one of the most successful if not the
most successful for image data
this is Alexson so as you can see you
got like player here and here and here
and here and here a lot it's much more
complex and yet we were able to whoops I
thought it was in this other tab it's
over here I guess yet it was able to be
applied in this is 56 lines but 16 of
those lines are used just to source alex
net okay so really 40 lines that was
applied I don't even want to know how
long it would take to write this and in
tensor flow or how many how many lines
that would be not that lines are like
the ultimate measure of anything but it
would definitely be pretty confusing
looking as opposed to just this
right okay so anyway that's TF Lauren
that's a little bit on why you might
want to do abstraction there are other
modules and libraries for doing this
with tensor flow feel free to check them
out I just I felt like TF learn was the
simplest like I actually liked TF Lauren
I just thought if you're going to do
abstraction you might as well do it with
TF learn
caris is a really well-documented one
it's an old one so and it works with
tensor flow or theano is the back end so
a lot of sample code that you're going
to find might actually be in care is
because Karis has been around for a
really long time as opposed to TF learn
which has not been around for very long
times they're going to find less
examples of models and that than that
but at the end of the day tensor flow
hasn't even been around for that long
anyway tension flow is still in beta so
either way you're going to have a hard
time finding code even just written in
tensor flow but for the most part every
time a new paper comes out people
implement it because they want that to
be the the main github or whatever that
gets traffic for that implementation of
something in some framework or library
or whatever so people are actually
pretty quick at doing it anyway that's
it for this tutorial if you have
questions comments concerns whatever
feel free to leave them below otherwise
as always thanks for watching there's
all this for subscriptions and until
next time