Recurrent Neural Networks (RNN) - Deep Learning with Neural Networks and TensorFlow 10 Video Lecture | Machine Learning with Python - AI & ML

what is going on everybody and welcome
to another deep learning with neural
networks Python and tensorflow tutorial
in this tutorial what we're gonna be
talking about is kind of getting into
the main types of algorithms people are
running with deep learning so most
people are not doing like your
traditional multi-layer perceptron just
a loan which is like the deep neural
network that we did to start because
this isn't really it's not the best
solution for really any problem so most
people end up doing something else and
there's two major forms that people
usually choose to work with and that's
going to be either a convolutional
neural network or recurrent neural
network and you can also have
combinations of those two and then there
are other models out there and there
will probably be more to come
it's just those are the two probably
largest types of deep neural networks
that you're going to come across so
we're going to cover those two first
we're going to start with the recurrent
neural network now the recurrent neural
network tries to solve a problem that is
pretty much inherent that I think I
guess probably the first thing to start
with is like most people are under the
concept that we have only five senses
when really we definitely have more than
five how many I don't really know for
sure but for sure we have at least a
sixth sense it's not that we see dead
people it is a sense of time we have
this sort of temporal sense and physics
tells us time could be a whole host of
things it might not actually be like a
duration like we think of it but for
sure we do have a sense of time to some
degree so with a traditional multi-layer
perceptron just traditional deep neural
network feed-forward backprop whatever
we have no sort of explanation of time
or maybe not even just time but order so
like the order of events matters
significantly okay so for example if we
say if we have like a sentence like
Harrison killed Billy
traditional deep neural network would
not know the difference between Harrison
killed Billy or Billy killed Harrison
right order is a matter of life and
death so order matters but also in terms
of like let's say you've got a machine
that you're trying to play catch with
and that machine needs to know is the
ball traveling away from me or towards
me and to know that it's going to know
it's going to take each frame basically
and decide okay that ball is definitely
moving away or that ball is moving
towards me maybe I need to duck or
whatever order is very important and of
course even just with language so for
this is typically why recurrent neural
Nets networks are more used with
language data and then for like imagery
data we use something like a
convolutional neural network if we just
are trying to analyze a single frame but
if we're trying to go beyond that
usually we're going to use some sort of
recurrent neural network it might be a
recurrent neural network in a
convolutional neural network tied
together but generally like a
convolutional neural network it's not
going to help you with like successions
of frames it's like that so anyway or at
least to my knowledge they want I could
be totally wrong anyway so before we
jump in also to the recurrent net I'm
just going to explain what our current
net is and then we're going to draw an
LS TM cell because an LS TM cell or a
long short-term memory cell is the most
I hate to say it's the most common but I
think it's the most common it's one of
the most common how about that's better
one of the most common cells that are
used with recurrent neural networks
there's also the gated recurrent unit
and there's even like a basic recurrent
network cell I've never even looked at
it I couldn't even draw one for you if
you asked me to but anyway
so yeah we're going to do that but also
I just want to address some questions
that people asked about AWS and their
graphics cards and all the people that
are using AMD so for the most part a
tensorflow Theano
and all the other all the other
frameworks basically for doing deep
learning are going to be the same
they're all doing the same thing it's
all weights and biases or outputs times
weights plus bias right it's pretty much
always going to boil down to those kinds
of concepts so you don't have to be
using tensor flow so I'm pretty sure
Theano has
AMD and OpenCL support so if that's you
then you you could just go that route
you don't actually have to use
tensorflow also people were asking about
well why not just teach us on AWS well
it OS is really expensive to be doing
machine learning on or at least I would
call it very expensive to practice
machine learning on but if you have like
a plan of attack and you know exactly
what you want to do and maybe you've
kind of tested it on your local machine
and all you really want to do is just
deploy it really quickly on a gigantic
cluster of GPUs then sure AWS might be a
good option and that's something I'd
like to learn about down the line so I
may or may not actually cover a tutorial
on that I really hate doing those kind
of like setup tutorials like just doing
this stuff was a pain in the butt as it
was so anyway we might do that you can
always look into it and see what you
think but it's it's a lot more expensive
you're looking at at least fifty two
cents to like a dollar an hour for
anything decent when you could just go
to eBay and buy like a GTX 650 for $70
and just run it overnight while you
sleep and that's the easiest thing to do
and the most cost-effective thing to do
in my opinion anyway let's get into a
recurrent neural network on a on a deep
neural network you had your input data
which was like X and then it was fed
into a cell and then you might have had
other X's too and they might have been
fed into that cell as well and they
would have weights and all this but it
was just like straight-up input data and
it didn't matter if like let's say this
is X 1 X 2 and X 3
it didn't matter like what order of
these things came in it didn't matter
like nothing nothing about order
actually mattered in this case now with
the recurrent neural net the way things
work is you've got your let's say your
input data we'll say this is just x1 for
now so x1 its input or sent into
whatever your activation function would
be so let's just say a 4 here and then
under a traditional neural network we
would just simply have output that's
kind of like the traditional neural
network but with a recurrent neural
network what instead we have is the
following
besides being output oh the output we
also
ricoeur right back into that cell so the
output now becomes part of the input the
next time around so this is basically
let's say at T at time step zero you
just have this coming in to the
activation cell but then at time step
one oops I went too far at time step one
you have this coming in and also
basically whatever was output here comes
over here right and then goes right back
into that cell so that's kind of that
why it's called a recurrent neural
network is we have this kind of notion
of repetition so another way to look at
this is like this so and each layer is
basically you know you've got like your
new input plus the input from the
previous input so from X 1 to X 2 to X
three we're kind of bringing along with
us the previous input data so if you
think about a sentence like Harrison
drove the car where each word is a
feature so our features then would just
be like Harrison comma drove comma the
comma car and then maybe we have like an
end of sentence character or something
like that but we'll just ignore that for
now in a traditional neural network
Harrison drove the car is treated
exactly the same as the car drove
Harrison so in the former case we have
an ordinary vehicle under the control of
a human in the latter we have a
self-driving car so recurrent neural
networks are basically trying to solve
this sort of sequential temporal problem
and it you don't actually just simply
have to use this on a temporal type of
problem it doesn't have to be something
that is inherently temporal so one of
the first examples that we use it on is
actually the m-miss data set and it
actually does quite a bit better I guess
I can't say quite a bit better but it
gets in theory or in practice quite a
bit better return anyway we'll talk
about that when we get there I think
it's kind of silly people fight so much
over tiny percentage increases like some
stuff it really does
are like in self-driving cars you need
that to be as as perfect as possible
100% need to understand the blob of tar
versus a baby in a blanket you know kind
of situation but there are many times
where that like any bitty percentage
difference that means everything to some
people I think doesn't really matter
that much but anyway so that's your
recurrent neural network and from there
we actually have one more thing for me
to draw for you all and that is the LST
M cell so a lot of people the recurrent
neural network is great but it it
becomes even more impressive when we
implement the LST M cell because if you
think about it what happens if like what
happens when you have like really long
sequences here you're probably gonna
have a lot of data that doesn't even
necessarily matter or maybe you only
need like one of the cells to remember
that or you know you don't need every
seldom every little bit of data that has
come in the past you only really need
one of the cells to remember it and then
maybe there's even some data that you
don't even you don't even need to
remember so for the LST M cell just take
note that basically the LST M cell is in
here it's this cell that's right here
okay so that's we're going to be drawing
next so with your LST M cell you're
going to have as usual some input data
that's going to come in and then of
course it's going to spit out some
output data but what's magical is what
happens on the inside draw a nice big
box hopefully that'll be enough space
for me we will start with the recurrence
so this R occurs around and the first
thing we have is what is called either a
keep gate or a forget gate basically
obviously they mean pretty much the same
thing it's like from this information
from like basically the previous
information that's recurring around on
us what do we want to keep or what do
you want to forget so the first thing is
just the question of a I'll call it or
forget
gate it's really hard to write letters
that small anyway so that's your keeping
forget gate so the first thing is what
do we keep what do we get rid of and
then from there you've got your input
right this this X input that's coming in
that's coming in and then we have a
question of what do we want to add so
we'll say add what
so from this new input what do we want
to add and then finally we have the
question of what do we want to output so
again as the information recursion you
got your forget gate or your keep gate
whichever one you want to call it from
there once we've decided we've got a
block of some information basically
we're going to send that on to the
addition what do we want to add from the
new input that's being added over here
so what do we want to add from there we
are going to decide ok what do we want
to output and what we output obviously
goes back goes back up here and then out
here to the actual output that we're
going to actually spit out alright so in
the next tutorial what we're going to do
is apply a recurrent neural network to
the MS data set by simply modifying our
model code from the previous tutorials
to just be a recurrent neural network
rather than a multi-layer perceptron if
you have any questions comments concerns
whatever leave them below otherwise
thanks for watching thanks for all the
support and subscriptions and until next
time