Deep Learning with Neural Networks and TensorFlow Introduction Video Lecture

what is going on everybody and welcome
to a new section in the machine learning
tutorial series and that is deep
learning with neural networks tensorflow
and of course python so neural networks
are not new by any means but they are
currently the state of the art and are
achieving things that pretty much no
other machine learning model is doing
right now
and really this is fairly recent so the
neural network has been around since the
1940s but basically it was pretty much
worthless until very recently there's a
few things uh in the 1940s it was more
of like a concept and there was really
no way anybody was going to do anything
with it and then in the early nineteen
seventies I think 1974 Paul where Bose
came up with a way to kind of neutralize
the threshold which we'll talk about
later but so that kind of helped a
little bit but pretty much the neural
network was worthless until around 2011
or 2012 where with deep learning and
just massive data sets we've come up
with neural networks that are doing some
pretty incredible things and even this
first example that I'm going to show you
I think is very powerful at showing how
incredible these neural networks are at
creating models of data so anyways the
neural network is just biologically
inspired and again it came out in the
1940s when we knew very little about the
brain and we still know pretty much
nothing about the brain we know a lot of
stuff but we don't know how it works so
this is kinda it's just biologically
inspired this isn't necessarily how
everything works or how we think or
learn or anything this is just mostly
how we think we might think or something
like that so anyways that's enough on
history let's jump into the neural
network and all that alright so now
we're going to run through the theory of
a neural network and how it works so
obviously a neural network is a network
of you guessed it neurons so how does a
neuron look a basic neuron well to start
you've got these little things called
dendrites and then those connect and
you'll have the the nucleus and then it
goes down the axon to the axon terminal
that has some more little squiggly
things
so you have your dendrites which are
these your nucleus is this little green
thing I'm not gonna write out nucleus
and then you have your axon and then
your axle and then you've got the axon
terminal okay but of course a neural
network again it has a network of
neurons one neuron won't cut it but two
neurons will so then another neuron
might be here okay so these are the
dendrites and then it connects to the
nucleus you got your axon and then your
axon terminal great okay and our current
knowledge tells us that these don't
actually connect there's like some white
space here but between them messages are
sent and that's your synapse now
interestingly enough none of these terms
are present in a neural and the neural
network an artificial neural network
that is so from here on out like this is
a biological neural network anytime I'm
referencing a a biological neural
network
I'll call it a biological neural network
but from here on out I'm just going to
call the artificial neural network a
neural network and I expect that you
understand talking about an artificial
one so anyway interestingly enough
nothing in the neural network model uses
dendrite synapse or axon axon terminal
really anything which is interesting but
anyways they don't but this is kind of
what we're modeling after right this is
the biological neural network that we're
trying to model and what happens is you
basically have inputs right so in this
case like looking over here right you've
got three inputs that might come through
so so you start with just raw data so
this might be a 1 0 and a 0 or something
like that those inputs come in they go
through the nucleus in theory this does
something sends the data down the axon
possibly if it fires and passes through
synapse to the next neuron ok so
something like that happens and but
Meeny sometimes the neuron doesn't fire
there is no communication at the synapse
and so on ok so that's kind of the
theory of how neurons in our brain
works now let's take this model and
model a neuron in the sense of what
we're going to be dealing with in the
artificial neural network let's model
that so the artificial neural network
model of a neuron is is fairly similar
basically everything is going to start
with your raw input data right and then
eventually it becomes just regular input
data so I'll just call this input data
so we'll say the input data we're going
to have an X 1 and X 2 and X 3 in
reality you might have like in the first
example that we do you'll have 784
original input values okay
so it can be quite a big a big number so
you've got your input values and then
what happens is those input values are
passed along to a sum so they get passed
along and they are just summed together
now along the way they are weighted and
so this has a weight one this has weight
2 and this has a weight three and so
each of these weights are unique weights
so you've got the original input value
times the weight and then all of those
are added together into this this sum
now in neuron based on the input either
fires or it doesn't fire so how do we
determine whether or not it fires well
it gets passed through a threshold
function right so that's this and a lot
of times you'll see this depicted like
this well hold on that's a little better
ok so sometimes you'll hear this
referred to as like a step function
because it literally like looks like a
step but all that means is at some point
like this is 0 and at some point once
you pass a certain threshold of X the
return value for Y is 1 right we're at
the y axis now so at some point when you
pass a certain threshold you'll fire but
if you haven't passed that threshold you
won't fire ok but if you do fire or
basically with in with a regular neuron
if you don't fire you just don't fire
but with an artificial neural network
that
corresponds to either a 0 or a 1 right
you're one of those two things and then
that's 0 or a 1 an artificial neural
network goes and is fed through another
artificial neuron right this 0 or 1 now
becomes one of these values right it's
one of the new input values to another
connected neuron and this will go on for
however many hidden layers that you
might have now you might be confused
what that is but we're about to model it
but this is just a single neuron now
before I leave this little image one
thing I will change is typically people
are not using a step function as the
threshold simply because a 0 or a 1 is
not ideal you'd rather have something
more along like a scale or something
like that so in general people aren't
actually using a step function they use
a sigmoid function which looks more like
this okay something like that and it's
called sigmoid just simply because it
looks kind of like it has an S shape it
does not look like this though right
it's still on a graph and it just kind
of goes up and it kind of looks like an
S but obviously it never goes backwards
ok anyway so that's actually now the
threshold function but of course that's
not anymore a threshold so this actually
is called your activation function so
this is your basic modeled neuron but
this actually breaks down as you build
the network right your output let's say
is a Y is simply a function of your X's
and your w's which are vectors in this
case that's it right and you might have
used some sort of relation to your
sigmoid function here but for the most
part it's just a function of your XS
times your w's right your your input
data times the weights so this is a
single neuron now let's look at what a
neural network to scale looks like so
probably when you see a neural network
shown you probably see something more
like this like you'll have X 1 X 2 and
we'll just do X 3 for now
okay and that's your input and then
you've got the actual the other neurons
so I'll just draw a neuron one actually
let me make some space here I'll do one
two three four and then we'll do maybe
one two three and then we'll just have
one here okay and what's going to happen
and in fact actually we can do we'll do
this it doesn't really matter we'll talk
about that in a second so then all of
your input goes through all of your
neurons in the most basic model let's
say so all of these are connected okay
and each of those connections has a
unique weight associated to it and then
again from here each of these are
connected to each of these again with
another unique weight and then these are
connected to these again unique weights
okay so that's your neural network now
over here you have your input and this
is your input layer and then here this
is your hidden layer one right that's
this area that's your hidden layer one
and then here we have a secondary hidden
layer so that's hidden layer two and
then this is our output right and
theoretical output here might be one
zero one right and one zero one might
correspond to something now in many
cases instead of one zero one and in our
example actually it'll be something more
like one zero zero right or zero one
zero something like that and I'll
explain why that's the case but in our
first example that's more like what
we're going to be seeing so if your
might as well shake that so this is a
deep
our own Network we've just modeled a
deep neural network why is it a deep
neural network well it's because we
haven't hit a second didn't layer if you
just have one hidden layer that's a
regular neural network if you have two
or more hidden layers well
hot-diggity you've got yourself a deep
neural network it's as simple as that
and that's why I'm really not going to
split up doing like a neural network in
deep neural networks because they're the
same thing it's just the number of
hidden layers and in the code that
you're going to write there's really you
know you're just adding layers but as
far as modeling one layer versus or
modeling nineteen layers it's pretty
similar so I'm not really going to
separate these two things they're really
just one so that's it that's really all
there is to the neural network so what's
the big idea like why did this take so
long to basically come to fruition and
be the best machine learning model that
we know of right so first of all the
biggest thing here is the required
amount of input data so not input
features right these are just these here
are input features that doesn't
necessarily have to be huge the larger
it is the more descriptive it might be
but actually the biggest thing is the
input you know the number of samples
that you have so in the first example
that we use the number of samples I
think to train is 60,000 which is
actually very small uh most of the you
know big commercial neural networks that
are doing crazy things might have five
hundred million samples or something
like that after about five hundred
million it appears that with diminishing
returns you're not really seeing much
much better results but that's about
what it what it takes all right right
now so recall with like let's say the
support vector machine if you followed
along that tutorial we had what was
called a convex optimization problems
perfectly convex in the in the sense
that as you optimize or the actual graph
the graph of optimization looked like
this right so if you wanted to optimize
you could just take big steps big steps
up we took two biggest episode you take
little steps and then little steps right
it's very simple right but with a neural
network your optimization graph is
probably more like this it's like it's
some some crazy stuff going on there
okay so the optimization problem isn't
as simple not only that but with the
support vector machine we basically had
two variables we had W and B that we
needed optimize whereas here we have
even in this really simple neural
network I'm not about to count all the
lines but we have a lot of connections
there like each of those lines is a
unique weight okay that's a lot of
unique weights that's a lot of variables
and that's a really challenging
optimization problem both in the
mathematics but also mainly in the
computation required for a machine to be
able to solve that also you need a lot
of data right so you need a lot of
samples a lot of connections the more
samples you have the more weights you're
going to have of course right so it was
just kind of like this perfect storm
that was both stopping and then also
what is allowed in neural networks to
grow today because we have both huge
data sets available now and we have
computers capable of processing and
optimizing against those huge data sets
so when those two things came together
that's really what changed and allowed
the neural network to truly shine the
way it has today so like for for simple
classification tasks the neural network
is actually I think you know close in
performance to other algorithms so for
example you know the support vector
machine state of the art might be like I
don't know let's let's just say an
example 97% accurate on this figurative
sample set and then a neural network
might achieve 98 or 99% okay and that
little tiny percentage is what
everybody's fighting over right now like
this is what people are spending years
trying to trying to eke out that last
little percentage and that's important
but to me that's not as impressive as
the other things that neural networks do
so for simple classification tasks
that's really great that it can do
relatively similar accuracy but what's
more impressive is actually the
modeling aspect that neural networks do
and we actually don't really fully
understand how the modeling with neural
networks works but it it actually it
does very very well so there are a lot
of digging and analysis that we can do
to demystify some of the answers but
because of however many variables we're
dealing with it's just not not really
possible for us so so for example like
consider the following like jack is 12
years old
Jane is 10 Kate is older than Jane and
younger than Jack how old is Kate okay
so most of you who you know probably
didn't follow but right but if you think
about it might take you a second but the
answer is eleven right and in order to
get that answer you kind of have to
employ a little bit of logic which is
something that machine learning
algorithms up till very recently have
not been able to do they can do good at
classification tasks and stuff like that
but they're not very good at like
modeling logic or figuring out how to
model that logic and up till very
recently if you wanted to make an
algorithm to answer that question you
would have had to model that logic
somehow and you'd have to know a lot
about linguistics and stuff like that
whereas with a neural network you
actually don't you take the neural
network and you give it a bunch of
examples just like the one I read off
match it to the the answer and keep
doing that all right you do that
you know a few million times or you know
let's say 400 million times and it turns
out the neural network can actually just
figure that out on its own it figures
out its own model how to figure out how
to model the logic and all that it does
that on its own right that is what it's
so impressive about the neural network
okay and even in the example that we're
about to do in the coming tutorials
I think is impressive the actual output
is not that great but the methodology
that we're going to use in the output
that we get I think is fascinating and
incredible but anyways enough on that
one question that people might have is
like where do you get these required
millions of samples of things so you
have a few options that are free at
least so one option is like image data
is something like image net you can just
Google image net and it works a lot like
word net if you follow like
annal TK series I have it works very
similar to that if you're not familiar
you can kind of poke around NL TK or
even imagenet and you'll figure it out
like if you just kind of click around
you'll see how everything is organized
but anyway so for image data you can use
image net next for like text data your
first stop probably should be the
Wikipedia data dump so you can get the
entire dump of Wikipedia which is
actually pretty cool and again neural
network can begin to model things for
you so you can do a lot of really
interesting things with Wikipedia other
things are like chat you can usually get
like chat logs or you could crawl reddit
and stuff like that for input/output
stuff like that um for speech I really
don't know many off the top of my head
I've not never heard of like a huge dump
for speech but there is a website called
tat Oba I don't know if I'm pronouncing
that right but anyway you can go to the
text-based version of this tutorial and
you can get links to all these things
I'm talking about just go towards the
bottom of the tutorial and then also
probably the largest data set that I've
ever heard of that's also free anyways
is common crawl most of you probably
don't have computers that I don't even
know a PC that would be capable of
running through common crawl unless you
had like I don't know a ton of GPUs in
SLI like you'd have to have more GPUs in
SLI than I know if a motherboard that'll
fit them I don't know how you could do
that anyway you prod that's more for
like institutions and stuff but if you
you know if you work for somebody who's
got a huge server or you have a lot of
money and you can buy the AWS stuff
common crawl might be a great and
basically common crawl is is its base
it's like every website it's like parsed
okay so that's just incredible
apparently it's petabytes in size so
there you go you need a hard drive that
can even hold petabytes so good luck
okay so a few hard drives all right also
if you need because obviously the the
the main point of failure for everybody
with machine learning is always like
data sets like where do you find the
data sets right and can you imagine
doing like a hundred million examples
for something like you know handwriting
100 million examples I get right so one
place you might always want to check out
is like the machine learning subreddit
you might ask there if anybody knows of
data sets so a lot of times like Google
just doesn't
find datasets but like through my own
research I've kind of just stumbled
across various datasets so you can kind
of ask around unlike the machine
learning subreddit or something like
that
and other people might be aware of these
datasets so anyway should be relatively
obvious now why companies like Facebook
and Google are so big into AI and neural
networks especially right they actually
possess the required volumes of data to
do some very interesting I'll by creepy
sometimes things okay so now that we
have that out of the way how are we
going to be working on neural networks
well we're going to be using tensorflow
which is a relatively new package from
Google it's still actually in beta at
the time of me filming this there are
other packages for machine learning like
diono and torch and now pretty much work
in the exact same ways simply because
what all this boils down to is that
thing I just showed you a few minutes
ago which was it's at all a function on
your X's and w's right that's it
okay and so if you learn how to do this
in tensor flow you should know how to do
this in the know for example okay you
just have to learn different syntax but
the actual modeling part is the same so
we really just need to pick one
framework I'm choosing tensor flow just
because tensor flow has some interesting
future capabilities or current
capabilities that I might use in the
future like distributed computing and
stuff like that so anyways in the next
tutorial we're going to install tensor
flow that's all we're going to do is
install tensor flow if you already have
tensor flow installed you can skip the
next tutorial and for that reason I'm
just going to put this one in the next
one out at the same time because
hopefully some people won't need it so
all we're going to be doing is
installing tensor flow on Ubuntu and I'm
going to do that with a virtual virtual
machine on Windows so if you want to
follow along perfectly you'll want to
have tensor flow on a boon too but for
the first you know handful of tutorials
it won't matter what operating system
you have tensorflow installed on I'm
just going to be using a boon too
because it's it works on a boot okay but
if you're on Windows and you've never
used a virtual machine or anything like
that you might want to just follow along
the next tutorial
and get all that kind of setup that way
everyone's kind of on the same page but
if you have like a Mac you can solve
tensorflow on your Mac and be just fine
also you can put tensorflow on docker
with Windows I'm going to put it on a
boot - with a virtual machine anyways
that's what we're going to do in the
next tutorial if you have questions
comments concerns whatever feel free to
leave them below otherwise as always
thanks for watching and until next time