Artificial Neural Networks Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to this module on
artificial real liquids so automation
hero networks where a computing math
model inspired by biology so we have
neural network architectures that have
been proposed for a variety of different
magnetic stats like regression
classification clustering feature
extraction etc so these architectures
essentially are networks of simple
computing entities so this is like a
very simple threshold entities that are
connected together in a specific network
architecture that give rise to complex
computing functionality Norfleet there
has been a significant resurgence in
interest in artificial neural networks
especially in under the domain of deep
networks about which we will see in one
of the later modules so as far as this
module and the discussion about
artificial neural networks is concerned
in this course you look at only the
classification tasks and many of the
ideas we talk about here for
classification or generalizable to
regression at fault while for the other
kinds of analytics tasks will need
different architectures and but we are
not going to cover that in this this
course so the inspiration comes from
biological neuron so let's not worry
about the complete complex structure of
a neuron what we really have to focus
here is on the input and output so the
neuron receives inputs from the
dendrites or from the right branches
from other neurons and when the input
signals is above a certain threshold
it's going to produce an output is going
to be transmitted by out the synapses to
Iran cetera further down the line so
these connections through the dendrites
and synapses are going to result in a
very complex network and even though the
computing done by each element is a very
very simple summation and thresholding
they will be sum total of this taken
across active network can do ways to
mainly complex publications so the
computing unit is something that's very
simple so it's going to take a set of
inputs x1 to xn and it's going to
compute some function on function is
very simply compute over
and then it'll produce an output so
we'll look at what this function is
going to be in detail in the next few
slides so the initial model for this for
a biological neuron was proposed by
McCulloch and Pitt's in 1943 it's called
the McAuliffe it's unit it gives only
binary signals so zeros and ones so
either an input is active maybe no it's
cases represented by one if it's not
active in this case is affected by zero
and the nodes also produce only binary
results or the outputs could be either
zeros are one so the edges between these
different nodes were directed unweighted
they could be of two types
it could be excitatory or inhibitory and
again like I mentioned earlier they
transmit binary signals so what is the
computation that happens here so let us
assume that the metal of the unit gets
input x1 to XM through an excitatory
atreus so these are positive interests
and inputs y1 to my em throw em
inhibitory edges so that means these are
interests that could produce the
depression in the function or could
actually stop the functioning of the
neuron so the assumption that was made
is if M is greater than or equal to 1
that is that is at least you want
inhibitory edge right and if any one of
the inhibitory it this is one so if
there is a one inhibitory input then the
unit as a whole does not produce any
output regardless of what the inputs x1
takes an arm if none of the inhibitory
inputs on one or if there are no
inhibitory inputs at all the unit
computes the summation of x1 takes in
right let's call it X and if X is
greater than the threshold that is
specified for each unit it's the greater
than threshold theater then the result
of the computation is a 1 and the result
is a 0 so that's very simple since you
can think of it as adding up all the
inputs that come to the newer and if
then the summation is written in either
show theta y1 always output 0 so the
inhibitory ages in some sense here
acting act as a gating signal so if it
is 1 the output is always 0
if the inhibitor is equal to 0 then the
output is the result of the computation
so this essentially in the vekl up its
unit is implementing this threshold
function you should input this below
theta I'm going to see output of 0 this
input is about theta efficacy output of
1
this essentially step function so what
kind of computations can you do with
this you can actually do almost all your
familiar boolean operations within
MacKillop it's Mira
right so you can think of doing an and
operation where you have two inputs x1
and x2 and the threshold is set at two
so if only both x1 and x2 are 1 so it
will be greater than or equal to the
threshold and therefore the output will
be 1
and for implementing it or you can set
the threshold at 1 so either x1 or x2 is
1 then the output will be 1
how about implementing and not the unit
you can implement a not unit by having
x1 act I said inhibitory input so this
circle here indicates enable input so if
x1 is 1 that means they said they knew
Ron will you inhibitor the output will
be 0 on the other hand if x1 is 0 then
the output will be whatever is according
to the result of the computation but you
can see here that the threshold for this
view on a set at 0 therefore the output
will always be 1 as long as there is no
inhibitory / okay so if x1 is 1 then the
output will be 0 x1 is 0 output everyone
that's fundamental it's not function so
once you are able to implement this
kinds of and or and not then you know
that we can connect all these neurons
together and then implement any boolean
function that you want but it says is
this what really we are interested in
right so we are not really interested
with that because we want to be able to
do more complex classification problems
where we would like to learn simple
things like linear surface surfaces or
more complex surfaces that separate two
classes so that has been in the goal of
classification we have looked at so far
so in 1957 Rosenblatt proposed a very
simple
extension to the McAuliffe its model she
called a perceptron the board crucial
thing about the perceptron is that a it
introduced weights at the inputs crucial
differences from the perceptron from the
McAuliffe its model is that the
perceptron introduced weights at the
input and then it the output could be
either a 1 or a minus 1 depending on
whether the weighted sum of the
England's is greater than a threshold or
not it's a computing rate for the
threshold theta so I'm just repeating it
so the output of the neuron is 1 if the
weighted sum of the inputs is greater
than or equal to theta it's equal to
minus 1 otherwise so what is the goal
here in perceptron learning the
perceptron learning we are essentially
trying to learn a hyperplane are trying
to learn a separating surface as we have
done in the past in the other
classification problems we are trying to
learn a separating surface that can
separate one class from the other so
what would the classes be in our case
classes in our case would be plus 1 and
minus 1 so this essentially means if WI
X is greater than equal to theta it
essentially defines the equation of a
hyperplane as we have seen in the
previous modules right so if this so if
you can take the theta to the other side
so we like to WI X I minus theta is
greater than or equal to 0 so you have
seen that there was greater than 0 it is
on one side of the hyperplane this
lesser than 0 it is on the other side of
the hyperplane and we are going to say
that it happens to one side of the
hyperplane belong to class 1 data points
the other side of the hyperplane belong
to class - no so now the question is
given a set of training data that gives
you the X X the vector X and the desired
output Y how would you find these
weights W is says that the perceptron is
actually implementing that I proclaim
implementing the right separating so the
way to do this as follows we start off
by randomly initializing
weights to some valium and then you look
at the prediction that is made by the
weight right so the prediction that is
made by the the current setting of the
weights let us call it whoa and that
target is the two class of the data
point X so that's okay this will be
either plus 1 or minus 1 and likewise
voy is also plus 1 or minus 1 so your
goal is to make sure that your
perceptron output matches the target
value so the perceptron training
algorithm has a very simple rule so at
every presentation of an input point you
change the weights by namun that is
proportional to the difference between
the target value and the actual output
release times the like the input on the
particular bit so WI changes by an
amount that is proportional to t minus
Rho times X I so eita here is a small
constant maybe point one or point zero
one it's called the learning rate so one
thing to note here if for a particular
input X I produce the correct output if
the class is minus 1 and I produce minus
1 the class is plus 1 and if it is plus
1 this expression evaluates to 0 you can
see that this expression evaluates to 0
and therefore no changes in the weights
will happen so essentially what happens
here is you change the weights only
whenever you make a mistake and that to
you make the you change the way
proportional to the input variable so if
X is saying a small value say 0.1 or 0.2
then the changes in the weight will be
small and as well as the first way when
X is a large value let us say 1 or 0.95
and things like that then the change in
weight should be large so this is
essentially because the larger the input
variable the more important it is going
to be in the prediction of the output at
least is the way we have set up this so
far so that's essentially the simple
training room so whenever you make a
mistake right he take the vector for
which you have made a mistake and add
some small fraction of the vector to the
weights so this looks like a very simple
rule
but then back in the fifties perceptrons
created a lot of humankind people saw
that time the perceptrons were able to
learn from scratch trying to solve some
things which are considered hard
learning problems and then they used the
perceptrons they were able to solve that
right so much so you can see here the
hype was that they were going to build a
computer that expects to be able to walk
talk see write reproduce itself and be
conscious of its existence such a
significant amount of hype and it's
always hard to live up to any any height
that is so disproportionate to the
actual effort that was achieved at that
point right so let's take a look let's
just step back and take a look at what
can a perceptron learning they said
newspaper article really true I what are
the limits to the perceptron learning
ability so here is a very simple
perceptron here okay so it has two input
variables X 1 and X 2 and that's a
threshold of 1 right and the weight W 1
is 0.9 and W tools too so if you look at
it essentially it implements this
straight line here so everything above
the straight line these light colored
regions belong to one class and the dark
colored regions belong no another class
so we know that these are the data which
are linearly separable we saw this in
the case with SVM's so these are data
that are separated by a linear
hyperplane or the linear separating
surface right so all the data points for
which the W transpose X evaluates to
greater than 1 will protect a class of
+1 all those that evaluate to less of
than 1 we get a class of minus 1 right
so again let's go back and look at the
simple logic function that we saw
earlier so it can implement an odd so
essentially all requires you to have a
hyperplane of is passing here so
everything to this side is become +1
everything to the side becomes minus 1
and likewise you can implement and so
you can draw a simple hyperplane so I
would think that this side becomes plus
1 everything this side becomes minus 1
or 0 depending on how we want to put
let's look at another one look at a
simple problem just like our and ant the
XOR problem so Minsky and Papert in 1967
69 you know famous monograph called the
perceptrons showed that well a simple
problem like x aw right so where the
truth table is given here is the inputs
of the same you put a 0 if Santos are
different the output of one a simple
problem like Exxon is not linearly
separable you cannot draw a hyperplane
that separates these two classes so
forget about walking forget about
talking and doing all those wonderful
things that were claimed in that
newspaper article perceptrons cannot
even solve such a simple problem as XR
which essentially says that two things
are same you output a zero two things
are differently on put a 1 if it can't
recognize the similarity between the
simple inputs like zeros and ones why
can't it do complex applications
so once Minsky and Papert show this is
kind of dampened the research into
neural networks for a long time until
that was available much later so what
sir tones can learn only its linear
decision profits that's the takeaway
message here so does this make this
whole idea of neural networks completely
useless because they can learn only
linear decision boundaries in case of
SVM's we saw that we could get it to do
nonlinear boundaries by by going into a
kernel expansion is there something
similar that we can do here let's look
at let's look at how we can change the
representations right and try to do
something more clever right so if you
look at the original problem the exhaust
problem so I have my inputs x1 and I
have main put x2 and now we can see that
in this space the problem is not
separable
but let us look do a simple
transformation on my data points so
instead of looking at x1 I will define
my first variable s naught X 1 and X 2
similarly I will defend my second
variable as x1 and not x2 so if you
think about it
so we can now plug in different values
of x1 and x2 here and see what the
outputs will be and then you can see
that when X 1 is 0 and X 2 is 0 the
output is going to be 0 right when X 1
is 1 and X 2 is 0 right see the output
here will be X 1 is 1 and X 2 is 0 the
output here will again be 0 and X 1 is 1
and X 2 Z the output here will be 1 and
we know that 0 1 the output has to be 1
so that we get it here and likewise for
the symmetric case this will be the
output
and so you can see that this is again
going to be 1 and when X 2 X 1 X 2 both
are 1 again the output will be 0 0 and
therefore this is the resulting point
now obviously with this representation
like the data power points are linearly
separable so now the task becomes 1 off
finding the right representation says
that the data becomes linearly separable
for the next level next stage of
computation so people realize this very
quickly so even though a simple
perceptron cannot solve complex problems
like it's all right which are not
linearly separable you could actually
stack layers of your neurons and then
have the first layer compute something
that is simple right so you can always
compute not of X 1 we saw that earlier
and I and also can be computed by a
single neuron right so you can just get
the have layers of neurons that I simply
compute your features and of not X 1
comma X 2 and then have another neuron
right which takes the output of these
neurons combines them together and
produces the output that you want so
people very quickly realize that
stacking these kinds of neurons in two
layers right allows you to do more
complex computation in fact it is easy
to show that stacking these neurons in
two layers actually builds a universal
function representation that allows to
represent any boolean function you see a
combination of neurons
so what is the problem now we know how
to solve this more complex problems why
didn't the research in your electrodes
pick up again right so that so the
question here is when I start connecting
all of these neurons in two layers right
how do I find the weights so the
perceptron learning algorithm make no
longer work in this case it actually
does not work in this case and people
who are struggling to come up with a
mechanism for training all these weights
right so you can see that there now we
started putting these things in Tulane
so there is one input layer and one
output layer so there's one input layer
there is one output layer and in between
this you could have many layers of
neurons you need typically call hidden
layers because you don't observe their
outputs directly right let's so now we
have this many many hidden layers of
weights and we know it's it's little
hard right to find out what these
weights should be and so in the mid-80s
around 83 an algorithm was proposed
called back propagation which allowed
you to learn the weights of this these
all these hidden layers right so for the
rest of the presentations we'll be
looking at the standard 3 layer networks
right so there's an input layer X 1 to X
T ok and that output layer which we will
denote by f of X and one hidden layer of
neurons so these take the inputs from
the input layer do the weighted sum do
your thresholding function and then
we'll produce an output and then the
neuron in the output layer will take all
these outputs of the Gideons take their
weighted sum and take that the Scholl or
not and then produce you instead of
using a regular instead of using a heart
threshold you use a kind of a soft brush
roll right in order to do these
computations this is needed so that we
can derive more efficient training
algorithms later so the output of the
hidden units will be given by G of the
bias term this is the theta that we had
since of TTL so we're going to call it B
1 and plus W 1 times X and the output of
this will be denoted by H of X and the
output of the the final layer of neurons
is given by some function whoa of B 2
plus W 2 times H of X and so now the
goal here is to figure out what this W 1
and W 2 are going to be so this is
called a three layered network even
though there are only two sets of
weights that we have to learn so the
layers here talk about the the neurons
here so we use for each input variable
we assume that there's a separate neuron
that is activating the hidden units
right so this is called the standard 3
layer Network structure so what are the
different activation functions you can
use so we already looked at 1 which is
the threshold function right you know we
can also have just a linear activation
function that basically takes the
summation of weighted summation of all
the inputs and outputs it as it is right
we can also look at the sigmoid 'el
function that sigmoid logistic function
which takes the summation input and then
squashes the input so that it lays
between 0 and 1 right and then there is
a steep raise somewhere around the
threshold so that if the transitions
rapidly from 0 to 1 and if you're
interested in having signed outputs then
you can think of using a hyperbolic
tangent where the inputs are going to
the outputs are going to transition
between minus 1 and plus 1 and again
around the shown
so there are parameters that control
where the threshold would be and how
steep the rise would be so another
transition function is sometimes used is
a squashing function which is zero
before the threshold and one at a
certain distance higher than the
threshold and in between you have a so
linear approximation and to the step
function in this called the squashing
contract so typically in most of the
neural network architectures that we
look at we will be looking at either the
hyperbolic tangent or the the logistic
Sigma
or the linear activation because these
are differentiable and this allows us to
delay efficient training algorithms for
the same if you are so if you are doing
a classification problem then the output
layer could be either a hyperbolic or a
logistic sigmoid and if you are solving
a regression problem the output neuron
could be linearly or not so that you can
do a cooperate regression fail the
hidden layer almost always has to be a
nonlinear function and with a little bit
of work you can show that if the hidden
units have a linear activation you might
as well not have them at all and what is
the function that is implemented is
something which can be as well
implemented by a single layer of neurons
and so we look at that the next module
will look at how you would exactly find
out these weights given the assumption
that you are working with that sigmoid
and logistic function so the function
for a sigmoid a logistic thing is given
by F of net is 1 by 1 minus e to the
power of minus net so there's the
function and you look at given that this
is activation function how we are going
to derive the weights of the two-layer
standard three-layer so that's in the
next class