Practical 2.0 – NN forward Video Lecture - Torch A Practical Hands On Tutorial

we will see now how a neural network can
produce an output which is performed
through the feed forward or inferior
pass we will start by introducing the
basic perceptron we will also further
see how we can embed the threshold
within the model and therefore utilizing
a activation function which will be a
step function therefore we introduce the
logistic unit which output is
differentiable with respect to the
parameters and the input finally we see
how to combine multiple logistic units
to create a neural network and we will
see what are the equations that will
allow us to perform new fearin step or
feed forward pass let's begin with the
perceptron it was invented in the 1950s
by Frank Rosenblatt so there is a series
of input to this system let's call them
X 1 X 2 X 3 and so on until the last one
X N and then the output of this guy we
call equality side will be simply the
summation of all these input x specific
weights that I'm gonna call theta 1
theta 2 theta 3 and so on up to the last
one theta so we have that then it's
equal to theta 1 times X 1 plus theta 2
times X 2 plus theta 3 times X 3 plus so
on until last Reston
so it's theta
times X n we can notice we can write
these in a smarter way so if we define X
being the vector x1 x2 x3 up to xn and
then theta the same way so our vector
theta is going to be the vector over
theta1 theta2 theta3 down to theta n
then we can have that our z here can be
written as the product between theta
transposed and X so what is this what is
all this stuff for example I could use a
perceptron in order to decide whether
you pass or not the class for example x1
could be the homeworks x2 could be like
how much how many equations the answer
in class x3 could be the extra material
that you cover and the last one could be
like let's say the final project
of course we have that these parts will
not contribute in the same way so we may
have like theta one its equal to 30 then
say that 12 its equal to 20 sata3 could
be equal to 10 and the last one the
theta 4 in this case if we have an
example with just 4 inputs and 4 weight
it could be the final project which is
County like for 60% and then at the end
we can say that we can have that our
output so the H based on theta of X its
equal its equal to 0 if the summation of
these values some meaning like if you do
the homework completely then you get 30
points if you get the questions all done
you get 20 points so they are already up
to 50 then if you do extra material then
you can get up to 60 and then if you
can't do perfect perfectly the last
project you get up to 120 let's say we
put a threshold so how much how
difficult is the class equal to let's
say hundred so if your summation of all
the scores Z each below or equal our
threshold is called T and we have here
that T equal 100 which is our bias then
you fail the exam is going to be zero
instead you pass the exam or the class
if you achieve anything above the
threshold which was 100 here in this
case we can use the perceptron in order
to evaluate whether you can pass or fail
a specific course which has these
specific weights let's see
to reformulate this problem by
incorporating this bias T within the
system because we don't want to carry on
this T and you will see we see that this
is going to help us in us following
formulation we can incorporate the
threshold thing by simply adding one new
node to the perception that we would
call x0 x0 it's the same thing as a 1 so
basically X 0 is going to be always to 1
and we have that our Z which is define
as the product of theta transpose and X
but in this case we have that the vector
X its equal to x0 x1 blah-blah-blah up
to X and so in this case we have that it
belongs to our n plus 1 and the same way
theta is going to be theta 0 theta 1 up
to theta n and also she belongs to our 2
n plus 1 and so in this case we're going
to have the same as before but again in
x0 we will have x0 it's equal to 1 so
the formulation becomes basically theta
0 times 1 plus the rest no theta 1 x1
plus and so on upto theta n times xn and
when we write our final equation for the
Aged
we can simply say it's 0 if Z is lower
or equal 0 and it's 1 if Z its greater
than 0 we simply have brought the
threshold T from the right-hand side to
the left-hand side so basically this new
set it's equal to all Z minus T and
therefore we understand that - t it's
equal to theta 0 to simplify further
this notation we can introduce a
activation function that we are going to
call G which is the step function so if
I have my if I make the plot here here
we have said here we have G of Z and
then the function is going to be 0 until
0 include it to 0 and then it goes to 1
when it's greater than 0 so we can
simplify H and become simply G of our Z
which is G of theta transpose X and this
is also called activation so this is
going to be a so from the previous case
we will have that our parameter theta
it's equal minus 100 our bias so how
difficult is the the task
how how difficult is it to pass the the
exam for example and then there is the
list of values we saw before so there is
the 34 the homework there is the 24 the
questions in class there is the 10 for
the extra material and then the last
part is the 60 for the final project and
X each one for X 0 all the ways and then
it's going to be some values from
probably 0 to 1 which says how good you
have performed in those different tasks
and both of them belong to our n plus 1
let's say now we would like to have a
system that can predict whether you will
fail or pass a specific class but you
are not the professor so you don't know
how much each part of the class or the
homework or the exam actually wait for
specific for that for the total so let's
say we have the same system with several
inputs and we would like to find the
combination of parameters theta that are
actually correctly predicting whether
you can pass or fail a specific exam so
in this case we have that every
parameter is somehow tricky to figure
out because if we have small variations
in parameter the output age of theta
will not change unless is summation of
all the parameters is higher than the
bias then we are going to have a
positive outcome but if it's blow is
going to be zero output so we would like
to have a software way to determine the
output score so we understand how we
place for the specific
set of input we can do this with the
logistic unit so in this case it's very
similar to what we have just seen with
the perceptron but the only thing that
is going to change is the activation
function instead of having G the step
function G is going to be the logistic
sigmoid function which is equal to so we
have that Sigma of said it's equal 1
plus X to Z at minus 1 so I can also
rewrite this as 1 divided by 1 plus X of
minus Z so let's see how it works how
this plot what is the plot of this
function so let's draw a couple of axis
if Z tends to minus infinity we have
that minus Z tends to plus infinity then
we have that X of minus Z tends to
to plus infinity and therefore we have
dead 1/1 plus X of minus dead it's going
to tend to zero plus the symbiote is on
the zero side then instead let's see
what happens if we have that Z tends to
plus infinity we headed - set tend to
minus infinity therefore we have the
exponential of minus Z is going to tend
to zero plus in therefore 1 divided by 1
plus exponential of minus Z so this was
0 and therefore is going to be 1 divided
by 1 this tends to 1 and therefore here
we have a stays at 1 so and then let's
see for Z equals 0 we had that
exponential the of minus Z is equal to 1
and therefore we are going to have that
1 divided by 1 plus 1 is going to be 1/2
so it's very similar to what we have
seen before
so if we have values that are let's say
above roughly 5 the output of the
sigmoid is going to be roughly 1 if we
are below minus 5 then we are going to
be basically roughly 0 the United is
that now if we apply some small
variations on the weight we can move to
Z slightly to the left of the axis in to
the right so we actually move the output
which is going to be the sigmoid of the
scalar product which also is going to
change in value in this way we can try
to make some small variations on the
parameters and see how the output
changes also in this case we have to
remember that there is the bias here
always the plus 1 which is connected
down here and with the weight theta 0
which is basically our our bias just to
recap we have that X belongs to R 2 n
plus 1 theta belongs to R and plus 1 and
then we have that Z it's equal theta
transpose X and then our hypothesis
based on theta of our input X it's going
to be in this case our sigmoid of said
and then basically we had that this
belongs to the interval 0 1 we can
decide whether
we have passed the exam or if you'd like
a binary a binary output we can simply
ask we can simply have this the H of
theta of X greater than 0.5 if we are
above 0.5 then it's going to be passed
under Y it's going to be a fail but in
this case again we see we can see that H
of theta is going to be continuously
varying between 0 and 1 so we can see
how small variations of each of the
singular inputs or each of the singular
parameter will affect our final
activation and therefore our final
outcome here we have a neural network on
your network simply is a combination of
multiple logistic units in this case we
have a three layers neural network so we
have an input layer then we have a
output layer here and then in the bed in
the middle we have a one singular hidden
layer the dimensionality of the input
which we call s1 for size of first layer
it's going to be four so there are just
four input the size of the hidden layer
s2 second layer it's also for in this
case and the size of the output layer s
capital L or s3 in this case is just one
of course will be there will be always a
x0 here set to one which is connected to
all the other neurons then we are going
to have also here a zero
activation zero of the layer 2 which is
also a +1 which is all which is the bias
for the last node in this case we can
perform more calm more complicated
operations for example each of these
single the singular activations a1 a2 a3
a4 of the second layer can compute a
operations on the input and for example
evaluate whether specific combinations
of the input are above a specific
threshold or bias then the final outcome
will simply score based on the output of
the previous units the final output it's
it's like before called H in this case
we have the matrix capital theta and
it's also a function of the input
capital theta it's a collection of
parameters for example in this case we
have two layers so we are going to be
having theta of layer 1 which goes from
layer 1 to layer 2 and theta 2 which
goes from layer 2 2 layer 3 and more
detail we have that theta 1
first row it's simply the it's simply
the vector theta which is used to
compute a one-off to the second row of
big theta for the first layer second row
it's going to be also another vector
theta which is used to compute the
activation number two of the second
layer and the only role of theta two
it's is going to be the vector all the
parameters
to compute the only activation a 100
layer 3 which is also equal to the final
output age of big theta of the input X
so let's see now a summary of all the
all the equations that we have seen so
far a bit sparsely so let's see some
definitions now just to recap what we
have seen so far and then we see an
overview of all the equations that we
are going to be needed in order to
compute the forward pass of a neural
network so we have that AI of j is d i
activation in layer J as we saw here in
the red circles circles we have a 1 is
the first activation of D layer 2 a 2 is
the second activation of layer 2 it
reads the third activation of a second
layer and a 4 is the fourth activation
of the second layer on top we don't have
to forget the a0 a second layer the bias
term and also of course and of course
here x0 as we can see x1 it's also the
same sa activation one of layer 1 x2 is
going to be the same oh say oxidation
two of layer 1 and so on so if the
activation 3 of the year 1 negative
Asian 4 of layer 1 and the same way the
final activations of the activation the
only activation of layer 3 in this case
it's equal to the final H hypothesis or
with parameter theta capital theta on
the input X capital theta
is the connection of mappings to go from
layer J - J plus one so theta one for
example capital theta one allow us to
map the input that is the layer 1 to the
activations of layer 2 and let's see now
the full equations for the for the
activations in this case of layer 2 so
we have that a 1 activation 1 of the
second layer it's equal a sigmoid of
which is going to be theta
perp for the first layer and then first
row first column which starts from 0
because we have a bias term in the input
and this is x x0 which we know it's
equal to plus 1 then we have plus theta
first row first element of X that is not
the one for the first layer x1 plus so
on until the last one we have theta
first layer first row and the fourth of
x4 and there you go and the same happens
for the other activations so activation
second activation is going to be second
row so there's going to be our sigmoid
nonlinear function of the matrix that is
mapping the inputs of the
Slayer we go second row and element zero
which is multiplying the bias all the
ways don't forget then we are going to
have still mapping the input one second
row because the second activation
element 1 of X 1 plus all the others and
the last one is going to be mapping the
input to the second activation of the
fourth value of the input X and we have
the same for a 3 and then we have the
last one here so it's going to be the
last activation so the last row of the
input the input parameter collection
collection fourth row element bias plus
stick from the input fourth row first
element of the input plus all the others
and the last one going to be F D from
the input fourth row and also fourth
element of the input we can write this
in a much more compact way so let it be
a to the vector
a 2 to a 1 of second layer 8 to second
layer up to a 4 second layer which is s
second layer so we can write that a 2
it's simply in the nonlinear functions
of the sigmoid of what of these
operations here right so here we have
that first row of theta 1 and then the
second row and then we have the third
row so we can simply write the matrix
theta for mapping the input and multiply
by the actual input so what it does this
one is simply performing a matrix vector
multiplication so row first row by row
by the column which is going to be first
row as we said before here first row
first element first row second element
and so on until first row last element
multiplied by all the elements of our
input included Tobias and then all these
huge here amount of operations can be
simply done written down more compactly
this way then the final one we have the
output which is the hypothesis based on
the whole set of parameters of respect
with the into the input which is also
called the only activation a one of the
third layer it's going to be equal to
our sigmoid nonlinear function and then
we are going to use the map for the
second layer because we go from the
second layer to the third and then it's
going to be the only row but we still
write number one here
and then first the first element is the
element that is I connected to d2 the
bias times a0 from the second layer plus
our matrix for the second layer first
row on the row element 1 multiplied by
the activation 1 from the second there
plus all the others up to the last one
capital theta for the second layer the
only row and the fourth element of the
activation and here as well if we define
a hat equal the vector with +1 and a and
this one relates to the second layer
then we can express our final output H
of theta of the input X is equal our
sigmoid of the mapping for the second
layer multiplied by the activation with
the bias term from the second level and
there we go we have completed the whole
set of equations that we need to compute
the forward pass for a neural network
one more definition so we have that SJ
is the number of elements
for layer J so as we saw before s 1
equal 4 s 2 equals 4 and s 3 equal 1 in
this case by using this notation we can
write that theta J it's a matrix which
belongs to R off so let's have Adam here
we said that theta J is mapping DJ layer
to the J plus 1 layer so there will be
as many rows of the matrix is the number
of output so the number of elements
there layer is called s so we go from s
J plus 1 and the number of columns is
going to be number of elements we had in
the input which is s J and we had not to
forget about bias so plus 1 in this case
so if I draw here the matrix is going to
be number of output so s J plus 1 and
the number of number of columns is going
to be SJ number of elements of the
current layer plus the one for the bias
because this one is multiplied by the
bias plus our current input so this one
is SJ and this is just one element
and the output is going to be the vector
of s J plus 1 elements and it