Practical 2.2 – nn package Video Lecture - Torch A Practical Hands On Tutorial

in this practical we see all the pieces
we seen in the practicals 2.0 and 2.1
coming together and playing nicely in
torch using torch NN package we start
with reviewing the Jacobian and the
haitian partial derivative formulation
this is quite essential to understand
how partial derivatives are computed in
torch we will create then a logistic
unit with the NN package we will perform
the forward pass step by step in order
to apply the knowledge we have acquired
the previous practicals we we start with
an N linear which requires an input
dimensionality and an output
dimensionality we will learn then how to
use the curly bracket operator we will
see how to access the weight and the
bias of and inner layer and its own
derivative of the cost function with
respect to the its own weight and bias
we will see then how to zeroes these
partial derivatives and then we will
introduce the sigmoid non-linearity we
will see how to use the forward method
of the module linear and sigmoid our
modules so by writing an N dot module
I just represent both of them then we
will also implement the back propagation
pass step by step again to verify our
understanding of the underlying
algorithm we will start by defining a
loss function with the N n MSC criterion
mean square error then we'll see the
attribute size average of the criterion
and how to change its volume
moreover we see the D forward method for
a criterion now expects an input and a
target whereas for the module it just
expect and sing
moreover we'll see we will see how to
compute the gradient of the error with
respect to the output of the network
using the update grad input and then how
to back propagate the gradient
computation through all the modules of
our network which also required the
current input to the specific module and
the gradient add the output of the given
module we will see also how to
accumulate the gradient parameters
basically how to compute the partial
derivative of the error function with
respect to the parameters of a given
that you're in the specific case the
linear is the only module which has
parameters then we will introduce the
NMS equation which allows us to perform
the forward and back propagation with
just two simple commands so we'll see
how to add modules to our sequential
container how to forward an input
through all the modules within the
sequential container then we will
forward the output of the sequential
within our loss function where we
specify the input which is the
prediction of the network and the
expected target then we perform the
backward step which is actually for the
criterion most of the time like the same
of update of updated input the backward
step actually performs both the updated
input and the arc grad parameters
usually criterions don't have trainable
parameters but if they do then it will
perform that as well then we see how to
get a specific layout of a sequential
with the get method
we see how to zero the partial
derivative of the error with respect to
the parameters of the model with a zero
our parameters and then we back
propagate the gradient of the criteria
through the network and with the
backward method to which we have to
provide the input to the network and the
Grad criterion finally we can update our
parameters based on the learning rate
alpha and finally now we can train we
will train a generic neural network by
using stochastic gradient descent and
mini-batch gradient descent we will do
so by using the forward backward and
update grad parameters together with
zero grad parameters functions with just
illustrated we can do the same by using
instead n n stochastic gradient which we
simply provide the network and the loss
function we decide and then all we
actually need it's simply to ask the
stochastic gradient to perform a
training of our network based on the
specific criterion on the data set that
we provide as last point we will see how
to perform a regression task with a
three layer neural network input a
hidden layer of three neurons and just
one output neuron we will see this one
at the following address github.com at
call my user name torch machine learning
with torch so far we have used the
Jacobean formulation this means when we
have a partial derivative dy on the
vector X this is going to be a 1 Rho
matrix
where each component is going to be
simply the Y in the x1 up to the last
one the Y scalar on the X and if we have
instead the Y vector in the X so it's
only variable which is a scalar on the
denominator this is going to be equal to
the column vector of the derivatives of
the components so we have d y1 in the X
and down until the last one which is
going to be the Y M in the X and there
we go finally if we have a combination
of the two so we have the Y vector and
the X vector this is going to be the
Jacobian so which is the Y 1 in the X 1
and we go down till the last one so it's
going to be the Y M above the X 1 and
then if we go on the other side we're
gonna have still here the first row
versus the last element of N and then
down we have the last one element here
the Y M and X and and this is the
Jacobian of on Y
the last case in its when we have a
scalar so we have dy on the X X matrix
and this one since we use the Jacobian
formulation if we say that X belongs to
R M times n then the partial derivative
with respect to the matrix X is going to
be the the dimensionality is going to be
n times M so we have that the dimension
height is the transposed basically of
what it is on the denominator so like it
happened here in the first case we have
a vector column vector X so we have
since this at the denominator we are
going to have a row of derivatives in
the same way here we have a matrix of
the denominator and which is dimension M
times N and therefore the partial
derivative with respect to that matrix
is going to be in the transposed in
terms of dimensionality so it's n times
M for this very same reason let's
compute now the dimensionality of the
partial derivative of the error with
respect to the parameter theta capital
theta of layer L so a cut of layer L
it's going simply to be belongs to the
dimension s layer l plus 1 on the other
side up to here we have the the error of
the layer plus 1 is going to be of
dimensionality s n plus 1 and therefore
if we compute also the transposed
we are going to have one x s plus 1 and
therefore we had the multiplication is
gonna be having as many rows as SL plus
1 and as many columns as s l plus 1
small and we know that the matrix
capital theta of layer L goes from goes
to s layer l plus 1 and it starts from
SL plus 1 and we can see of course here
that we have the partial derivative with
respect to a matrix of s n plus 1 times
L plus the bias is going to be in the
transpose so SL plus the bias times s of
layer PI plus 1 in porch we are using
instead the patient formulation
and in this case we have that dy on the
capital X where X belongs to R M and
it's also belonging to R and M in this
case when we have to do the weight
update so we have our matrix theta we
can actually write these on entities
legitimate theta plus learning rate ETA
x de respect to D theta and we saw just
now that this all element here will have
the same dimensionality of dicen in the
Jacobean formulation we should have add
a transposition to the second part
but as we said in torch we use the
Haitian formulation where basically all
the results of the the the partial
derivatives are basically the transposed
of the normal kind of notation this is
also called enumerator layout and this
one the patient is also called the meter
layout you can read more about this on
Wikipedia and look for matrix calculus
let's run torch we would like to create
our see sigmoid unit let's require the
package nn let's say our neuron has m
equal five neurons and then the final
output has dimensionality string so we
can create D linear which is going to be
creating the weight of the input it's
going to be an an n dot linear going
from n to K so if we print linear with Z
equals from five elements of input which
is the dimensionality of our X vector
without accounting for the element X
zero the bias and it goes to our output
so we go from X the input of size five
from the output H theta of X which has
size three let's see what's inside this
linear gina has a bunch of things inside
let's start with the things that are
more they are making more sense to begin
with so we have linked dot weights and
this is the matrix three times five
meaning it should to a dimensionality of
three which is the output and takes five
element which are the element of T the
size of the input X and we have another
one which is being dot bias which is
basically our first column of our theta
matrix so we can call here now theta one
it's going to be torch
captain ate the first term the liars and
the second part in weight and we
concatenate along the second dimension
so I cross the columns and this actually
creates a new tensor so if I display
theta one it's going to be simply the
bias vector for all point 43 Oh point 44
minus zero point twenty two and minus
zero point twenty eight and then there
we have then we have the the rest of the
body the rest of the weights
let's see again in so what's I seen here
so we have something that is called
graduate so let's see what is that Brad
wait and all they are all zeros okay and
link dot grad bias and all these are
zeros as well so the first one what is
called grad wait it's going to be
basically the partial derivative of e
with respect to the tip wait basically
of the this module and grad bias is
going to be the e with respect to the
bias it's in our case if we consider
grad theta 1 this is simply going to be
doorstop cut in dot grad bias stuck in
front of graduate on the second
dimensionality and this is the vector
which we call which is the partial
derivative of the error with respect to
the parameter theta 1 in case those
gradients are not already zero we can
zero them with being zero grad
parameters and if they were not zero now
they are zero for sure we will see later
why we need to zero these parameters and
it's basically connected to the fact
connected to the way we can train a
network if you use a batch or mini batch
gradient descent we have to accumulate
the partial derivatives with respect to
the parameters for several iterations so
these are this is the place where
this partial derivatives are accumulated
and therefore we have to start with a
clean accumulation point so we want to
observe them before starting then so the
first part is done so I think we have
then we have an output which is empty at
the moment because we haven't insert
anything in the module we have a grade
in put which is the derivative of the
output with respect to the input of this
block and we don't have anything at the
output nothing and the impulse on it
doesn't it's empty at the moment and the
last part we haven't covered is the type
which is torched double tensor and this
shows the type of module we are using at
the moment let's create the second
module after the linear we need a
sigmoid function so we have sigmoid
equal and then dot sigmoid let's see
print sigmoid that's a and then a
sigmoid and let's see what's inside
so we have a again way a grant input a
gradient input gradient which is empty
we have a type that is torch double
tensor by default and then also the
output is empty let's see how this Sigma
2 looks like so let's require new plot
and then we can do for example 2 Z equal
torch in space from minus 10 to 10 with
21 points and then we can have new plot
plot Z and sigmoid to which we forward
the design everything and here we have
the sigmoid it goes to 0 basically after
minus 5 for numbers that are lowered
then minus 5 and then it goes it hits
the 1/2 for X equal
zero here and then after X for X greater
than five it approaches the one alright
so let's go on and let's start now with
the forward pass so we can clear the
screen and let's start or work or
forward pass
so let's have our input vector X just
I'll Forge random normal distribution of
size we said n so this is our X and then
we have that a 1 it's equal x by
definition we have that H theta the
hypothesis final hypothesis the output
of our network it's going to be the
Sigma 2 which we forward the dinner
which we forward our input X and this is
gonna be our output let's have a look so
it's three dimension as expected and
those are the values over 3600 point
5500 point 29 let's try to reproduce
these values by ourself so we actually
understand what's going on so we have Z
2 it's equal to theta 1 which is the
concatenation of the bias column with
the weight matrices for for definition
how we saw in the previous lessons
multiplied by what by our input to which
we put on top the plus one so our a 1
hat so we have George hat of George ones
just 1 1 and then we have the a1 and we
concatenate in the first dimension
so z2 is the weighted input and then we
have to apply a sigmoid to the Z tone
let's do this by hand so we have a to
equal Z to clone and then I applied if I
don't clone I'm gonna overwrite its
value so I just want to preserve
whatever it was there so we are gonna
have function function function of the
scalar Zed which simply turn returns a 1
divided by 1 plus what laughs the
exponential of minus Z the scalar it
shows this one and we end a function and
we close the parentheses and let's see
our output
8'o and there you go all point 36 1300
555 10 and all point 29 24 so we had
just seen that our computations are our
network is actually computing what we
expect what we have seen before in the
theory let's perform now the backward
backward pass this is a bit more
elaborated so backward pass over spec
propagation or deck penetration so we
have to define first a loss function
which is going to being in this case the
mean square error criterion so if we
print loss is going to tell us MSE
criterion and then
if we would like to know what it does we
can press question mark then we have n n
dot MSD criterion a description of how
it works
creates a criterion that measures the
mean square error between n elements in
the input X and the output Y so loss of
x and y where X is the input and Y are
the basically targets or the labels is
equal to one above n where n is the
dimensionality of the vector x and y aw
summation of the squared differences we
can also see the division by n can be
avoided if one sets the internal
variable size average to false that's
what we are gonna do because in our in
our previous lectures we we haven't
performed any size average
let's bring the content of loss and we
see it contains a gradient put input
gradient basically a size average that
is true body falls and the output it is
zero to begin with so we are going to
say love's dot size average is actually
false because we don't want that
division so you can check what yes
it did work so let's compute now the
error based on our targets so we have to
define our targets we have Y which is
going to be torture and a front not run
end because the output of the network
goes from 0 to 1 since we have applied a
sigmoid non-linearity so R and of size K
so this is our Y let's see the API of
the the loss function the criterion
basically if we rewrite forward we are
going to need an int
and then we need a large target or label
so we can do e the error for the
specific sample the only sample we have
here it's going to be equal loss forward
which we send the output of the network
patrols which was called the H subscript
theta capital theta and D targets or
labeled Y and here we have E which is
0.35 let's try to compute this by
ourself so as we saw in the description
before the Criterium performs the
difference of 8h theta minus y the
target on rise to the power of two and
then summing all together so there is no
1/2 factor as we saw in the in the
lesson before so we have to actually
keep in mind this this 1/2 missing here
which is going to be coming back of
course when we perform the derivative of
the square right and here we go we have
the exactly same result so we are still
performing forward but we are forwarding
in the criterion to compute the error
which is required in order to perform
the back propagation steps so let's
compute now the derivative partial
derivative of the error with respect to
the output of our network so this is
simply our loss we can write update grad
input and we send H theta which is the
input so we have the same API of the
forward we have input in target and then
the target are the Y and the rebound
so if I print the
in the page as we thought as we said
before since we are using the Haitian
notation it's hey it's going to be
having the same the same dimensionality
of H of theta let's compute this by by
hand so we can see whether it is correct
or not so we if we perform the
derivative of this expression here we
are going to get we had 2 times simply
the difference between H theta and Y and
there you go we have the same numbers -
all point 37 or point 19 and minus one
point 10 so far so good what we are
seeing here in torch is exactly what we
have seen before in the in the in the
theoretical slide let's compute the
error now at the output so we have Delta
- it's going to be in the sigmoid -
which we say to update guard in place
and we send side the input we already
have to send the input which is Z - and
then the output derivative so the
derivative which we find at the output
of this plot which is going to be
actually the input derivative of the
criterion and here we have Delta - which
we of course have an element of 3 it's
going to be a size of thing so let's
verify this is correct so we can see we
start from the edge with a clone and
then we have symbol and then we have a 2
which we do a similar of 1 minus a 2
and there you go so we just we just
apply the formula number three now we
can compute the partial derivative of
the error with respect to the parameters
of the dinner module so to do so in
tours we can do and in accumulate
gradient parameters which provide the
input to the module which is X and the
grad input of the following module or
the grad output of the current monitor
so in this case is Delta 2 and then we
can check what this as done
looks like it hasn't done much so inside
lane we have grad bias and cut wait so
we can recall before get where our zeros
so if I go back up we had the they were
zero so let's see if I print them again
what they are so if you print now theta
there you go so this is the desired the
partial derivative of D this is the
partial derivative of the error with
respect to the capital theta matrix
which is simply the concatenation of the
first column which is the grad grad bias
and the other matrix matrix which is the
grad wait let's verify also that this is
this computation is correct so we can do
so by performing the column vector
multiplication time the row vector so
that we have the partial derivative of
the error with respect to the parameters
of the linear model let's write down so
it's more easy 2x
so we have Delta to to which I'm
specifying which I'd like to view is a
column vector so it's gonna have so it's
going to be as many rolls 18 it and just
one columns to which I multiply the
concatenation of a simply a 1 and my
eggs on the only dimension and then this
instead of drag to view as a one row and
how many columns it cares and there we
go we got the same result
nice let's go one step further and let's
compute the partial derivative of the
error function with respect to the input
of this module which is actually the
global input it's not useful for
training but it may have other utilities
for example if we have multiple layers
we are gonna need that value in order to
compute the Delta at the previous layers
so let's just call this a sling grad
input which is not Delta 1 because Delta
1 we have also D we take a cut in
account also the non-linearity in this
case we don't have yet they not
non-linearity there is no non-linearity
at all so it's simply going to be the
partial derivative of the error function
with respect to the input of the linear
module this is equal to simply linear
updated grad input which I provide the
input and the Delta the partial
derivative at the output so grad output
basically it's called and this one
it's going to be of the same
dimensionality of the input which is
which was five this is still because we
are using the Haitian notation or the
denominator layout and let's verify
these numbers are also correct
so as we recall from equation five maybe
so also today before we can compute this
by using the matrix lean dot weight
transpose to which I multiply to each
which is applied to the Delta two and
there we go so we just verified it all
the computations are correct and we
obtain consistent results it's it's
pretty nice we input so far so many
commands and it looks like quite
daunting to follow Kali has been done so
far this is because we haven't actually
used properly the NN package the
inanimate provides us also an odd other
other immunities which make the whole
training a breeze and especially in the
forward and back propagations i can be
run in just two lines of code here we
have typed down all the steps just in
order to verify the correctness of these
steps and now we can instead see how we
can type all these in a much more
compact way but which will hide of
course some intuition and some gotchas
that will actually strengthen our
understanding of the algorithm and how
the package works so let's define a
network is been a container which is
called sequential
sequential simply allows us to have a
sequential series of blocks one after
each other where all the forward steps
are performed automatically so the
output of the first block is sent to the
second block to the third to the fourth
and etc and when we perform the Grad the
input steps all the guard inputs go from
the last to the first block so we can
add net work in your module then we can
add also our sigmoid module and then we
can print the network and it will show
us that we have a simply a sequential we
have an input which is sent to the first
module which is sent to the second
module which is basically the output of
the network to perform the whole forward
pass we can write simply read which is
our prediction of the network which is
our basically which is our age of theta
with the find apartment we call it now
prediction so we don't overwrite our
values that we have written before it's
simply a net word-for-word of variable
input X so now we can see a prediction
is this and if we check H capital theta
it's exactly the same so that's perfect
we saw just with one line we can compute
the output of the network we can
complete the error --air equal loss
which we forward our prediction and the
correct liquid wine so are equal this
one and we can have the previous e it's
the same so so far so good
and now the equal part so we can compute
grab criterion
input laws and backward to which we send
again red and white so we can compare
Brad criterion with D and D H which is
the same before proceeding with the
backward step for the network we have to
actually clear out the grad bias and
wait so now that we have a network I can
have netiquette first and we get the
first module within the network so if
I'd like to see what are the partial
derivative of T
error with respect to the weight I can
print them here as this night gate 1
grad bias concatenated to NAND gate 1
dot grad wait a second
along the second dimension so these were
the numbers we got before so we have to
zero them otherwise they will sum to
dicen so we can simply do this by typing
net zero
grad parameters so if we call again the
same function we had all zeros as lovely
we have net backward to which we input
our input X and the Brad criterion and
this one has perform a bunch of things I
think the last thing that has performed
we have this minus zero point zero nine
five eight so this is exactly the Lin
grad input we have computed before you
can see minus zero point zero nine five
eight zero 13
nine zero zero eight two six and they go
down we said we had the same numbers
here so when we perform a backward step
return as the input gradient of the
current network module whatever we are
using let's call it this guy again so we
compare these numbers and we can see we
have minus 0.08 six zero minus 0.06 one
five zero second line zero zero four six
one third 9-0 2284 that's perfect so we
have we computed the the gradient of the
error with respect to the parameters all
the parameters of the linear module
which are the only parameters in this
case with just one instruction that was
the network : backward to which we send
Eckstein put of the network and the
gradient of the error with respect to
the output of the network the age of
capitals data so let's see how the
parameters are updated so to update the
parameters I can define a red learning
rate alpha equals 0.01 for example and
then I can say Network
listen update your parameters based on
the learning rate ether bomb finish the
network has updated its own parameters
based on the current parameters and the
gradient of the error with respect to
deep parameters let's check that that is
actually what had happened in reality we
can have D we think we have the respect
to H no so the e in d theta one it's
equal this thing here above right and
then we have
theta 1 minus alpha which multiplies the
theta the theta 1 so this one is this
one and let's see there we go
theta let's called Neil and let's print
Daniel and we have what we expected so
the new parameters we where we have here
the first is the column relative to the
bias and here it's the instead the
matrix the new the new matrix theta one
it's exactly what we expected so this is
actually computing this has been
computed by torch because we actually
perform now we just check inside the
internals of the Dean so linear dot bias
and in a that way to be heard them
concatenated across the 2nd dimension so
we have the first one here leading bias
and then there is the matrix here lean
weight which has been computed as the
previous theta and the full matrix here
which includes the bias terms the bias
vector minus
Etha to perform the gradient descent
step multiplied by the de in the theta
one
finally let's see how we can train a
system with all we have learned so far
so now we are confident that the torch
works and the library nn it works
magically it does require really few
instructions in order to compute the
forward and backward back propagation
basically there are two instructions one
for forward one for backwards and then
we have the two zero sometimes the
parameters the Grad parameters so let's
have an a full and an example of working
example or almost working example so
let's see how to chain a full system
with a script right hmm we have training
so we have our X it's going to be our
design matrix and then we have our Y
it's going to be our divorce or targets
it should like matrix or vector the
first one has size n times N and the
second one is going to be of size M
examples times capital K all right
control you to actually clean the line
that's cool um so we're gonna have again
a for loop which is going which is going
to go for from the first example so
number 1 to the M sample perform what so
we are going to have our prediction
which is equal to the network we have
defined forward to which we send the I
example of the design matrix then we can
have a again a local error which is
going to be equal at Project us forward
or work which we provide the prediction
and the label of the ice example
and then we can have the grad loss which
is going to be those blackboard backward
of what we had to send the same
prediction and why and then we have now
to zero we had to zero the accumulated
parameters so this is in this case we
are talking here of this stochastic
gradient descent in this case so we have
to zero every time zero
Brad parameters and then we have network
report to which we provide the input so
it's going to be capital X of I and then
we have to drag grad loss and then the
last one is gonna be in network to which
we draw say update majors because again
we provide an update every time we
provide a new example in the case of
stochastic purely stochastic gradient
descent plate parameters and here we
have our Etha the learning rate and this
is the full example work an example for
applying stochastic gradient descent
given that we have a design matrix X and
a design label matrix Y Capital y and
capital X if instead we would like to
perform mini-batch gradient descent this
is a little bit more code right but it
performs better in terms of convergence
and in terms of speed up
if we use multi dimensional input which
I'm not going to illustrate right now
but mini-batches provide some specific
advantage both computationally and also
optimization wise so we are going to
have here four I again it goes from one
to the number of examples we have but
jump with vege size so in this case if
we go like from one to 1024 with page
size of 128 we are going to have high
equal one 129 257 and etc up to the last
value so we can do we start by zeroing
the parameters and then we do for J
equal goes from zero to the - sighs - -
so at the beginning I the summation i p+
j is equal to 1 and for the last is gone
for the last value of j is going to be
batch size minus 1 actually
plus one batch size this is gonna be
about size minus 1 there's gonna be
batch size so it goes from one 228 and
the second term is going to be 129 to
256 MB - 60 to 57 and so on 2d whatever
is the output job then we have if we
exceed D index M so if I plus J it's
greater than our index M then please
don't do anything otherwise the same
thing of the forest so
prediction internet forward then we are
going to have local error and like
before and then we have again local grad
loss and then we follow with net
backwards and then we end it here after
when we finish to probe to process the
batch we can actually update parameters
based on being cut with a learning rate
and then we are actually done yeah
last part it's actually so we actually
don't have to write all this code
neither so torch allows us to write even
this code so that's why it gives us so
many very nice treat let's say this way
so the treat is basically if I have I
have to provide a data set to start with
so we have a data set which can be a
empty table at the beginning to do it
and then this table has to provide a
size which simply returns M the size of
the data set so so far nothing too
complicated and then we have for I equal
1 to the size of the data set please do
populate data set so element I it's
going to be our table
of first element is going to be the
example and the second element is going
to be a t-table and and here so once we
have this kind of data set which returns
a size is required and then each each
item is the table of example and of the
input and the target we can simply
create a trainer so we have local
trainer like a personal trainer a local
trainer yeah funny right so this is
going to be a stochastic gradient
trainer which we send the model and we
send loss we decided to use and all we
really need to do is say a trainer
please train my network on the data set
I'd like so basically to train your
network you just read really nice green
those two lines of code of course sure
you have to specify a network and a
dysfunction and of course a data set
otherwise you wouldn't be training on
anything so sure you have a necessity of
defining a data set a network loss
function which you know Suites your
needs and then you create a trainer
which is going to be training your
network in just one line we can see a
full example of how to train a network
with the trainer the stochastic gradient
from torch at this address let's have a
look so here we have the machine
learning with torch repository in
specific the first topic its regression
with MLP
multi-layered perception which is simple
simply a neuron fancy way of calling a
neural network usually a multi-layer
perceptron and Makepeace are used for
pattern recognition classification tasks
in the field of image and speech
recognition nevertheless they can be
effectively used for regression check
out MLP regression section to find more
to find out more about it so you can go
here and then you can see what I'm
trying to do in this tutorial so
basically I will explain how to perform
regression with a neural network so
while I was reading the bishops book
pattern recognition and machine learning
I got to the point in which a three
layer multi-layer perceptron with only
three hidden neurons and one output
linear neuron so for neurons you know
overall was used to regress seeming
lessly some contiguous functions and
here is the image so we have the
illustration of the capability of a
multi-layer perceptron to approach a
proximate for different function a
comprising f of X equal x square f of X
equals sine of X f of X equal absolute
value of X and last one is the F equal H
of X for egg H is the step function or
the Heaviside step function in each case
and equal 15 data points are the
x-coordinates shown as blue dot have
been sample uniformly in x over the
interval from minus 1 to plus 1 and the
corresponding values of F X evaluated so
f of X are the Y targets or labels and
the X are simply the input one
dimensional scalar and also the output
is a scalar so we have size of the input
layer it's one size of the output layer
it's also one and then we have three
neurons which are the inter
hidden neurons these data points are
then used to train a two layer neural
network having three hidden units with a
tonnage which is similar to the sigmoid
that goes from minus 1 to plus 1 instead
activation function and linear output
units so there is no non-linearity after
the last layer the resulting network
function are shown by the red curves
here in the picture
and the output of the three hidden
hidden units are shown by the 3 dashed
curves and so just go through the just
go through the tutorial and then there
is the algorithm where I explain how I
use the trainer for training on the data
set and I highly recommend to click here
on the source regression and actually
play with the code so read the
description and use it interactively so
change some values and play around to
understand better how it works you have
different modes just go through it
carefully and you will get a lot of
intuition and understanding I believe
and that's the end of the overview of
the NN package we will see more features
in the next podcast
stay tuned have a good night bye a good
night because yes
it's night now I'm working at night
night