Practical 3.0 – CNN basics Video Lecture - Torch A Practical Hands On Tutorial

in this chapter we are going to
introduce the convolutional neural
networks or cnn's
specifically for image understanding the
concept here Illustrated can be easily
extended to different kind of input like
audio or video all it requires is going
to be a slightly variation in the type
of convolution that is going to be
utilized we will start with the
rationale and motivation why are
convolutional neural network needed in
order to process images we will see the
the fundamental aspects of a
convolutional neural network is sparsity
which exploits locality in the input
data and specifically we will introduce
the receptive field and the hierarchical
view of deep models and the second main
important aspect of convolutional neural
network is that they perform parameters
sharing which is based on the assumption
of homogeneity of the input data we will
see then how a convolutional layer will
perform a 3d convolution in the case of
image processing then we will talk about
the non-linearity layer and the decaying
learning speed across the network so we
will revise the logistic sigmoid and we
will introduce the rectifying linear
unit moreover we will introduce the LP
pooling layer where two specific cases
are the average pooling and the max
poling then again we see the rationale
and we will have some conclusions where
we will show what are the benefits of
using convolutional layers and pooling
layers then we will switch to the shell
and the command line and we will install
the pretty NN package in order to have
color coded network architectures which
is going to ease our understanding of
the models and will be fundamental for
the next practical and finally we will
introduce the ELab
seven profiling tool it's a tool I wrote
around October 2014 and now has more
than 200 commits from all the members of
a lab and others with which we will
profile a convolutional neural network
in torch and compared the results we
have computed before manually in the
field of image understanding let's find
out why convolutional neural network had
had a major role but let's start with a
neural network which goes from let's say
an neuron in input to three hidden
layers of size twice as the input so 2n
2n 2n and then capital K output let's
compute the number of operations that a
similar network may require if use for
image processing let's say we start with
an image which is a color image so we
have three planes RGB and which are of
size 256 x 256 and then in that we have
the three colour channels RGB so in this
case n is going to be equal to three
times 256 x 256 which I can write as 3
times 2 to the power of 8 and 2 to the
power of 8 which is 3 times 2 to the
power of 16 overall we can count here 24
we can have 2 4 6 7 n so we have 7 n
plus K neurons which in this case is
going to be
seven times three times two to the power
of 16 which is roughly 1.4 million we
have to go from the first layer to the
second layer we are going to use a
matrix which has height 2n and with n
then here we are going to have a square
matrix to end to end another square
matrix here to end to end and then here
something like okay and one so if I
count how many multiplications and
additions we are gonna have so we are
going to have 2 n square plus 2 times 4
n square plus 2 capital K and so we can
neglect this term here we have overall 2
times 4 8 plus 2 roughly roughly 10 and
square which is equal to what 2 times 5
times 2 to the power of 32 and 3 to the
power of 2 so this is actually is going
2 to the power of 33 and this is roughly
390
yeah multiplication max Mac which is
multiplication and accumulation let's
say we are on my macbook pro which i
which has a intel core i7
which runs at 2.2 gigahertz so we can
see how much how much time is gonna take
to process one image which is color
image of 256 x 256 pixels so we have the
390 yeah
/ - 0.2 gigahertz which gives us 118
second which is roughly 3 minutes so we
have here seen that to process this
image here is gonna take us three
minutes just to go to send forward one
image think about the training where we
are going to use several images in both
directions forward and back propagation
this is really impossible to do all this
way so what can we do we can be smart
and we can reduce the number of
computations by exploiting the
underlying distribution of our data so
if we actually think about what we are
trying to achieve here and we are not
just blindly applying a neuron
to whatever problem it is we see that
our images are heading our images have
very distinct specific characteristics
given that they are natural images
images they are not random noise so the
two main points we want to keep in mind
images are highly specially correlated
means if two points on these screens are
closed nearby it's likely that they will
be that we will have the same color most
of most of the time they will be black
in this case but if you take two points
close to some natural images both those
values will be very close in value and
moreover in images objects are made of
parts
let's see how to exploit these two
particular aspects of images in order to
reduce the amount of computation and
therefore speed up the system so the
first thing that it's very important to
keep in mind when we speak about
convolution on your network is that they
introduce sparsity here we can see that
we have a fully connected layer L minus
1 which is connected to the layer L with
all the connections as we have seen
moreover that will be d plus 1 for the
bias here on top which is connected to
the three neurons on the right hand side
of layer L the same here we are going to
have the bias on top which is connected
to the single single numeral of layer l
plus 1 so how can we exploit this
correlation in order to reduce the
number of computations we can say that
for example the first neuron here can
simply see a subgroup of the input
because input pixels nearby will have
some kind of will carry some kind of
information relative to the object or to
the location where they are coming from
but it's very it's quite unlikely that
pixels that are far away will have some
kind of contribution for characterizing
the specific location at which the first
neuron may be looking at so sparsity
basically means that the first neuron of
the layer L is gonna just look at this
neuron here the second you are over here
and the Newark cover here and of course
there will be also on the bias
for the same way the center you will
just see this guy here and this way and
again we have the plus one for the bias
analogously we have the three
connections for the last one in this
specifically neural network we say that
this name this layer here will have a
receptive field RF of 3 that means that
will see just three neurons from its
previous layer let's move on and let's
connect also on the layer L to the layer
l plus 1 so we are going to have first
connection second connection sir
connection also here the receptive field
is 3 but the receptive field with
respect to D layer L minus 1 is going to
be 5 why is that because the neuron add
the output layer can actually see 5
neurons from the input space so in this
case we have that the neurons at the
intermediate layer the red one we just
see a portion of the input space and
therefore will have the notion of edge
the neuron afterwards in in the higher
as we go high in the heart high arcing
will see more pixels and well and will
be able to have a better understanding
of what's going on in the input image
because they can combine the response of
neurons that are in the intermediate
level which builds up to generate a more
strike in a high order representation we
can draw here therefore here in this
direction a global view direction
eventually the last neuron of this
network will be able to see the whole
input image and will be able to
distinguish between different objects
that we are going to propose to the
network for example for a classification
task as we can see here the number of
computations is being greatly reduced we
will see later a numerical example that
is going to give us some better
understanding of the order of magnitude
of reduction so we said once more
sparsity give us a reduction in
computations
by exploiting
local correlation
of the data
the second main principle on which
convolutional neural network based their
strength is the parameter sharing so the
first one we said it was the reduction
of computation due to sparsity and this
one is that it's practice sharing which
is going to ease the convergence of the
training of the model how is this done
how does it work so we see basically
that each of these neurons in the red
layer will share all the weights for the
same kind of connection so we can
represent this by drawing here for
example in the purple this connection
here then for example in blue we have
this one and then we have the last one
in pink finally we don't want we don't
want to forget our bias of course which
is the same which is gonna is gonna have
the same parameter associated with in
this case each of these neurons are
looking for the same kind of feature in
different regions of the input as we saw
when we perform the computation of a 1
here so a 1 is going to be just this
partial input here we can call it X for
the moment
so we are gonna have a projection of the
eggs for dinner on one so we can put a
one here so in this case we project the
first part of the input on the specific
feature theta which is share across the
whole the neurons in the hidden layers
and each of these neurons going to look
at the different region of the input
space to which is gonna still look for
the same kind of target same kind of
activation the same kind of kernel we
can call so what we are gonna do
basically it's the projection of portion
of the input with respect to the same
specific parameter vector which is a
kind of feature we are gonna train our
system on basically and then we are
going to take of course our
non-linearity in order to boost the
correct result and to attenuate the
opposite parameter sharing allows us to
reduce the number of parameters which in
turn helps convergence in terms of time
and in terms of relative error
less time and less error so let's see
how the convolutional layer actually
works
so the normal K is the generic case we
are gonna have an input that I'm gonna
call just X which is a 3d tensor of
dimensionality and one 2d maps and two
times n 3 and 1 maps basically or
channels and these maps have height and
2 and with n 3 for example if we are in
the input so if we are actually talking
about the real input so image input so
in this case and one could be 3 which
are these three RG and B channels and
the n 2 and n 3 are going to be the
height and the width of our image for
example in the case we saw before it
could be for example 256 x 256 height
and width then our output instead why
it's also going to be a 3 D tensor of
dimensionality M 1 maps so we have M 1
2d maps of dimension M 2 times M 3 and
how do we get these maps so we also have
a collection of kernels we call them K
which is for the tensor in this case and
it's made of M 1 3d and one times
p-1 times p-2 kernels k okay i finally
we have a term B and this case is a
vector so it's 1d and this is my M one
of dimensionality m1 bias turn so how do
we compute Y we said we had M 1 to d m2
x m3 y I feature maps so each feature
map it's m2 height x m3 a bit so we have
a 3d input which is N 1 maps or channels
of height + 2 x + 3 then the output is
going to be an again a 3 D tensor of m1
channels of maps of height M 2 times M 3
then we are going to use a collection of
m1 3d kernels of dimensionality and 1 as
the input times p-1 times p-2 kernels ki
and I here cap goes goes from 1 to M 1
and then at the end we'll have also a
term bias 1 dimensional which is the
same size of the number of channels of
the feature maps Y M which is the bias
bias term now you can see how to compute
these feature names we have that each
and every feature map y:i is going to be
equal x convolved with each of these ki
and - which then I sum my scalar VI and
we have this one for I that is going to
be equal 1 up to M 1 and we had this guy
over here this is the convolution symbol
and basically what this equation says
here is that we project our input X on
two different subspaces ki and then we
shift them by a specific bias term it's
very very similar to how we were
computing the weighted input Z which was
equal to
my theta matrix times my activation or X
if it's in the first day here
so as we said here we have that our each
of our feature maps why I is going to be
required the convolution between the
current input X which is the activation
basically from the previous layer
convo convolved with the kernel each of
the kernel m1 Kerner's for the specific
layer ki and then we shift by a specific
bias bi so how does it work this one is
a 3d convolution which can be expressed
this way so if we have for example a
image or even feature map F or variables
L m and and we convolve with a G and M
and it's going to be is going to be
equal the summation over all the
variables you be in the value of F in u
v w x the kernel which is flipped and
transposed as we sum so we have n minus
0 m minus V and n minus W if this is
still very cryptic I'm gonna draw
something perhaps it's gonna be more
easy to understand afterwards so I have
here my X
which we said it's and one here and two
here and and three here inside we have
our colonel
could be for example 3x3 and this guy as
we see from this equation he is going to
move this direction in this direction
from where it is for example in this
first location it's gonna be outputting
my first value is a 1x1 pixel so this is
1 1 1
ok so then it's going to move across
towards the right and down so at the end
I'm going to have something that is like
this and maybe cuz also down here just
one here in the depth dimension is going
to be of height m2 and with 3 and this
stuff is repeated times m1 times so that
we can end up having our final output Y
with m1 maps and then the height is
going to be M 2 and with empty
you can be recalled from the classic and
neural network case after we compute Z
from the linear operation on where we
have the matrix multiplied with the
activation from the previous layer then
we are going to forward and with
computing the new activation as the
nonlinear function applied to our way to
the input in this case in our case our
way to the input is called feature map
and the projection is done with
convolution we have not applied the
nonlinear function to our feature map
all our weighted input we can start by
having a look to the nonlinear function
we already seen in the previous lesson
so here we can draw for example Sigma of
Z and here we have Z and if this is one
it's gonna be half roughly if it is
gonna be 5 and if there's gonna be minus
5 we are gonna have something like and
then it goes
this way and the problem that we have
with the Sigma is that if I compute the
Sigma prime inside we have that if this
was one this is going to be 1/2 this is
going to be 1/4 still the same size same
x-axis -5 5 we are going to have
something that is kind of very annoying
as we can remember every term Delta in
our neural network was multiplied by
something here was multiplied by the
derivative of my activation so every
Delta every time we go in layer after
layer it's gonna be multiplied by
something that is at most one 1/4 in
when the Z is close to the origin and
otherwise just goes to 0 so this creates
some problems because the training
basically slows down across the network
so the layer more close to the cost
function can actually change their
values quite accordingly to what
gradient descent says and establish but
the layers in the the lower layer the
one closer to the input so the one with
less abstraction will have some gradient
that are basically very very very small
and therefore the Grad gradient with
respect to the parameters is gonna be
also very small and therefore the
learning appears to be slowed down
incredibly far for the early layers so
since we have to learn here networks
that are quite longer than the they are
called deep neural network because they
are deep they are deep in length meaning
there are many layers stacked one after
each other
we cannot have this problem of learning
that slows down so much due to this
particular effect so we can try to
remove this effect by changing our
non-linearity and we can use this one
the rectified linear unit as a
definition
actually this is defined simply as z+
which is the positive part of Z so it's
completely trivial and let's say this is
1 and this is 1 so notice that I've
changed the X scale between these two
charts from the left and right hand side
for the derivative we are going to have
of course it's not therefore
differentiable in Z equals 0 but we
don't really care too much so if this is
1 we are going to have simply 1 here and
0 and we can define that in stays here
in 0 at this point in this case we can
see that the problem of having the
derivative very low was well it's not
going to happen here and we don't have
this slowing down across the network due
to the non-linearity this technique was
introduced in the paper of Alex
krajewski and Geoffrey Hinton when they
submitted their the first deep neural
network in 2012 for the imagenet
challenge and the writing in their paper
they show that rectifying linear unit
helps a lot speeding up the training the
last layer used by convolutional neural
network is the pooling layer so what
does this layer do and how it works
let's go in how it works and then what
it does we explain what it does so let's
think of having here a piece of feature
map one of the m1 slices
sample coming out from the convolutional
layer that is going after the point was
nonlinear function and here we may think
it for example this first region here
and compute the LP norm
so that we end up with our first output
here then we apply for example astride
equal to the size to the width of this
running window so our second window is
going to be in this position here and
then it's going to perform again the LP
norm which is gonna give me my second
value here and so on so I can keep going
just for clarity if we would like to
compute the P norm of vector this is
going to be equal to D P norm
P root of this summation across all
components of X to the power of P
moreover if P DP norm tends to max of
the components of X when P tends to plus
infinity max pooling can be simply seen
as a specific case of the most more
generic LP pooling what does it do
basically if we see in the
dimensionality if you apply a stride
equal to the size of the kernel here the
running window we are going to obtain an
output which is shrinked in size so if
we started with m2 in height and m3 in
width we are going to get
for example m3 half and m2 half the
number of channels is still m1 so the
pulling operation works across the
spatial dimensionality and reduce the
number of neurons by dropping them if we
choose for example the max operator and
therefore just the maximum activations
are carried across the network and are
gonna be then used for later on for
later computations sometimes the l2 norm
is used instead to average out all the
activations - instead having higher
robustness for noise so there are
different cases which require different
kind of norm different kind of p-value
so for this reason I presented here the
generic pooling LP norm pooling rather
than just the max pooling which is most
commonly I think used now that we have
seen what are the main blocks of a
convolutional neural network we can see
how the initial example where I was
computing the number of computations and
the time to process one image can change
if we do apply if you do use
convolutions we can start here by
drawing our input which was 256 256 and
3 in depth our input image we said we'd
like to have twice as many neurons so
twice as many pixels in the feature map
if we can say they are not exactly
pictures of their activations
activations in the feature map so in the
input space we can call each point of
this image a pixel
in later layer in the feature map they
will be called activations instead so we
said to n therefore let's make it one so
we still have 256 256 let's go oh six
here steel to n so again six 256 256
then here once more and then the last
one is going to be simply K K vector so
to go from the first block to the second
block let's say our kernels K ki are 5
by 5 we said there are three kinds they
can be three if we are in the first one
or six if you are in the other cases
here so in this first point we are going
to have if I count the number of
computations of Max we're gonna have 256
x 256 times 6 that is the number of
activations
which came from a convolution of a
kernel that is in this case three times
five times five so this was the first
number of computations so for the second
one is going to be the same but the only
thing that changes is going to be the
string that becomes six then again from
here to here who's gonna be out began
six so I can say this one x we said we
have two times two times one time so we
are gonna have you can write 1 plus 2
plus 2 multiplier so this is gonna be 5
and overall this is going to be equal
247 million
max for the final part we have that we
start we have a matrix that is of height
K and the width is going to be 256 x 256
times 6 which gives us K times 6 times
256 x 256 just going to be K multiplied
by 3 93,000 if we think about the
imagenet classification we are gonna
have that K is equal 1,000 therefore
this is going to be equal 393 million
overall we have five hundred and forty
million Mac
and if we divide by two point two
gigahertz of my nice MacBook Pro we get
something like 250 milliseconds per
image so we went down from three minutes
to Quatro second which in my opinion is
pretty neat so we would like to keep in
mind these numbers here can we do better
yes we can and we can use the pulling
operation which is going to be speeding
up our system by at least 15 times so we
said we start with a cube a data of 256
x 256 and 3 in depth and for example we
can apply now convolution with a stride
so instead of applying the kernel every
pixel and moving it on the horizonte
line or elegantly and vertically by one
pixel we can move it for example by 2
pixels so in this case we are gonna have
half half less pixels for in the
x-direction and half a slice pixel in
the y-direction
so we will end up with 128 pixel here
and 128 pixel here and we said we were
going to double the size in depth so
again we are going to have 6 here so
with this configuration see how many
computations is gonna take so we have
128 x 128 x 6 which is the number of
activations and then times my kernel
which is going to be 5 times 5 times 3
and this one is roughly 7 medium and 300
max and this first was come
Ellucian with stripe then we are going
to apply the second convolution six in
depth 128 in X and the 28 in Y and again
we said six here and this guy is gonna
be the same it's going to be the same
because we may use padding so this is
not the valid conclusion but it is same
convolution if you use the MATLAB kind
of jargon so in this case we don't have
to subtract let's say four from the size
of the height and width
since the kernel is five should have
been from 128 should we go should have
gone down 224 button is complicated to
actually have nice numbers so we can
just pad and then we have with zero
padding we're gonna keep consistent the
dimensionality and we don't have to
worry too much let's see how much this
convolution takes so we are gonna have
128 x 128 times 6 and then we have our
kernel which is going to have five by
five by six actually six by five by five
fourteen point seven million Mac then
let's say we apply pulling so we may
start with 600 2828 we end up with six
64 64 the number of computation is going
to be 64 times 64 times six times our
kernel is going to be for example two by
two in this case and this guy it's 98
thousand Mac so it's very small compared
to the amount of computer
that the convolutional layer takes then
we have the last convolution so we're
gonna have 6 64 64 6 64 64 and therefore
the output is gonna be 64 times 64 times
6 times the kernel so it's going to be 5
times 5 and 6 and this is 3.7 million
and let's even apply one more pudding
here so we go from 6 64 64 206 632 32
exercise let's compute the last one 32
times 32 times 6 times 2 times 2 which
is 25 thousand max at the end we are
gonna have the last block which is 6 32
32 and this one goes to be a linear
layer because is going to be the kasev
classifying my K output here I haven't
computed the nonlinear functions but the
nonlinear function they are simply the
number of neurons activations in the
output layer in the first case is gonna
be six times hundred 28 times hundred 28
which is 98 thousand so here we have 98
thousand of non-linearity mac and we can
see it's small enough that we don't care
i compared to the million of operations
so the final part we have 30 talk times
32 times 6 which is equal to 6 thousand
and the final matrix matrix is going to
be K and 6000
and if K equals 1000 for the image net
for example we are going to end up with
six million Mac if we take the summation
of all the convolutions we are going to
have that disease adds up to twenty five
point seven four convolution and the
last one we said it was six million for
the linear part so overall we end up
having twenty five point seven plus six
it's going to be 31 and seven million
max divided by two point two gigahertz
you're gonna get 14 milliseconds which
is roughly seventeen times less than the
previous number we would like now to
test and verify the numbers we just
computed and just to be sure it they are
reliable and and let's see also how much
time how long our computer my computer
actually will take in order to process
an input image in these networks that we
have just shown before before starting
with the the networks let's set up our
system with some new packages to
Prague's install creating an end
moreover we'd like to get cloned the
torch seven profiling repository from
the e-type website
and there we go so we can see get into
towards providing and in here we see
there is the profile model to up we can
do torch profile model - ah - each we're
gonna have a help display it shows us
how to use this file
moreover it shows an example so we're
gonna call th profile model the name of
the modern-day where we are going to
define our model and then we are going
to send the specific resolution and
moreover we are going to compute max
instead of both addition and
multiplication multiplications let's
create the model from out for now our
network so we can go on inside the
folder model models and we can create
our modified so let's call it three
convolution and doctored war so since I
don't know how to do this by heart
we are gonna just use the interpreter
here below so we can require and then at
the beginning and then we have we also
require pretty and then so it true works
so we have net equal n n dot sequential
and then we have our first layer so
let's copy here okay so let's go here
we're gonna write our first layer so we
go out net add what a new layer and then
we want a special convolution which goes
from three because we were starting from
input image of three channels 206
channels using a kernel or five by five
with a stride of one both directions
and let's put padding off to both sides
or to the left to the right top and
bottom so that the dimensionality of the
output is going to be the same dimension
of the input then we gonna add our
prelude let's copy these two guys here
so we have first one and second one is
gonna be my first alright so let's go
with a second layer we can to just go up
it's gonna be from 6 channel to again 6
channel 5 5 no stride parting it's ok so
ok then add another radio let's add
these two guys and this one and then
again one more we set right so this one
is exactly the same and then radio and
now we can add we have to reshape and
view differently the cubic feature map
into simply one long vector so we can do
this by using the view
module to which we say just resize it in
1 1 1 only one only one chunk home so if
we print now net we see here it is a
sequential
you can see there are seven models in
this equation there are three
convolutions the first one was through
three channels to six channels with a
kernel size of five by five striding of
one in both direction and adding some
padding of two pixel per each boundary
the second module is going to be non non
linearity and a rectifying leaner unit
then we have another special convolution
which goes again from six to six kernel
size five by five stride one and
padding-top so we can preserve the
dimensionality again another nonlinear
function the last convolution six two
six five five one one in two to four the
padding last nonlinear function and then
the to view the 3d volume into one long
shot so I have to add now the last
linear layer since I don't know was the
size instead of computing by hand I can
do a check so I can do let's say X is my
input so it's my torch tensor of
dimensionality 3 because it's RGB then
we set 256 256 so if I ask what is the
size when I forward to the network world
X sweet this is dimension the
dimensionality is 3393 whatever so we
came to the net which goes from this
dimensionality to let's say 1000 classes
for the image and competition for
example so if I print the full network
again here it
sometimes you can tell it's quite big as
a matrix if I bring my whole network
here we have it's a sequential with 8
the years and the last one is a dinner
the last nonlinear function is not
required actually it speeds even
training without the last trusted linear
function this is always there and this
is my output here we actually need to do
we need to add I forgot 0 minus 1
as we said before we were estimating 147
million max for the convolutional part
and here we have 149 so I think it's
pretty reasonable median then for the
fully connected part the dinner layer we
were estimating 393 million and we have
here 393 million which is also quite
reliable so overall we had 540 million
and here we have five hundred
forty-three million so it's very very
close to the computations we have made
and overall we were estimating the CPU
can perform one make per clock cycle so
we were dividing 540 by 2.2 we were
getting 250 250 milliseconds here
instead we get the forward took quite
half of the amount is hundred and twenty
nine milliseconds and we have that this
CPU my computer performed 4.2 Giga max
per per second so we have basically two
max per clock cycle and otherwise all
the numbers are consistent
let's make now a test for the other
network or the one with them pulling and
let's see how it changes so let's go
back to our model here so let's save
these as three com+ pudding one this is
my personal plus four so let's put a
polling layer after this convolution and
after here and let's change the first
convolution instead of having a stride
of one let's have a stride of two into
so we have here to modify this last
layer so let's see how to change it so
we can do torch neck equals a coin
models.com the hole and it's like cool
so if we print Network we are gonna have
this architecture where we can see we
had a convolution with there goes from 3
to 6 with 5x5 kernel straight off 22 n
padding to n 2 that is basically for for
both directions then we have the
non-linearity and the new convolution
six to six five five one one two two
non-linearity and then the pooling again
one similar chunk with the convolution
non-linearity important then there is
the view so you've met forward
George and sir run and end of three 256
256 and then we actually asked the size
of Busan we have the new number here and
we can go here and replace these tips so
this is f3 called fold down below let's
make it fullscreen and we can call like
this one instead of the count we have
three count for everything ice should be
the same
let's go BAM it was quite fast this time
right so let's compare this come let's
check some results
so at the beginning we were saying that
we were getting the first convolution we
were estimating 7.3 million and here is
7.5 it's pretty close then we had 14
point 7 and here we have 14 point 8
we actually forgot to take in account
the bias terms and which is adding some
little overhead then we were saying that
the the first non-linearity would have
taken ninety eight thousand and ninety
eight thousand eighty s also the polling
is taking ninety eight thousand and
ninety eight thousand it is as well
following we had another convolutional
layer with 3.7 million and here we have
3.7 million finally we have the linear
layer was taking six million and six
million it is here we can even see in
the last column the percentage of the
computation which is very important it
should be always theoretically
distributed across the whole network in
order to have a more balanced
distribution of operations and also of
processing of the information if we go
down below we can see that the
convolutional part takes 26 million and
we were estimating twenty five and seven
so I guess it's correct overall there is
a total of 33 medium and we were
estimating 32 divided by two point two
key guards we were assuming one mac per
clock cycle we were estimating 14
milliseconds and my computer just
finished running the profiling in 11
milliseconds and this is pretty I would
say accurate estimate we caught on paper
so far
now we could hypothetically write down
the network that is using only linear
layers but we can see that it's code is
not gonna even be loading in main into
memory but just for exercise we can do
this we can do linear for example and we
have my huge MLP multi-layer perceptron
or a classic and neural network so we
can write a local and equal 256 x 256
times 3 and then we have our local net
equal and sequential and then we have
the first layer we were saying net pad
we add an end of linear which goes from
n to 1 and then we are gonna do net add
and then the trade-off
then we have the second layer which is
very similar to the previous one but
here we have from 2 & 5 and the third
layer which is going to be identical in
this case third layer and then the
output
which is going to be net add linear of
the wine and there we go into the output
which is 1000 and then at the end we
have the turn but this one won't even
roll into my computer's memory so we are
free to try it yourself
I want already crashed my computer
previously so I want to hit again and
that's it for today stay tuned because
in the next video we will see some
pretty standard architectures from the
first one like Dylan net and the Alex
net up to the most recent Google net and
resonate and why they the some specific
architectures have been chosen and what
are their good points and what we can
learn from this