Practical 3.2 – CNN models Video Lecture - Torch A Practical Hands On Tutorial

in this practical way we go through very
famous architectures for convolutional
neural network that have contributed
substantially to the state of the art in
image recognition we will start with the
net which was trained on the modified
nice data set of 60,000 images
representing numbers from 0 to 9 of size
28 pixels by 28 pixels then we will
introduce the image net challenge and
how it has incentive aided the
submission of very powerful
convolutional neural network models we
will start with alex net then we will
move on to the Google net which
introduces the inception architecture
and we will conclude with the rest net
which introduced the bypass connection
Lynnette 5 has been one of the first
convolutional neural network its
architecture has convolution of
subsampling another convolution again
another subsampling and 3 fully
connected layers let's see how to
implement this in torch as we have seen
we have a special convolution because
from one layer to six layers with
kernels of five by five then we have a
max pooling with a kernel of two by two
and a stride of two by two then we have
at an edge non-linearity it follows a
special convolution with six going from
six layers to 16 layers using also a
convolution of by five by five there
follow a spatial max pooling with a
running window of the mesh 92 pixel by 2
pixel in a stride of 2 and 2 in both
direction ality moreover there is one
more tonnage non-linearity then we have
a few of the 3d output converted into
one D output so that we can apply a
linear layer going from 16 times 5 times
5 220 then we have a non-linearity in
tonnage going from 120 activation
284 activations the last nonlinear
function and then the output connections
that goes from 84 neurons to the last 10
output neurons and then at the end we'll
turn the network let's see how it looks
in the terminal we can open our wrapper
will type in torch we will require
clicking and then so that is going to be
easier to understand the structure and
architecture of these networks require
and here we have the architecture as we
have seen we have 12 layers starting
from the spatial contribution max
polling done age special convolution
again special exfoliating at an age then
we have a view which gives us a 1
dimensional view of the three
dimensional feature map and then we have
a dinner for four hundred two hundred
twenty non-linearity another linear a
non-linearity and a final output layer
the image net challenge consists in
correctly classified 1 out of 1000
glasses and for each class there are
1000 training images overall there are 1
million images on which convolutional
neural network can be trained the first
network has been Alex net we can see
that it's top one single model single
crop accuracy it's roughly 55%
that was a huge increment with respect
to the traditional computer vision
techniques then we have patched
normalized alex net patch normalized
networking network and then we see the
Google net or Googler net in honor of
the first network visa so before Lynette
which achieves an incredible 68% then we
had the introduction of the resonant
which goes up to an astounding 77%
at the end we see the inception v3 which
which gets even higher accuracy still
for a single crop and a single model
evaluation not all these Network have
been designed in order to maintain
computations under a specific value as
you can see from this chart on the
x-axis I represented the number of
operations that each network requires on
the y-axis instead you can see the
amount of top one single crop single
model accuracy the size of these blobs
represent the size of the networks in
terms of million of parameters we can
see in the bottom left there is Alex net
and the base normalized Alex net way
smaller just above Alex net we have base
normalized network and network which
obtains higher accuracy with a similar
amount of operations but with way less
parameters 10% above of base normalized
network in network we can see Google net
which is the smallest network over all
these architectures it has the highest
accuracy density is the chip 68% with
roughly 3.5 giggles then we can see the
the network turns to the right and they
follow an hyperbola if you check my
article and analysis of tip neural
network models for practical application
I explained with greater detail this
figure and many other more relationships
of the best network train on the image
net challenge finally we can see that
ResNet 101 has five Giga operations more
of the inception v3 which is instead a
better compromise between high accuracy
and still small operations we will then
start with seen how the first
architecture has been created the Alex
net The Alfred Ardex Krajewski submitted
the first convolutional neural network
to the image net challenge
it's architecture is as follow let's see
how we can implement this in torch as we
have seen there are two identical
branches let's start by building one of
these two branches and then clone it the
first branch FB one is a sequential
model which has a special convolutional
which goes from three channels RGB to 48
using a convolution of eleven and eleven
applying a stride of four and four Anna
padding of two and two for each side
with which accounts for four pixels in
both Direction allottee then we have a
rectifying leaner unit we have a special
Mac spawning with a window of three
pixel by pixel in a stripe of two pixels
and two pixels this means that the size
of the windows is higher than this
statement stripe this was done in order
to prevent loss of information but it's
not any more commonly used instead of
directly find in your unit introduced by
our exclusive screen is now extensively
used and extensively and very commonly
used although it has some drawbacks like
duplications of filters in oppositional
phase then we have a special conclusion
going from the 48 names 228 names using
a convolutional kernel of five pics by
five pixel astride of one in one in both
directions in parting off two pixels and
two pixels in this way the output size
does not change we start with 27 pixels
in input and we also end up with 27
pixel in output we have again a
nonlinear function follows and a spatial
max folding again with the running
window of size three entry enough stride
of two into another special convolution
going from 128 maps to 192 maps using a
kernel size of three by three a strength
of one by one and a parting of one by
one which introduced two pixels in both
Direction allottee and therefore the
input and output are the same size 13
pick certain
pixal both in input and output again a
nonlinear function
another special convolution going from
the 192 maps to another 192 using 3x3
Koerner a straight of one by one and a
parting of one by one which preserves
again did mention IDT starting with 13
by 13 activations and also outputting a
13 in height and 13 in with activations
another nonlinear function finally there
is the last spatial convolution going
from 190 to 200 228 again using three by
three corners astride of one by one and
parting off two pixels in both direction
ality which preserves again the
dimensionality of the maps there is a
nonlinear function and in a spatial max
volume with a running window size of
three by three in a stride of Toboso
which have the dimensionality of the
maps going from 13 Peaks a squared - 6
pixels squared then we clone our first
branch and then we are set the kernel
values in order to break symmetry
otherwise the two branches would be
converging identically to the same
solution then we concatenate the two
branches one after each other with the
NN dot-com cut module finally we apply
the classifier which is a fully
connected layer as we have seen in the
previous lessons therefore there is a
sequential container which contains a
view of the 3d input feature map there
is a drop out connections which
introduce regularization in the network
riesling a connection coming from the
view of 256 times 66 to 4096 activations
then there is again a rectified linear
unit another dropout another linear
layer which goes from 4096 to 4096 which
therefore has a matrix of 4096 square
again rectified linear unit and then the
final linear layer which goes from 4096
neurons to the output 1000 euros
finally there is a lock softmax which is
part of the criteria which we shouldn't
care too much at this moment the last
instructions simply put in a sequential
the features which are detecto
concatenated ranges and a classifier
let's see how it works we can run our
interpreter by typing th we can require
20 nm and then we can require Alex Ned
and here we have it we can see there are
two identical branches with the
convolutional non-linear and max poland
i guess we have just defined after the
two parallel branches we have the fully
connected layer
now we see a GU cleaner with first outer
is Christian CKD this is the
architecture there are two interesting
point it introduced the inception module
where we can see different convolutional
layers with different kernel sizes
applied to the same input moreover we
have auxiliary classifiers which are
used for improving performance moreover
there are other versions of inception
architecture so Google Annette was the
version v1 then we have D 2 v3 and v4
this architecture is a little bit more
complicated with respect to the one we
have already seen let's start let's
start by defining some shortcut in my
case SC means special convolution SMP
means special max pooling our Lu means
reloj this way I can say space
afterwards the first part is the
function which allows me to create an
inception module given the input size
and the configuration table an inception
module is a concatenation of several
modules in parallel therefore we use the
NN concat module the conclusion one is a
sequential module which is a special
convolution with input size equal input
size output size equal to config 1 1 and
a kernel of dimensionality 1 by 1 then
there is reloj non-linearity at the end
we add become 1 to detect cut module we
continue with count 3 which is a
sequential as well it has a 1 by 1
special convolution layer and then it
has another special convolution with a
kernel size of three by three with
stride of one by one and parting of one
by one so the dimensionality is
preserved finally both the layers have
directly fine linear unit
linearity also comfrey is add to depth
cut following we have convolution 5
which is also sequential of two special
convolutions
the first one is performed with a one by
one kernel and the second one has a 5x5
kernel stride of one and padding off to
baffle so that we have four pixels for
padding for both directionality which
preserves the dimensionality both layers
have a real non-linearity and we add
confide to the depth cut finally there
is the pore layer which is a sequential
as well in which we have a spatial max
pooling and a special convolution and we
add also this module to the depth cut
the first layer of the gulanite is a
factorized convolution we start with a
sequential model we enforce continuity
of the tensor we insert and then we want
to view the the RGB color input image as
three of one plane so that I can use a
convolution to go from each of these
planes to eight separate maps using a
special convolution it goes from one
input to eight output a kernel size of
seven by seven stride by two by two and
a padding of three pixel by three pixel
then we have a depth wise convolution
which is performed with the and in
parallel module to which we add three
convolutional layers finally with the
factorized convolution is a sequential
of the depth wise parallel module to
which we apply a nonlinear function and
then to which we apply a special
convolution going from 8 x 324 to 64
maps using a kernel of one by one
to build the rest of the network we will
build separately these blocks we will
have a main zero block which is sent to
both a main one in softmax zero main one
is sent to both main two and softmax one
and then mean to goes to softmax to the
main zero is going to be a sequential to
which we add the factorized convolution
then we have a spatial max pooling a
special convolution another special
convolution a special max boiling and
then we have two inception layers
another special max porting and another
inception layer the main one block
instead is made of three inception
modules the main tool instead is made of
an inception module a special max
pooling in other two inception modules
the first auxiliary classifier called
soft max 0 is the sequential module
which has a special average pooling and
then it has a special convolution of you
a linear and repelled another linear and
then the final block soft max the soft
max number one is also an average
pooling special convolution view a
linear drop out linear clock soft noise
and the final soft max number two will
have a again a special average putting a
view a drop out a leaner and docsoft max
here we use a special average pooling
instead of special Max pudding because
we want to minimize the error introduced
by the main so many layers by averaging
it out now we can assemble all these
blocks like they would be legal blocks
in the final architecture we will start
from the end we seen that block 2 is
going to be a sequential which simply
has the concatenation of main tool and
softmax toe
then we're gonna use split split is
perform with a cone cut off block to the
Cascade of main to end softmax tool with
the softmax one again these two branches
that are in parallel are in serious with
the main one so we are going to use a
sequential which has main one and split
one main one together with all block one
is empowered to softmax zero therefore
we use a cone cut to which we input
block one and softmax finally there is
the last serious connection we have the
last severe connection where we see main
zero connected to split zero so this is
very similar to how someone would draw a
electrical circuit with resistors in
parallel ending series using con cut in
order to make him make branches in
parallel and sequential in order to make
branches in serious finally we can call
model block serie and we return model
let's run our interpreter required
pretty NN require approval net and let's
go for screen and it's quite long let's
try to understand what's going on so at
the beginning we have the main
sequential which contains two blocks the
main 0 and splits then if you go inside
every block we will end up with the
first block where we have that the RGB
map is split in three separate layers
and a convolution to each separate
layers it is performed
then we haven't really known inheriting
a special convolution and again are we
go then there are the classical layers
of the network like spatial max pooling
spatial convolution radio special
convolution radio again
spatial exporting then number eight we
have our first inception motor where we
can see it's a con cut of four modules
in parallel the first one is a special
convolution with the kernel of one by
one the second is a series of special
convolution of kernel one by one in a
convolution of with layer three
basically there is another one with one
by one and five by five and the final
one it's the max pooling layer follows
another inception model patient max
pooling another inception model then
here we have our conquered which is our
branch for the auxiliary classifier so
in the first part we're gonna see more
inception models and if we go follow the
red line on the left we can see we have
our first equation which is our first
oxidation Seafire can go up and we can
move inside the network here inside the
network we have an exception module
another inception module another
inception module and here we have number
two goes into an inception module on the
first right and we can go down following
the second red line where we see our
second auxiliary classifier okay climb
up follow again the network with a
reception module a special exporting
under Inception another inception module
and then we have the final classifier
define architecture we're gonna
encounter today is the rest net which
comes from the publication on the
procedural learning from image
recognition timing and Microsoft
Research this paper uses the theory from
the highway network in order to train on
a neural network for the image net
challenge we see the domain principle
here are some connections that are
coming out from the network and bypass
couple of layers we can see here a
better view each couple of layers is
bypassed by our identity connection in
this way the network can be simplified
by gradient descent
moreover the gradient signal has a clear
path in order to back propagate through
the early layers of the network avoiding
therefore gradient degradation for this
last Network I will show you how the
network looks like in torch and then you
for exercise should try to write down
this network by yourself so that you can
be sure you know how how this network
can be generated so here I have my last
night 18 I can do th require reading and
then also we requires you GN n which is
a library for CUDA use for this
convolutional neural network and then I
can do torch load last night writing and
here it is
so from the beginning we have a
sequential with 11 modules we start with
a special convolution going from 3 to 64
then there is a special batch
normalization also this layer similarly
to drop out helps regularization then we
directly find in a unit follows a
special max pooling then we have our
first block cough bypass connection we
have a sequential of a sequential in the
second sequential we have a Concord
table where we can see the first branch
it's a sequential of spatial convolution
special batch normalization real special
convolutional special h normalization
which is bypassed imparted by a an
identity finally the two output which
are tables since we use a conquered
table are then added together using C
add table 2 which is appended a
rectifying linear unit follows in the
sequential the second sequential out of
2 which number 5 heads so we see again
is exactly as the one above so we have a
conquered table which split the network
in two parts we have a special
convolution special batch normalization
defined in our unit I can especially
convolution and a Bayesian special batch
normalization which is bypassed partly
by a NN identity therefore the two
outputs which are going to be contained
in to a table are summed together with
the NN seat at the table and rectifying
dinner you will need to follows then we
have the layer 6 which is simply another
sequential so basically these two module
number two here a number one could have
simply have been if we could have simply
been in the main branch
I believe this specific architecture was
done this way because allows generally
to add more branches and more features
in case we use more complicated networks
this was unnecessary here so module five
could have been simply removed and
module one of five module two or five
could have been simply been in the main
branch then module six in second on a
sequential which contains first
sequential and second sequential as
before these two modules could have been
in the main branch and this special
convolution and special batch
normalization rate of spatial
convolution special batch normalization
are bypassed by a special convolution
which changes the number of maps using a
one by one cannon so before we were seen
a identity because the input of the
convolution was 64 and the output of the
last convolution was also 64 in this
case the input of the convolution is 64
but the output is hundred twenty-eight
so since we have incremented the number
of channels we have to as well increment
the number of channels in the bypass
connection that's why we use a special
convolution going from 64 228 will
connect with kernels of one by one and
stride of tomato because also the first
convolution here has a stride of tomato
in this blog we see that the output
number - it's a special convolution
whereas before it was a identity this
because we go from 64 Meg's 228 before
we were going from 64 to 64 but the
stride was of 1 whereas now we have a
stride of - so since we make the size of
the output feature map a quarter of the
input we have to double the number of
channels in order not to lose high
amount of information follows another
bypassed connection of 128 128
convolution with a 3x3 kernel special
based normalization and reloj special
convolution again hundred 28 228 with a
3x3 kernel extract of one imparting of
one so to preserve the mentality and
follows a special batch normalization
which is bypassed by an identity since
we preserve the same number of channels
across these modules finally we sum the
two and we apply up non-linearity and so
on we will have another sequential with
another bypass connection which uses a
special convolution because you go 428
over 256 and in this case also we use a
kernel of 1 by 1 but a stride of 2 by 2
because of the convolution and the layer
number one as a stride of two by two and
it goes from hundred twenty-eight maps
to 256 using a kernel by three by three
follows a summation of the two and
non-linearity another sequential where
we have the preservation of the number
of channels of the feature Maps
therefore there is an identity summation
and non-linearity
another sequential again we have a
special convolution which has a stride
of two by two and therefore we also
increment the number of maps from 256 to
512 and therefore the bypass connection
is performed with a convolution with the
kernel one by one going from 256 maps to
512 is using us right off to bed
summation and non-linearity or following
another sequential in this case try this
one by one and therefore we find an
identity and we have again a summation
of the tool and a non-linearity finally
we have the final special average
pooling which also as we saw in the case
before with Google net improve the
robustness against noise by averaging
out the noise we have a view of 112 and
therefore we have this singular linear
layer going from 512 to 1000 so here we
have to notice that this network really
does not have any dinner layer since you
remember from the previous videos we
have seen that the dinner layer are very
expensive especially for addicts net and
the network we have shown in the
practical 3.0 thank you for listening
and let's stay tuned because the next
practical I will show you how to train
these networks using the op team package
from torch