Practical 4.0 – RNN; vectors and sequences Video Lecture - Torch A Practical

in this chapter we will see how to use
neural networks not only to deal with
vectors but also with sequences we will
start by viewing the differences between
a vanilla classic neural network in a
recurrent neural network then we will
see a sequence of combinations of
vectorial and sequential input and
output and their respective applications
for example we will see how to map a
vector to a vector a vector to a
sequence a sequence to a vector a
sequence to a vector n again back to a
sequence and finally a sequence to a
sequence so here we will introduce a new
notation each of these circles represent
a whole layer of a neural network
previously we were representing with
each and every circle each and every
scalar activation of a given layer from
now on we will represent with a circle
the whole layer so so it represent a
vector the input is called X the hidden
vectorial representation is called H and
then the output of the network
it's called Y hat whereas our labels are
still Y we can make a very nice
correspondence with the combinatorial
logic here on the right hand side where
the output depends solely on the current
input a recurrent neural network
introduced a recurrent connection coming
from the previous state or previous
internal representation in order to be
able to represent this network without
recurrence we will replicate the network
over time indexes where all the
parameters are shared among the same
note here the analogy goes to the
sequential logic where the current
output of a system depends on the
current input and on the specific state
in which the system was neural networks
are very powerful abstractions which
allow us to play with different kind of
data the simplest kind of network we
have
so far deals with vectors and produce
vectors let's see how to represent this
kind of first type of neural networks so
we have an input here which is then
mapped into a hidden layer and then
finally we produce an output example of
this kind of networks are used for image
classifications bounding boxes
regression so that we can perform
localization and in general any kind of
f of X regression so can also be called
as function approximation and basically
classical neural networks can be thought
as system that are optimized over
function regression this case was vector
to vector u can be represented by the
following notation so we go from a
vector X to a vector Y prediction so Y
would be the label Y hat is going to be
now our prediction of our network so we
go from one vector X to another vector Y
either one it can be a one element
vector that means just a scalar for
example we could predict the price of an
apartment so that would be like a scalar
given a set of features or given a
scalar instead as an input like would be
the x axis we could predict a Y which is
a multi-dimensional function for example
let's see how our classical neural
network is able to classify different
images representing the digits from 0 to
9 of the Amnesty the set in this chart
we see
the projection of the vectors
representing each and every digit from 0
to 9 on a 2d subspace on which we have
maximized the variance so this is the
view in which the digits look the most
separate between each other and we can
see that is kind of hard to find
separation planes that are able to group
individually each and every separate
digit if we train a neural network in
order to do this task it will learn to
move the space across the different
layers in order to achieve a
representation where the classes are
linearly separable and the linear
separation will be performed by the last
logistic unit so after I train my
network on this data set I reproject
the output representation corresponding
to each of these samples and what we get
is this we can clearly see now that each
and every digit can be easily separated
by each other using simple linear
separation hyperplanes what if we
provide one input but then we would like
instead to have a sequence in terms of
output let's say so we have one singular
input here our X and then we have a
sequence of internal states or hidden
variables which then produce a sequence
of output so we have our input here and
it produce the first output then given
the first input we produce the second
output and moreover we can produce for
example a third output an example of an
application that would use a similar
scheme is image captioning
where we will just provide one input
image X but then we actually expect a
sequence of words which are describing
the content of the image so in this case
I will write here y hat function of an
index T and this is vector to sequence
here we see an example of vector to
sequence mapping the input vector is an
image like we can see here in the output
is a sequence of symbols representing
words in a sentence we have different
performance from the left to the right
we have descriptions without errors then
we have descriptions with minor errors
then the captions are somehow related to
the image or on the right hand side we
see completely unrelated captions this
task would have been impossible to
tackle with simple neural network and is
our first example of how a recurrent
neural network can be used in practical
applications we have now two elements we
have vectors and sequences so we can do
all their combinations now let's do
Seaquest vector so in this case I will
provide to my network a sequence 1 2 3
of symbols which are fed into the hidden
layer but then we produce an output only
at the end so we insert the first symbol
into the network and then we insert the
second symbol which will be affected by
the symbol that we inserted before we
insert up to the last one and again this
one will be affected by whatever we have
input before at the end we can ask the
network to output a specific result
possible applications our equation
solvers
or even better we could even create
system which is able to learn to program
we call this program executor so in this
case before we saw that the classic and
neural network which is mapping a vector
to a vector can be thought as a system
that is optimized over functions a
recurrent neural network can be thought
as a system which is optimized over
programs we teach a system how to
perform simple tasks given some inputs
and the internal state of the system we
expect an output and everything is just
learned by gradient descent so we are
creating a system which is learning how
to perform tasks given our data and
given our expected result so in this
case we said we input a sequence and we
expect a vector output so we have our
input sequence and then we expect our
prediction which I will just call last
element of our possible output sequence
so it would be like sequence to sequence
where I just considered the last element
of the sequence here
we are talking about sequence to vector
mapping our input data it's a sequence
of instructions more specifically here
it's a sequence of characters which are
making up a program our output will
simply be the print value on screen so
in this case the recurrent neural
network has to learn the semantics of
this programming language and also has
to learn how to compute each and every
instructions in order to make a reliable
diction in this case we can talk about
optimization over programs let's combine
these last two schemes into a new system
so that we can input a sequence which is
then condensed into the state of the
system and then when the state of the
system is updated with the last element
of the input sequence if we start
outputting an output sequence let's draw
it just to have a better understanding
of what I just said so we have our first
sequence we have 1 2 3 which is input to
our system and then after the system has
accumulated a history of inputs and it's
state has been updated given the input
sequence we start outputting some
results so here we propagate to the next
state let's say we have three states
update after the input has stopped and
then we output here our output sequence
which can be of any kind of length
doesn't have to match the input length
this kind of scheme reflects the life of
a pH this unit you get much information
across the years so we have first year
second year third year and maybe a
fourth and even a fifth and ok and then
it becomes said but ok let's go over and
after this many years of input from the
specific field and domain of studies we
improve our internal state to a better
and better and better representation and
then afterwards when you have crafted
your perfect state you have someone that
wants to hire you so that you can
actually spit out years of good work
based on your experience and all the
companies are looking for is basically
this very nice representation if we in
the same way we would like our network
to perform a similar task we input some
specific input
for example there could be a sequence of
symbols that can be like the words in a
sentence then we have a representation
of a concept that the sentence tries to
express and then we can think to output
this concept in a different domain
perhaps in a different language
this scheme reflects not only the PhD
student life but of course the neural
machine translation which internal state
shows very very interesting
characteristics as we shall see in this
case we went to a sequence to sequence
but then we also came through a
condensed representation which is
actually a vector so in this case we can
call this scheme also encoder/decoder so
we have an encoding of a sequence into a
pictorial representation and then we
have a decoder which transforms and
recovers the sequential nature of the
domain we were working with from the
internal vectorial representation so in
here we can write this one as our input
which is a sequence we go through a
hidden pictorial representation without
time index and then we again expand into
a Y hat so prediction which is the X
term over the time index T as we said
this tape of a encoder/decoder scheme
has very interesting properties here we
can see a 2d projection with maximum
variance among embeddings of different
words that we have provided to our
network that has been used as a neural
machine translation we have input one
sequence of words representing a
sentence in a given language got a
compressed representation of the
sentence and then spit out another
sequence of symbols representing words
in a sentence in another language
this chart represent the projection on
the 2d plane with
highest variance of the words that are
sent into this system let's see for
example the green cluster we can see
here that all these green terms are
month names not surprisingly they occupy
a very very close region respect to the
other words because they can easily be
swapped without turning the sentence
incorrect
although the meaning will change because
of course different months are
corresponding to different time periods
but they don't influence the correctness
of a sentence moreover if we instead
insert sentences and then we project the
embedding space on a 2d space with
maximum variance we can see clusters of
very similar meaning for example here we
have period of times appearing all close
together you can find more illustrations
if you follow the reference below
finally we had the last combination
which is direct conversion of a sequence
into a sequence so let's draw it we have
here our input sequence let's just draw
3 symbols which are fed into our network
state hidden layer and automatically it
starts outputting as well our output
sequence so we first send the first
symbol and then we expect already some
kind of output then given my first input
and my second symbol we expect to have a
second prediction again given the
previous state and the new input we
expect the output to change again and
I'm very sure every one of you are very
very familiar with that a practical
application of this kind of scheme and
yes it's your lovely outer correction
which tries to predict the next word
given the character we have just
inserted so the pink circles represent
the symbols representing each and every
character I input into my phone and the
blue balls represent the completion of
the current Ward and such
perhaps for the next word we can call
these on t9 or autocorrection which
output a sequence of symbols which are
for example the word given that I input
a sequence of other symbols which are in
this case characters so we can mix and
match we can input characters and we
expect perhaps a probability
distribution of the most likely word
completion choices and therefore here we
go from our final combination which is
sequence to thickness finally the
sequence to sequence mapping we said can
be used for making the autocorrection on
your phone here I just picked a very
nice project where the outer has made a
package for the item text editor which
can be used to write novels in a fantasy
style since it's been trained on fantasy
corpora here we can see the rings of
Saturn glittered while V and then we ask
for a prediction harsh eyes lead to men
looked at each other and then you can go
on they were enemies but let's see what
happened the server robots weren't
concerned so you can have fun with this
project you can find on get up on the
link below each circle represent a
vector and in depth so going from the
bottom layers to the high level layers
we represent different layers of the
network on the x-axis instead I
represented the time index so as we move
from the left to the right we see
different time steps before in the
previous lessons I was representing
with a circle each and every activation
of a specific layer in this case here
each circle represents the complete
layer with all its own activations stay
tuned because in the next video we will
see the equations that govern the
recurrent neural network and how they
relate to the equations of our vanilla
neural network moreover we will see how
we can train this system with back
propagation through time