Deep Learning with Tensorflow - The Long Short Term Memory Model Video

Hello, and welcome.
In this video, we will provide an overview
of the Long Short-Term Memory Model.
The recurrent neural network is a great tool
for modeling sequential data, but there are
a few issues that need to be addressed in
order to use the model at a large scale.
Recurrent nets need to keep track of states,
which is computationally expensive.
There are also issues with training like the
vanishing gradient, and the exploding gradient.
A popular method to solve these problems is
the Long Short-Term memory unit, or LSTM for short.
LSTM is an abstraction of computer memory
that works in tandem with a recurrent net.
It's used to maintain, update, and regulate
the states of the network model, outside of
its normal execution flow.
The LSTM units are composed of four main elements.
The "memory cell", or "information cell",
is responsible for holding data, and the three
logistic "gates", define the flow of data
inside the LSTM.
The Write Gate, or the Input Gate, is responsible
for writing data into the memory cell.
The Output, or Read, Gate, reads data from
the information cell and sends that data back
to the recurrent network.
And the Keep Gate, or Forget Gate, maintains
or deletes data from the information cell.
These gates have a similar structure to the
neurons in a traditional neural network, since
they are multiplicative analog sigmoid-activated
nodes.
By manipulating these gates, a Recurrent Network
is able to remember what it needs, and forget
what is no longer useful.
This is what a recurrent network would look
like, if we unfolded it at each time step.
The LSTM cells exist between the time steps
in the network's flow.
So the network's output can be input into
the LSTM, and the LSTM can send its output
to the net's input in the next time step.
Let's take a closer look at an LSTM unit's
structure.
The data input and the network's state are
directly connected to all the gates.
So each gate receives the same information
as the recurrent net's processing cell.
Additionally, the network's output is connected
to the Write gate, and the Read gate is connected
to the network's processing cell.
The Write gate sends its output to the information
cell, and the information cell can send data
into the Read gate.
These gates are all analogue logistic units,
which means their behavior is defined by a
logistic function.
For example, a value of 0 in the Keep gate
means "forget everything", a value of 1 means
"retain everything", and a value of 0.7 means
"keep exactly 70 percent of the current data".
This applies to all three of the gates.
For now, let's just focus on the individual
Gates themselves.
The Keep Gate is responsible for maintaining
the data in the information cell.
It receives the same input data and state
as the network, and then it calculates how
much of the current data should be remembered.
The Write Gate is responsible for inputting
new data into the memory cell.
Like the Keep gate, it receives the same input
data and state as the recurrent network.
However, it also receives the recurrent net's
output data from the most recent time step.
The Write Gate uses the inputs to determine
how much of the output data should be written
into the memory cell.
The Read Gate is responsible for sending data
from the LSTM back into the Recurrent network.
Like the other gates, it receives the inputs
and state from the network.
The Read Gate reads a value from the information
cell, and this value is interpreted as a signal
between -1 and 1.
The input data and state are then used to
determine how much of this signal should be
sent to the Recurrent network.
Working in tandem, all three gates allow the
model to freely regulate and manipulate the
data inside the information cell.
But you might be wondering why the LSTM uses
logistic gates.
The reason is that logistic-based functions
have very nice derivatives.
As a result, we can backpropagate through
the gates, which enables the recurrent net
to understand how to utilize the LSTM structure.
But the details of this process are beyond
the scope of this video.
The main point is that by manipulating values
through the gates, we eliminate the vanishing
gradient and exploding gradient problems from
before.
And since the gates allow the network to forget
the states that are no longer needed, the
computational load of the recurrent model
decreases drastically.
By now, you should understand the structure
and purpose of the Long Short-Term memory model.
Thank you for watching this video.