Deep Learning with Tensorflow - The Recurrent Neural Network Model Video

Hello, and welcome!
In this video, we will provide an overview
of the recurrent neural network model.
A Recurrent neural network, or RNN for short,
is a great tool for modeling sequential data.
The RNN only has one layer, but it maintains
a state, or a context, in order to remember
the analysis that it's done up to that point.
This state "recurs" back into the net
with each new input, which is where the net
gets its name.
Let's take a closer look at how this works.
Input data first flows into the model's
single layer.
The data is processed in the same way as a
traditional net, but as we mentioned the net
also receives the current state or the context
along with the input.
If this is the first data point, then some
form of "initial state" is used, which
will differ depending on the type of data
being analyzed.
After processing, the data is output with
a new context that represents the most recent point.
Then this context is fed back into the net
with the next data point, and so on.
We repeat these steps until all the data is
processed.
Keep in mind that since the context changes
at every step, the net can produce a different
output for the same input depending on the
current state.
This is what allows a recurrent net to perform
sequential data processing.
Recurrent neural networks are extremely versatile
and are used in a wide range of applications
that deal with sequential data.
One of these applications is image captioning.
Although it's not purely recurrent, Andrej
Karpathy and Li Fei-Fei created a model that's
capable of understanding the elements in an
image.
The model can assign words to the elements,
and string them together to form a caption
that describes the scene.
You can take a look at two examples of that
here.
Another application for recurrent nets is
music composition.
A network can be trained using MIDI files,
which are a type of standard for digital audio.
After learning about rhythm and melodic patterns,
the net can output data that can be converted
back to the MIDI file format, and listened
to.
We only covered a few applications, but variants
of the recurrent model are continuing to solve
increasingly complex problems.
You can see a few of those variants here,
but they're beyond the scope of this video.
Despite all its strengths, the recurrent neural
network is not a perfect model.
One issue is that the network needs to keep
track of the states at any given time.
There could be many units of data, or many
time steps, so this becomes computationally
expensive.
One compromise is to only store a portion
of the recent states in a time window.
Another issue is that Recurrent Neural Networks
are extremely sensitive to changes in their
parameters.
As a result, gradient descent optimizers may
struggle to train the net.
The net may suffer from the "Vanishing Gradient"
problem, where the gradient drops to nearly
zero and training slows to a halt.
It may also suffer from the "Exploding Gradient",
where the gradient grows exponentially off
to infinity.
In either case, the model's capacity to
learn will be diminished.
By now, you should have a good understanding
of the main ideas behind the recurrent neural
network model.
Thank you for watching this video.