Deep Learning with Tensorflow - RBMs and Autoencoders Video Lecture - Tensorflow

Hello, and welcome!
In this video we will provide an overview
of RBMs and autoencoders.
RBMs, or Restricted Boltzmann Machines, are
shallow neural networks that only have two layers.
They are used to find patterns in data by
reconstructing the input.
We say that they are "restricted" because
neurons within the same layer are not connected.
RBMs were first created by Paul Smolensky
in 1986, and they were further developed by
Geoffrey Hinton in 2002.
RBMs are useful in many applications like
dimensionality reduction, feature extraction,
and collaborative filtering, just to name
a few.
So let's take a closer look at the learning
process of an RBM.
We mentioned that RBMs learn patterns and
extract important features in data by reconstructing
the input.
So let's say that we provide an image as input
to an RBM.
The pixels are processed by the input layer,
which is also known as the visible layer.
The learning process consists of several forward
and backward passes, where the RBM tries to
reconstruct the input data.
The weights of the neural net are adjusted
in such a way that the RBM can find the relationships
among input features, and determine which
features are relevant.
After training is complete, the net is able
to reconstruct the input based on what it learned.
During this process, 3 major steps are repeated.
The first step is the forward pass.
In the forward pass, every input is combined
with an individual weight and an overall bias.
The result goes to the hidden layer, whose
neurons may or may not activate.
Then we get to step 2: the backward pass.
In the backward pass, the activated neurons
in the hidden layer send the results back
to the visible layer, where the input will
be reconstructed.
During this step, the data passed backwards
is also combined with individual weights and
an overall bias.
Once the information gets to the visible layer,
the input is reconstructed and the RBM performs
the third step.
Step 3 consists of assessing the quality of
the reconstruction by comparing it to the
original data.
The RBM then calculates the error and adjusts
the weights and bias in order to minimize it.
These 3 steps are repeated until the error
is sufficiently low.
Let's touch on a few reasons why RBMs are
such a great tool.
A big advantage is that RBMs excel when working
with unlabeled data.
Most important real-world datasets are unlabeled,
like videos, photos, and audio files, so RBMs
provide a lot of benefit in these types of
unsupervised learning problems.
Another advantage is that during the learning
process, the RBM extracts features from the
input, and it decides which features are relevant,
and how to best combine them to form patterns.
Also, RBMs are generally more efficient at
dimensionality reduction than principal component
analysis, which is a popular alternative.
As RBMs learn from the data, they actually
encode their own structure.
This is why they're grouped into a larger
family of models known as the autoencoders.
Autoencoders were first introduced in the
1980s by Geoffrey Hinton.
According to Pierre Baldi of UC Irvine, they
addressed the problem of back propagation
using the input data as the teacher.
Generally speaking, the main goal of these
neural nets is to take unlabeled inputs, encode
them, and then try to reconstruct them afterwards,
based on the most valuable features identified
in the data.
They're used for tasks that involve feature
extraction, data compression, dimensionality
reduction, and learning generative models
of data.
So let's take a look at their structure.
Most autoencoders are shallow networks, with
an input layer, a few hidden layers, and an
output layer.
As we saw before, RBMs are autoencoders with
only two layers.
Autoencoders use backpropagation in their
learning process.
Instead of cost, the metric used to assess
the quality of the network is loss, which
is the amount of information lost in the reconstruction
of the input.
The goal is to minimize the loss, so that
we have an output that's as close to the input
as possible.
At this point, you should have a better understanding
of the structure and applications of RBMs
and autoencoders.
Thank you for watching this video.