Deep Learning with Tensorflow - Activation Functions Video Lecture - Tensorflow

Hello, and welcome!
In this video, we will provide an overview
of several activation functions, many of which
are included in the TensorFlow library.
Activation functions are a cornerstone of
Machine Learning.
Generally speaking, a processing unit will
pass its input through an activation function
to generate the output, so the choice of function
ends up playing an important role.
We'll go through some of the most commonly-used
functions in order to develop a more intuitive
understanding.
But before we begin, we need to import a few
dependencies.
The following piece of code contains a basic
function that will plot the surface of an
arbitrary activation function.
The plot includes all possible weight and
bias values between -0.5 and 0.5, with a step
of 0.05.
The input, the weight, and the bias are all
one-dimensional.
The input can also be passed to the function
as an additional argument.
The Step function was one of the first activation
functions designed for machine learning.
This function essentially acts as a limiter.
If the input exceeds a certain value, the
output will be 1, otherwise the output will
be 0.
It should be easy to see that this type of
output lends itself well to classification
problems, like two-class logistic regression.
There are variations like the Rectangle step
function and a few others, but they aren't
commonly used.
And as it turns out, TensorFlow does not offer
a step function.
Sigmoid functions are an extremely popular
function family in the machine learning world.
The term comes from the 'S' shape that the
plots form on the Cartesian plane.
Sigmoid functions are useful because they
compress the input down into a bounded interval.
This helps when you need to combine the result
with other functions, like the Step function
from before.
The most commonly-used sigmoid functions are
the Logistic, Arctangent, and Hyperbolic Tangent
functions.
Logistic Function
As you'd expect, the logistic function is
widely used in Logistic Regression.
You can take a look at its definition here:
This function maps the input into the open
interval from 0 to 1, like so:
The Arctangent and Hyperbolic Tangent functions
are both based on the tangent function, as
implied by the names.
Arctangent is defined here:
and it produces a sigmoid over this interval:
Keep in mind that TensorFlow does not include
Arctangent.
The Hyperbolic Tangent, or TanH as it's usually
called, is defined like this:
It produces a sigmoid over the open interval
from -1 to 1.
TanH is used in a wide range of applications,
and it's probably the most popular function
of the Sigmoid family.
Linear Units form an important category of
activation function by combining the best
concepts from Step and Sigmoid.
In general, all the linear unit variations
stem from something called the Rectified Linear
Unit, or ReLU for short.
ReLU is a simple function that outputs a value
from 0 to infinity.
So say we have some input 'x'.
If 'x' is negative, ReLU outputs 0.
But if 'x' is non-negative, ReLU just outputs
'x'.
It may seem counterintuitive to use a basic
pseudo-linear function over something more
complex like a Sigmoid.
But ReLU provides some important benefits
that might not be obvious at first glance.
For example, during the initialization process
of a Neural Network model, weights are distributed
at random for each unit.
ReLU will only activate approximately 50%
of the time, which actually saves some processing
power.
The ReLU structure also solves the 'Vanishing
Gradient' and 'Exploding Gradient' problems,
both of which are well-known issues with the
training process.
And as a marginal benefit, this type of activation
achieves 'Biological Plausibility', because
it's directly relatable to the common biological
model of a neuron.
Different ReLU variations are optimized for
certain applications, but since they're implemented
on a case-by-case basis, they're beyond the
scope of this video.
If you want to learn more, you can search
for 'Parametric Rectified Linear Units' or
'Exponential Linear Units'.
The TensorFlow library includes ReLU along
with a few key variants.
You can take a look here.
So by now, you should have an understanding
of the activation functions supported by TensorFlow.
Thank you for watching this video.