Kernels Introduction - Practical Machine Learning Tutorial with Python p.29 Video Lecture | Machine Learning with Python - AI & ML

hello everybody and welcome to part 29
of our machine learning with Python
tutorial series and a continuation on
the support vector machine in this
tutorial we're going to be talking about
what to do when the realities of the
world hits you so up into this point our
support vector machine has been working
with linearly separable data now
unfortunately it's just the case that in
the real world you are almost certainly
not going to get linearly separable data
it's just highly unlikely so for example
what we have here is a two dimensional
feature set with the classes plotted on
you got the plus class in the minus
class here and of course if I was to ask
you okay what are the support vectors
right you would probably not know if I
asked you what's the best separating
hyperplane we we just don't have one
it's not possible right now with the way
that we're looking at things now one of
the most crude options that you might
have at your disposal is to simply take
another perspective and literally add a
new dimension so you could add a new
dimension using all kinds of functions
right so right now we've got x1 and x2
you could say something like x3 equals
it could be something as simple as x1 x2
just x1 times x2 so you could do
something like that and boom you've got
a third dimension
now the problem happens however when you
are working with let's say feature sets
that are maybe let's let's say you're
doing image analysis or video analysis
basically the same thing you might have
hundreds of features or thousands of
features even and so looking at the data
we have right here obviously we just
added one more dimension
from two dimensions or three dimensions
not a horrible thing but actually in
reality that's a 50% increase in in data
right and what was that main downfall of
the support vector machine all right
it was training it because if you have a
large size of data it is extremely
cumbersome to train it because of that
quadratic programming and optimization
problem that we have so is it really a
great idea to be multiplying our data
set maybe by 1.5 X right probably not I
mean we want to avoid doing that because
we're just we're weakening the algorithm
on ourselves so we don't really want to
want to do that if we don't have to now
what if I told you that you could do
calculations in plausibly infinite
dimensions or maybe even better put what
if I told you that you could have
calculations done for you implausibly
infinite dimensions without ever
actually having to visit those
dimensions yourself and without having
to pay the processing costs that would
be pretty impressive right well it turns
out that you might be able to do that
and the way that we would do that if we
could do that is by using kernels so
most people I think probably first come
into contact and maybe
lastly including firstly come into
contact with kernels with the support
vector machine and may find themselves
under the impression that kernels are
really just used it it's just like
something using a support vector machine
and that's it that's not the case a
kernel is a similarity function it takes
two inputs and outputs their similarity
simple as that being that this is a
machine learning tutorial some of you
might be thinking already well why don't
people just use kernels as machine
learning classifiers and I would say
that they do it just so happens that we
can actually use kernels to augment or
add to our support vectors
Jeane hopefully and allow us to work
with nonlinear data by transforming that
nonlinear data to another dimension and
finding a linear in creating a linear
linearly separable situation so to speak
as hard as it might be to speak such a
thing so keep in mind why this is so
valuable right out of the gate because
in that previous and that previous page
we were talking about okay well if you
have two-dimensional data is not
linearly separable you could create a
function that could create more
dimensions for you that are based on
those previous features and it would be
valid and statistically sound but you
would be paying a lot more in processing
fifty percent or even more so consider
let's say because you it's not just the
case you add one dimension it's
separable right it's like it what if you
have 30 dimensions already or is it
likely that if you add one more
dimension that you're going to get
linear separate linearly separable data
probably not you're probably going to
add more like 10 or 15 or maybe even
more right you might actually have to
have a hundred dimensions before you
actually get linearly separable data or
maybe you have to add a million
dimension okay this can get out of hand
really really fast so anyways back to
kernels and back to me what if I told
you that you could do this and not pay
the price
so with kernels there's really just one
major element to kernels and that is
that they are done using inner product
and for the purposes of what we're going
to be using them for and the purposes of
this entire tutorial
what is inner product what is dot
product how do they relate they are the
same thing if you pull up a numpy right
now you make a couple vectors and you do
MP dot inner or NP dot dot you're going
to find that the same data gives the
exact same answers when you run that
through so just keep that in mind you're
going to hear especially if you go to
another source or you're coming from
another source
some people say dot some people say
inner and it's just the same thing
I believe fundamentally you should be
using the inner product if you're if
you're writing a paper on this I think
you're its inner but I could be totally
wrong you can look that up on your own
so the question is can we use a kernel
and if so how do we know that we can use
a kernel so to find out if we can use a
kernel we know that in order to use the
kernel we have to be able to use inner
product and then in the way that we can
find out is basically we're trying to
get to some sort of new dimensional
space so up to this point we've been
using basically what we've got X we're
dealing in an X space okay because we're
our feature sets are denoted as X 1 X 2
and so on are we already have a value
for Y and that's the class so so the
next most reasonable variable will be Z
for Z space so the question is can we
interchange X and Z obviously you know
fundamentally we could do that right but
is it is every interaction in our let's
say our optimization algorithm and all
that in it for everything for the
support vector is every interaction with
the X space a dot product or an inner
product well let's find out
so going back let's start at the at the
ended kind of work our way backwards so
right away like let's say we've got an
unknown feature set which is a what
we'll call it X for now
right an unknown feature set is X how
did we determine what the classification
was well it was just simply the
classification was going to be Y and
that was just y equals the sine not the
sy the sine of and that was just w dot X
plus B right and really if we're going
to be perfect I did warn you that I was
not we're probably gonna drop those bars
pretty quick but anyway sine WX plus B
what's this well hot diggity that's a
dot so could we interchange X with some
sort of Z
value absolutely right because again
what is what is what is WX going to
return to us isn't that going to return
a vector or is that going to return just
a number a scalar value well it's going
to return a scalar so does it matter if
X is five dimensions or 50 dimensions I
realize I did a lower case in the
capital anyway five dimensions or fifty
dimensions does it matter no it doesn't
because there's just going to return a
scalar so modifying X you know our x
space to a z space an unknown dimension
space is that going to have an effect on
the classification algorithm that will
sign algorithm nope no problem there now
how about the constraints probably where
a lot of people stopped watching the
series okay so really in reality there
are two major constraints now we really
just worked with one because we were
able to get away with it and we didn't
do true quadratic programming so first
let's look at that one and then we'll go
and deal with the other the other
constraint that we have to deal with so
the first constraint was the requirement
that y sub I multiplied by X sub I w
plus B minus 1 was greater than or equal
to 0 right and again that's we didn't
have we don't have the dot there but
it's just kind of expected that you and
you know that by this point but anyway
that is there's your dot and since we're
really focusing on the dot I figure
might as well make it explicit right and
again right can we interchange X sub I
with a theoretical Z sub I yep because
that's a dot product right we got no
problem replacing that so that's one of
the constraints some of about that other
constraint well the other formal
constraint was that basically it wasn't
even really a constraint just a value
once we found if you recall the the
quadratic programming
was basically give us some values for
alpha so that was going to be the value
for eventually W and that was going to
eventually give us basically the sum
over alpha I Y I X I okay
and again I'll be good and try to stick
my dots in there dot there we go again
X sub I what's it doing looks like a dot
to me so yet again we have confirmation
that every interaction is a dot product
and if we go back to the really long
equation the first one that we wrote
what did we have at the very end of that
well as l equals hopefully i can fit it
on here that was going to be the sum
over alpha i minus one-half the double
sum and that was i and then we said j
alpha I alpha J that double alpha which
was what was scared us away initially
and then y sub I Y sub J dotted with X I
dotted with X J again right and that was
kind of how we that was the equation
that came before that W equals but again
we find that it we've all of the
interactions are a dot product right so
so we can get away with having no
problem with kernels so now what we need
to do is actually cut it is is talk
about what's an example of a kernel and
actually kind of see with our eyeballs
one how the kernel is going to transform
a feature set that we have and then to
actually work out the math to show you
why because so far I've shown you that
okay this is how we know we can actually
use a
colonel but i have yet to really show
you how we can go out to two more
dimensions without actually paying any
price so I've said plausibly infinite
you're not always going to go out to
infinite dimensions you might only go
out to six dimensions or eight
dimensions or whatever but the main the
main value here is that you can go out
to all those dimensions without actually
paying the processing cost and that's
the entire reason why we're going to use
kernels rather than some sort of
function that just creates new
dimensions like what we wrote out before
so that's what we're gonna be talking
about the next tutorial is actually
applying a kernel working out that
kernel by hand and truly showing it
because we can actually do it with one
of the kernels or polynomial kernel we
actually work that one out by hand and
conceptualize how the kernel works and
then as we move on there's some kernels
that we really are not going to be able
to really conceptualize so we can't
really do it but anyways at least one of
them will do so you can get the idea and
then finally we will move over to Python
and actually show an example of kernel
at work so in summary a kernel is really
not something that's specific or unique
to the support vector machine it's just
a similarity function we can restate
many of the machine learning algorithms
or even create our own machine learning
algorithms purely with kernels so kernel
just takes two inputs and outputs the
similarity of them using the inner
product inner product is a projection of
let's say X 1 onto X 2 basically how
much overlapping do we have going on
there which is how and why it's
considered to be a degree of similarity
so we know that we can use a kernel to
help us transform our feature space our
X space because every interaction with
that feature space is an inner product
reaction so anyways if you have any
questions comments or whatever up to
this point feel free to leave them below
otherwise as always thanks for watching
things for all the support subscriptions
until next time