Why Kernels - Practical Machine Learning Tutorial with Python p.30 Video

what is going on everybody and welcome
to part 30 of our machine learning with
Python tutorial series in another video
for the support vector machine and
really kind of in general kernels and
all that so what we've been talking
about is how to handle for non linearly
separable data and up to this point
we've been talking about the kernel and
how we're going to do some sort of
magical thing with a kernel of going
into plausibly an infinite dimension and
doing some calculation there or actually
having a calculation done there for us
so we don't actually have to visit that
dimension and yet we can get back the
answer somehow so that all sounds really
hand wavy so I think it's important that
we can actually kind of work this out
and see how that actually works so that
we have a little more faith in this
whole thing so it's kind of crazy so
first of all when you're you're going to
write out a kernel generally let's say
you've got X you know if we're going to
call our feature set X and then we're
going to call the new dimension or the
new space our Z space so you're going to
have you're going to say usually
something like this you're going to set
K of and we're going to say X comma X
prime okay so just for the record that
that like slash looking thing is a prime
I'm going to do my best to keep it a
slick a really slanted slash so it's not
confused with an X supposedly ^ 1 but
anyway so X and X Prime in that K is
just your kernel and that is going to
equal this new z space where you've got
Z dotted with Z Prime like I said I did
not want that to be a 1 so maybe that I
don't know anyway that's a prime so
first of all what is Z okay right so you
can kind of think of Z as some sort of
function that's being applied to its X
counterpart so Z equals let's say a
function
of X and then Z prime is a also a
function of X prime okay and when you do
a kernel let's say a kernel on something
comma something what are you actually
doing you're doing it you're you're
running first of all you're going to
convert X to Z right the constituents
will be converted right this gets
converted to a Z this is getting
converted to a Z Prime and then what are
you doing well it's the dot product the
dot product or the inner product between
this and this whatever that happens to
be and that's a kernel as long as you're
doing those two things it is a valid
kernel and it's important to note that
function and function are equal to each
other you have to do the same thing
right you you can't just kind of do one
thing to X and then another thing to X
Prime it needs to be the same needs to
be symmetrical so keep that in mind as
well also before we move on I don't know
that I'll ever really show it again but
a lot of times you might see rather than
you know K of X or whatever you're going
to see the kernel being denoted as a fee
or fire however you want to pronounce
that so you might see it in an equation
like y equals W Phi X plus B okay
something like that you're going to see
and that's just going to denote that hey
this person is using a kernel and not
just the straight-up formal SVM
calculation first of all inherently we
can see right out of the gate again what
does a dot product or an inner product
what does that produce it simply
produces a scalar value that is it right
and so all we need from this magical Z
space
well really it's this one in this one
right all we need from that Z space is
actually the inner product right and
that is the kernel so the question is of
course can we calculate the inner
product of that z space without actually
knowing that z space and needing to
process that z space so first let's
consider a really simple feature set
it's just going to be a nice two
dimensional feature set kind of like
what we've been working with really up
into this point so let's say your
feature set is um and I'll even let me
write that out fee I'm not going to
write that out because that's messy so
your feature set you're going to have X
1 comma X 2 that is your feature set
your feature set is a vector and then
let's say we want to take ourselves out
to some sort of Z space and the way
we're going to do it is with the
second-order polynomial so we're going
to convert this to a second-order
polynomial so how are we going to do
that well when you do that you're going
to start with 1 then you're going to
just take the X 1 then comma X 2 right
so this is going to create a new vector
for us right and this new vector is
going to come out to be six dimensions
right so you're going to start with one
for second-order then it would be just 1
X 1 X 2 and then X 1 times itself X 2
times itself and then X 1 times X 2
that's all you do to just we're going to
take it out to that second-order right
so so let's go ahead and write that out
it probably would have been easier the
first time anyway but so so first of all
our our feature set like let's say let's
say those X's are I don't know little
X's but the big X right that feature set
is equal to x1 x2 and all that then
we're going to say this new z space is
going to be Z equals and again it's just
a second-order polynomial so we're going
to
take it out - and actually I think we'll
go we'll go to another page so I'm
saying X equals I just wanted to have
some more space here so I can fit
everything hopefully so X is equal to X
1 comma X 2 this is a vector and then
we're going to we're taking out to the z
space and we're going to say actually
Z and Z is just a second-order
polynomial of X we've already kind of
explained that now we're actually write
it out so that's just going to it's
going to start with the number 1 comma
again this is going to be a vector it's
going to be common than just the first
two original values that we had so that
would just be X 1 comma X 2 comma then
it's going to be the first value
multiplied by itself so just just X 1
squared okay X 1 squared let me do the
same thing so X 2 squared comma and then
it's just X 1 times X 2 so X 1 X 2 and
again this is our new Z space so this is
our new z space where Z this new Z space
that the function that we've run here is
second-order polynomial okay and I'd
already went through that logic but it's
just one you know the first value the
second value the third value or really
the fourth value will just be the square
you know the first value times itself
then the second value times itself and
then finally that sixth final value is
just the first value times the second
value okay second-order polynomial so
that is our z space and remember we said
that K let's we'll write it out we said
K of X and X prime and sometimes you
might see k of x and y or some people
might even write x1 x2 okay just don't
worry about the notation just understand
when you see K and then two values
that's a kernel so K recall K of X and X
prime we said was going to be equal to
the dot product of Z and Z Prime right
and Z and Z prime are going to be
identical right because they're they're
going to be the same this same notated
vector at least so so what is the dot
product of Z and Z prime well what is
the dot product what is the inner
product well it's just each value times
itself so in theory we would know we
would know Z prime basically I'm just
going to write it out really quick but Z
prime is equal to this right it's just
rather than x1 it would be I suppose
I'll write it out so that's Z and Z
prime same thing it's just you know
obviously just throwing the prime and
then K X of X Prime
what are the two things we're doing
we're applying first that function to X
which we've just done to convert it to Z
but then that final thing is it's just
it's the dot product of Z and Z Prime so
you can really think of it now as K X X
prime equals Z dot Z Prime right and
what does that equal
well recall the dot product is just eats
can each kind of constituent of the
vector multiplied by that same place
constituent so what we're going to have
here is you know 1 times 1 so that's
just simply 1 plus and then it's X 1
times X prime 1 so X 1 times X prime 1
plus X 2 X prime 2 so X 2 X Prime
2 plus X 1 squared X prime 1 squared X 1
squared X prime prime
1 squared + X 2 squared X Prime 2
squared + X 1 X Prime 1 X 2 X Prime - ok
so that looks obviously really messy
those Prime's are pretty hard to truly
denote out here but hopefully if you
follow it along you got the whole primes
and stuff like that if not you can check
out the text-based version on Python
programming at net and I'll use
something like latex or something like
that to make it very nice and neat so
it's totally obvious what's prime and
what's 1 and all this but hopefully you
recognize that you would never write ^ 1
so what we've done here is actually we
have gone to the z space right because
you and i went to the z space and
actually performed the calculation
because we actually did the conversion
to the z space and then performed a dot
product of that conversion to get our
answer okay and as you can see that was
actually a really long and very tedious
my hand kind of hurts after writing down
all of those things right and so your
processor also kind of hurts because
that's a lot of stuff so the next
question is can we do this kernel
without actually writing it out or
without actually having to visit this z
space and the answer is yes and this is
again this is called the polynomial
kernel so the polynomial kernel is the
following so you're going to say K X
comma X prime and this equals 1 plus X
dotted with X prime to the power of P
you can have that power of P be whatever
you want so
so basically to write this out in
further notation let's say you've got a
number of dimensions and then P is
basically whatever you want it to be
what we just did a second ago was P
equals two and four will say however
many dimensions you want but let's say n
and n was also equal to but you could in
theory have P equal many more but first
let's let's write this one out and then
we'll we'll consider the difference of
adding and changing and manipulating P
and n here momentarily so first of all
do you see any Z in this and either of
these calculations I don't if there is
no Z ok so what would happen though if
we for example let's say we wrote out
let's expand this first so this would be
basically 1 plus X 1 X 1 Prime and then
basically it's Loess dot dot dot all the
way out to X n X prime n to the power of
P whatever that happens to be now
imagine though how much more difficult
is this equation if we changed n to 15
and P to a hundred now obviously you
know by hand or something that would
make this significantly more challenging
but by a let's say a calculator by
Python or or something like that by a
processor how much more difficult is
this equation not much more difficult I
mean it's it might it might cost you an
extra cycle or a few cycles or something
but for the most part that's not that
much more difficult as opposed to here
where we're actually going to the z
space and in this case that was just a
second-order polynomial this could get
way more complex
we're just simply going to like in this
case we've we've gone to the z space and
done the dot product and found our
answer and it's it's pretty ugly already
just looking at it whereas over here we
actually don't have to have to do that
much and in in reality if we were to if
we were to work this out we would find
that we we had the exact same problem
and then you can even you could pull
this down into their own little separate
separate vectors and it might give you
it give you a vector like a 1 comma B x1
squared x2 squared square root of 2 by X
square root of x whoops square root of 2
X 2 and then the square root of 2 X 2
times X 1 X 2 and it's important to know
that so that square root of 2 is simply
avoiding it's just a square root of 2
times X 1 times X 2 so keep that in mind
so what would happen when this is a
vector and it was multiplied by itself
what would you get in return like what
would that scalar value be right well
that would be your second order
polynomial dot product between the two
x's having been transformed to the z
space dot product those two things you
have the exact same scaler except you
went through a lot less work so another
kernel is going to be our RBF or radial
basis function kernel it's a lot more
complex and you can't really
conceptualize it as easily as we could
the polynomial kernel regardless the
kernel is K X X
crime is equal to the exponential so
it's an exponential function of the
negative gamma times the absolute value
of X minus X prime squared okay so in
theory this can take you out to infinite
dimensions and just in case anybody's
not familiar X you know an exponential
function like this so e XP x equals e to
the X so uh what's happening here is
plausibly you could go out to an
infinite number of dimensions now as you
may guess that could get you into a
little bit of trouble also
it's infinite dimensions I struggle to
say fundamentally but in reality it's
it's really not not really I guess I
it's hard to explain you can save that
for your doctoral dissertation if you'd
like so anyway one problem though that
you might get by going out into an
infinite number of dimensions as you
might guess is that maybe just maybe you
might wind up maybe your data is
shouldn't actually be there maybe
there's nothing there in your data but
if you're willing to go out to an
infinite number of dimensions it's
likely that at some point you're going
to find some sort of linearly separable
case and when your data really really
wasn't meant to be linearly separable in
the first place so in the next tutorial
what we're going to be talking about is
when you're using a kernel to kind of
force your data into some sort of linear
separability when you do that how might
you know when you've made a mistake and
then also not only that what can you do
to avoid this exact same mistake which
actually can be pretty common even with
a polynomial kernel but really RBF and
just for the record are
be F or radial basis function this is
your default kernel this is the one that
pretty much is going to be used
depending on whatever library use but
with scikit-learn for example the
default kernel with the support vector
machine is the radial basis function
simply because it's going to work most
of the time
fundamentally again and theoretically
you can actually kind of you can make
data sets that RDF will not be able to
linearly separate it's it's possible
mathematically to do that but in all
reality is almost certainly unlikely to
ever be the case that RBF won't be able
to separate your data so so anyway you
can you can get into trouble like that
so you want to watch out when using the
RBF kernel so anyways of course though
after you know just because you fit a
support vector machine you know you've
optimized your support vector machine
you can also one way one way you can
find out right away is what will going
to be talking on the next tutorial but
also once you test it and you know you
might actually have optimized it you go
to test it and you have you know fifty
percent accuracy and you're like what
the heck that's cut that's probably why
you ran into some trouble as what we'll
be talking about in the next tutorial so
anyways and what we'll be talking about
might help you remedy this or maybe your
data actually just is not separable it's
not good data so anyways if you have
questions comments concerns up to this
point feel free to leave them below
otherwise as always thanks for watching
thanks for all the support subscriptions
in until next time