Soft Margin SVM and Kernels with CVXOPT - Practical Machine Learning Tutorial with Python p.32 Video Lecture | Machine Learning with Python - AI & ML

what's going on everybody and welcome to
part 32 of our machine learning tutorial
in this tutorial what we running through
is some example code working with CVX
opt and working with kernels being
applied to our support vector machine
what kind of impact that makes and I'm
and also some pretty nice visualization
so this is not my code I grabbed this
from Matthew blonde Belle's github also
there's a nice blog post on this exact
how you kind of came up with this also a
believer link possibly no but also at
least a reference to Christopher bishops
pattern recognition and machine learning
book in this kind of models some
information from that book as well so
you can check that out if you want to
learn more about working specifically
with cbx opt I will just say though CVX
op it's useful in this case just purely
so you can see directly the impact of a
kernel and where it's actually being
injected and chained and modifying the
initial formal support vector machine
but beyond that honestly CVX opt is not
something you're probably ever going to
use and even like for a support vector
machine if you wanted to write your own
you would almost certainly be using live
SVM so just keep that in mind but mostly
this is for educational purposes for
looking at the kernels how they effect
and also visualizing nonlinear and also
visualizing the soft margin you can do
that in scikit-learn too no problem
but you know it's just a little harder
to actually see the kernels impact
anyway hey there you can check that out
also there is this I'm pretty sure we've
already linked to it before but anyway
see this is the actual quadratic
programming solver that we're going to
use it just does this really bland
equation basically where you just do
stuff in whatever values for these
variables you want so it's just
minimizing one half X to the power T P
times X plus Q to the power T times X
subject to so these are the constraints
G of X is less than or equal to whatever
you want H to me and then a times x
equals B simple enough so anyways that's
the solver that it runs
and links them hopefully description but
also in the sample code so or the
text-based tutorial also you can check
out there's this from actual CV exept
org just solving a simple quadratic
programming problem so you can kind of
check that out with a really simple
example of how you might solve you know
push everything through but you can just
look at that right now and kind of get
the gist also there's an even larger
tutorial right here I'll link to this
one as well from MIT and it's just four
pages but it has a it's a little bit
more in-depth as far as how the solver
actually works so if you're interested
in in specifically learning that check
out this this little tutorial here again
links in description and on the
text-based version so we're going to run
through this code really quickly but you
can see here we've got import numpy as
NP and then we're bringing in lineage
specifically we just use mountain pile
in out but he wanted to have that extra
savings of NP dot alright import CVX opt
and then the solver we kind of already
talked about that then here's your
kernels really simple we didn't really
talk about the Gaussian kernel but you
can look that up and you can also just
see it's in PE exponential minus Lin
Alec norm X minus y the power 2 divided
by 2 times Sigma whatever it is in this
case 502 the power to write but anyway
the polynomial kernel should look really
familiar one plus the dot product of x
and y we use rather than y x prime but
doesn't make a big difference and this
is just to the power of p where the
default is three but you can put
whatever you want linear kernel that's
just this the original formal criminal
it's got these SVM objects no problem
the initialization method just simply
sets the kernel set C defaults to none
and in this case if C is none this is a
hard margin support vector machine so if
you want to be well if you want it to be
soft margin you just set C to something
alright
whatever you want that penalty parameter
to be parameter to be coming down to the
fitment
basically he's just creating all the
values for that quadratic programming
solver again this would be you know if
if it is a soft margin then great and if
not then we're going to use these other
values solves the problem gets your
solution
get your alphas here find your support
vectors all that tells you how many
you've got finds the intercept which is
your be your bias whatever you want to
call it finds W great projection this
really is only useful for as far as I
knows where is getting your prediction
and sure enough that's basically all the
prediction method does is actually just
does projection and just returns the
sine of that projection so nothing major
they're bringing in pylab here really
showing its age but anyway brings that
in I think this is from like 2010 anyway
brings in pylab okay and then generates
linearly separable data so this will be
data that you can do no colonel and it
can be a hard margin classifier
this one's non linearly separable data
so this would be data that needs a
kernel right then we've got not
generating linearly separable but
overlapping data this is data that needs
a soft margin okay this splits up
training data this makes testing data
and then this is just some nice code
that will plot a linearly separable data
this one is for anything that's going to
have a kernel that's going to be all
kind of contoured around and then here's
some functions to actually test all of
that stuff I've just went ahead and put
them down here so let's just run through
it really quick so testing linear all
he's going to do basically is just
there's no penalty parameter right so
it's hard margin and the default kernel
was a linear kernel come
mousewheel right so if you wanted to
change that you could change that if you
wanted but it's just defaults to linear
kernel so we can go ahead and run that
really quickly sure enough there you
have it you've got your support vectors
here a nice low number of support
vectors we're pretty content cool
successful it's a simple simply
separable data set as well anyway and
then coming on down the nonlinear I
think by default nonlinear was using the
Gaussian kernel I just changed it to
polynomial since we didn't actually talk
about the Gaussian kernel doesn't really
matter but just note I did make that
change so we're just using a polynomial
kernel rather than the original but as
you can see you still get corrected or
correct separation of that data and one
thing I guess I'll show it really
quickly if you want like a visualization
of like how that we're like how do you
translate to another dimension and then
pull it back down to two dimensions like
so how do you go from three dimensions
let's say down to two dimensions and get
those contours a good thing that Google
is SVM kernel visualization I'll go
ahead and pull it up but you can look at
it yourself - let me pull up and change
the speed to pull this on over and we'll
play that so that's your two DS data set
and he's about to transform it there it
is into three dimensions and then it's
linearly separable so we slide in that
hyperplane great and then you can see
you've added that just basically the
circle that it passed through and then
you can see like basic at the end of the
day back in two dimensions it's just a
simple circle for the separation so it's
kind of a useful visualization for for
how that process actually works and then
finally we'll do hit go ahead and do the
soft margin run that real quick okay so
as you can see it is totally linearly
separable data it's just that there's a
little bit of overlapping going on there
not a huge problem still don't have too
many support vectors so we're doing fine
without it being soft margin this
just simply not linearly separable but
you could you could have a kernel that
solved that problem most likely so
anyways that's all the examples again
you can kind of poke on through and just
see how the kernels are affecting it and
in fact you can even come up to well we
can do yeah okay so right right here
UNC self kernel is always going to be
used it's just the case that if it's a
linear kernel it's just the dot product
of the two values then you've got I
believe there should be probably one
more kernel in the prediction most
likely oh well the weight vector okay
yeah if it's a linear kernel then it's
just zeros but if it's not self W is
just not in that case oh that's it
making sure we actually could okay so
anyways um I knew there was no surely in
the projection okay I don't know why I
wasn't finding that but anyway I had to
go down there myself
so yeah obviously in the projection you
also have to take your that feature set
you're trying to predict and translate
that as well grab that kernel but anyway
or kernel value I suppose so
that's just like a quick one through the
code if you have questions or whatever
feel free to leave them below on this
code in the next tutorial will kind of
wrap we will for sure wrap up the
support vector machine we're going to go
through all the parameters from for
scikit-learn x' support vector
classifier because one it's a decent
review but two it actually will help us
cover a few more concepts that we
actually haven't covered and kind of
bring you up to speed as far as what
people are doing nowadays
for solving the SVM so that's pretty
useful also we'll talk really briefly
right at the beginning about you know
what do you do when you know support
vector machine is supposed to be a
binary classifier so can you
so how do you classify more than two
classifications like let's say you've
got three classes how do you actually do
that so we'll talk about that and then
we'll launch into the SVM parameters and
all that so anyways if you have
questions comments concerns whatever
feel free to leave them below otherwise
as always thanks for watching thanks for
all the sports subscriptions and it til
next time