Soft Margin SVM - Practical Machine Learning Tutorial with Python p.31

what is going on everybody and welcome
to part 31 of our machine learning
tutorial series in the previous few
videos and really the previous mini
series we've been talking about the
support vector machine specifically the
last few videos we've been talking about
kernels and really we've been talking
about kernels and respect to non
linearly separable feature sets and
we're explaining how you can use a
kernel to translate your data into a
higher dimension do a dot product and
get the scalar value or do an inner
product more specifically actually do
the inner product get that scalar value
have it come back to you and actually
having had to never visit that whatever
that z space happened to be you don't
actually have to go to all those
dimensions and we at least exemplified
this with the polynomial kernel where
first we calculated it by hand to the
second order polynomial and just pulling
it out into the z space took a lot of
time and then doing the dot product also
took a lot of time in the z space
whereas when we actually did the second
order when we did the polynomial kernel
and kind of mimic to that second order
polynomial we saw that was actually much
quicker much fewer variables and doing
the dot product was just easier by hand
and it's also quite a bit easier by
processor and that was only second order
your order right so in that case we had
N equals 2 for the number of dimensions
and then P was also equal to 2 but
imagine it had we make mid P 100 and n a
thousand right would that have made our
polynomial kernel take much more
processing like exponentially more
processing no would it have caused an
exponential increase in just actually
right you know translating to the z
space and running that dot product yes
that would've been crazy so that's one
thing we've we kind of ended on the
radial basis function kernel and
explained how that can translate into
seemingly infinite dimensions and now
we're going to be talking about is where
that can you know problems that can come
up when you might do something like that
and one how to know maybe is your data
actually just never going to be linearly
separable or should you maybe try
another kernel maybe or something along
those lines so that's what kind of we're
gonna be talking about here and
specifically we're going to introduce
the soft margin support vector machine
so imagine you've got this data set I've
drawn up here and looking at it you know
is this linearly separable not know
right there is no straight line that we
can draw that will do this but imagine
that we run the RBF and we come up with
a decision boundary that probably looks
know something like I don't know
something something like that right and
in this case we're going to have our
I'll do an in a lighter green the the
actual support vector hyperplane so you
know coming through here probably comes
down here comes up here goes out here
and then here you come down here maybe
cross into that one okay and you go out
okay so so that would be in theory via
the radial basis function and now let's
go ahead and just circle all of our
support vectors here Bam Bam Bam Bam Bam
Bam Bam Bam Bam this one this one and
this one and it's out of this entire
dataset only two feature sets are
actually not support vectors okay
that is a major red flag why is that red
flag well it kind of signals that we
probably have some over fitment going on
here okay so if we can find a function
that like you know fits all of this data
we're probably in some trouble here
so so so instead of maybe doing what
we've just done I will put us back to
just have that that single decision
boundary what if we drew another one
that right almost you know basically a
straight line that separates the data
and if we're to look at this we can see
okay we've got a few violations right
you've got maybe this violation here
this is a violation
and basically yeah those just those
sorry punch my microphone punch you guys
so so you've got these two violations
here otherwise everything else fits this
line and again what if we actually drew
the the support vector hyperplane again
we'll just kind of make it a little
lighter we go here and then probably
here right and again if we circle the
support vectors this time we have two
here and two here and this was
dangerously close to being a support
vector but just understand this is a toy
problem I've just kind of drawn it and
not everything I've drawn is perfect but
anyways so in this case right you've got
two support vectors on either sides for
total support vectors and we also have
just plenty of data data points here
that are not support vectors right so
much fewer of our data is our support
vectors and that's that is in theory
better because you have less over
fitment going on so why is over fit you
know and over fitment is just a
statistical thing like you that's just
inherently bad but what is going to end
up happening like if you over fit to
historical data it's just likely that in
the future data you're just not going to
be just right because the future is
going to have noise it's gonna it's
going to be different than the past data
so if you fit that past data perfectly
you're going to screw up in the future
data so you'll know this though if you
do training and testing which is
something we've already covered up to
this point but the real question becomes
let's say you train and then you test
and you have 52 percent accuracy and
you're like wow geez that's that's no
good and then you're kind of wondering
hmm is or my is my theory wrong is this
something that's not going to work or
maybe should I use a different kernel or
something like that and you might not
really know one thing you could do is
take a shotgun approach and just use a
different kernel see what happens right
but the other thing you can do is just
query to see how many support vectors do
you have so maybe number of support
vectors and then divide that by number
of samples and if that's a large number
right you know the closer to one that is
the worse off you are and probably if
you're in reality on a real data set if
you're more than you know 10% let's say
you've got more than a hundred data data
points or feature sets rather samples
what everyone call it you got more than
a hundred probably 10% maybe maybe you
can go up to 20% but really probably 10%
anything more than that
you that should signal T you've done
some sort of over fitment so it might
not be the case that your data is not
linearly separable it might be the case
but you don't know yet but maybe you
should try um try different a different
kernel like let's say you you come back
with 52 percent accuracy a 52 percent
accuracy and you have I don't know
eighty percent are support vectors okay
if that's the case you've got some over
fitment going on maybe try a different
kernel but if you've got fifty-two
percent accuracy and only eight percent
of your data is our support vectors your
data is probably not going to work out
you can still try different you know you
go ahead try different kernel but but
probably something else is right so what
we've got here - all of this mess nice
and clean now what we've got here is
when you have some sort of let's say a
separating hyperplane where you actually
have some data that is kind of in
violation of that separating hyperplane
what you have is most likely a soft
margin classifier now in this case most
likely we wouldn't accept a soft margin
all the way up to this point but keep in
mind this is just for a simple example
for now so so what's going to happen
with the soft margin a soft margin
support vector machine what you're going
to have as opposed to just by the
obviously it should be an intuitive here
but the other one would just be a hard
margin like this Green Line this is a
this is a hard margin classifier right
that would be a hard margin
and just like the RBF is kind of like
the default colonel it's actually the
case that a soft margin classifier is
going to be your defaults because again
in the real world a lot of your data is
not going to be perfectly linearly
separable without some degree of
overfitting it so anyways so with with
with the soft margin what you're going
to have is you're gonna have some sort
of degree of error right so so kind of
like with with regression you know you
the distance right from the hyperplane
let's say to the data point that's going
to be your error right and and how might
we we allow some degree of error well we
have what we call slack okay and what
slack is going to be for us is generally
we're going to denote slack this little
symbol here and what we're going to do
is we're going to say okay remember
before that constraint basically where
we were we were saying that y sub I X
sub I dotted with W plus B and then it
was either minus 1 is greater than or
equal to 0 but in this case let's say
greater than equal to 1 okay and in this
case this was the original kind of
equation but now what we can do is
introduce this slack variable and say
okay Y sub I multiplied by X sub I
dotted with W plus B really just needs
to be greater than or equal to 1 minus
our new slack variable right so so we're
giving it some sort of leeway here now
of course it would slack obviously needs
to be let me just write it so I'll just
write it up here write slack must be
greater than or equal to I was almost
going to write it in Python terms but
anyways has to be greater than or equal
to 0 right it has to be a positive
number and if it was equal to 0 then
that would just be hard margin but the
more slack you give it the more soft
your margin is going to wind up being so
now you might have a total slack so if
we were going to say
hey what is the total slack well that
would just be the sum of I over that
slack variable of I write it's just the
this the sum of all the slacks okay so
yeh mathematic notation so obviously we
probably you know you might want it you
might choose like hey yeah we want to
allow for some slack but is it not kind
of philosophically and logically correct
that we probably want as little slack as
possible right we we would want to
minimize the slack and obviously that's
you know that's a scary a scary turn to
be thrown around here but yes we we we
really probably want to actually
minimize the slack so so how do we throw
that into our actual equation remember
in the in the old days when we actually
did the formal calculation of the
support vector machine we wanted to
minimize not many minimize and we wanted
to minimize what we want to minimize the
magnitude of vector W and then we said
just for ease of quadratic programming
we also since you want to minimize the
magnitude of vector W you also want to
minimize the one-half of the magnitude
of vector W squared right that's just
yeah that just works so then what you're
going to do is you're going to say plus
and then you're going to do this this
new thing that we just have right here
the sum of I over the total slack
basically the total sum of all the
slacks so now you're going to have the
sum of slacks I and I and then you're
going to have this new value and this
new value is C so now before it was just
to minimize this but now it's actually
we want to minimize all we want to
minimize all of this
okay so we've already explained what
slack is we already know what the eyes
are we already know the sum of all the
slacks and stuff so what is see I mean
just kind of looking at this equation we
want to minimize one half basically we
know we want to minimize the magnitude
of vector W okay we already kind of know
what that's going to do and then we want
to but really we want to minimize now
we're trying to minimize this whole
equation so no longer are we just trying
to minimize the magnitude of vector W
we're trying to minimize the magnitude
of vector W plus C times all of our
slacks right add it up okay so sometimes
a lot of times people are very confused
of what C actually does but if you look
at this this optimization problem it
should be pretty obvious like what if we
what if we raised what if we raised the
value of C here well if we raise the
value of C we're saying we want less
violations because there's really no way
like if we're going to try to minimize
this equation the more we raise C the
more we're going to punish for
violations of the margin and if we lower
C we're going to be more allowing of
violation because you can just you can
just see like right if you if you raise
the value of C that's multiplying this
right here so that's just making the
equation much larger and so another way
to kind of look at the value of C is to
because obviously as you as you as you
change C like C and slack it all of
those things have absolutely no relation
to the magnitude of vector W so except
in this this little minimization
requirement optimization problem so so
because of that C is actually kind of
like a it's where you get to decide how
important slack is in relation to
minimizing the magnitude of vector W
right so it's kind of your way of kind
of
you're tottering between you know
whatever personal values you might have
in the specific example that you're
running so C is something that you're
going to kind of set as hard you're
going to say okay I want C to be equal
to 1 or 5 or 0.5 or 0.2 or whatever and
obviously the smaller you make C the
less it you know the less it matters
what the slacks are right the less it
matters what those errors that violation
the less it matters right because the
smaller you multiply this these you know
that the sum of all of the slacks the
smaller that is your you're minimizing
the equation right just by changing C a
little bit so that is what C is that's
what slack is and that's kind of what
that's what the soft margin classifiers
doing and again the kind of purpose of a
soft margin classifier is so you don't
over fit your data and in most cases
that's what you're actually going to be
running is a soft margin and classifier
because I most real examples are just
simply not linearly separable and if you
take it to the number of dimensions
required to make it linearly separable
you've probably done some over fitment
so you know one way you can kind of test
this and compare it is is do a soft
margin classifier let's say in set C
equal to like a million right and then
see how accurate you are and then
compare it on that same data set where C
equals one right I'm pretty sure in in
in scikit-learn the default for C is one
but we'll look at that shortly but
anyways that is the soft margin
classifier now up into this point we
have not really we have not done any
Python for kernels we haven't done any
Python to show what I was talking about
even just like what if you didn't want
to use a kernel and just translate into
another dimension could you how does
that actually work to so visually seeing
how translating to another dimension it
can help you you know just to
conceptualize how all of this works so
in the next tutorial what we're going to
do is we're going to finally pull out
Python and we'll show a quick example of
translating to another space then we're
going to pull out
Colonels and CVX opted from the
quadratic programming and after that
we're going to kind of bebop back on
over to the scikit-learn version of the
support vector machine we're going to
run through some of the parameters there
and kind of point out a lot of the
things that we've covered so far along
the way and most likely most likely that
will be the the final support vector
machine tutorial and we'll be moving on
in to clustering and then after that
deep learning so anyways if you have
questions comments concerns whatever up
to this point feel free to leave them
below otherwise as always thanks for
watching thanks for all the support
subscriptions and until next time