SVM Parameters - Practical Machine Learning Tutorial with Python p.33 Video

what is going on everybody and welcome
to the 33rd machine learning tutorial as
well as the final tutorial in the
support vector machine section and so
what we're going to be talking about is
one more concepts and then we're going
to kind of do a bit of a review which
actually is going to cover a few more
concepts but also kind of bring you up
to speed with like the state of the art
of how people are actually doing support
vector machines these days so without
going get started we're going to first
talk about classifying when you have
more than two groups to classify into so
we're head over to my awesome artwork
here and you have two types generally
that you're going to use or two
methodologies that you might use when
you have more than one or more than two
groups that you're going to classify
into so the support vector machine is
simply a binary classifier so it can
only separate two groups basically at a
time or per decision boundary so the two
methods you have is OVR
which is one verse rest so that would be
one group versus the rest of the groups
and then you've got ovo which is one
versus one so let's go ahead and pick
those and first i'm going to start with
OVR which is just one versus rest so
let's start with group number one here
so it's not totally obvious I just have
feature sets each class is denoted by a
number and a color and their feature
sets being graphed out into vector space
here so let's start with the ones what
with one verse rest what's happening is
you're basically saying okay I want two
separate ones versus the rest of the
data so we're actually separating the
ones from the twos and threes so we're
going to have probably something like
probably this is a support vector and
this is a support vector so I don't know
something prolly around this would be
you're separating hyperplane okay so
that's the ones from the rest and in
fact let me get rid of those because
it's going to get kind of messy so okay
so that's the ones from the rest now
you're going to have the twos from the
rest and again that's going to be you
know you probably have like this as a
support vector this is one this is one
so this separating hyperplane might be
something like this
and then you're going to have the threes
versus the rest and that's a you know
something like this okay
so now you've got all your separating
hyperplanes now what's one of the
problems with one verse rest well it has
to do with a sort of weighting issue and
that is that when you have basically in
this case let's say you've got this is a
very balanced data set we've got three
groups that we're trying to classify and
each group has four samples in it so
they're all equally balanced but each
separating hyperplane is a little
imbalanced because you've got threes
versus eight so you got four data points
over here but on this side you actually
have eight total data points so that can
be a little more challenging to kind of
figure out which is which now the next
thing though is with one versus rest or
one versus one rather so let me clear
this up and one of us rest is generally
the default that you're going to use but
you really you're going to see people
that use either one but now let's cover
one verse one
so one verse one what it's going to do
is it's going to have a separating
hyperplane between each group so it's
going to be one versus twos and one
versus threes so let's do a hyperplane
for the one versus twos so it's probably
gonna be like this and then you're gonna
have another hyperplane that separates
the ones from the threes so it's going
to be like this and then you're going to
do the same thing it's going to be two
versus ones well we've already got that
hyperplane twos versus threes okay we'll
do this great then again we go to the
threes we say threes versus ones well we
already have that hyperplane two and
then the threes versus the twos turns
out we already have that hyperplane as
well so then when you have a data point
you might have something like this right
yeah it's a question mark trust me
so if you're a data point over here it's
really unknown at this point and the
question is well it's on this side of
the the hyperplane so it might actually
be you know a two right it's on that
that separating hyperplane between the
ones and the
so it might be a - but then we get there
and we're like hmm well what about twos
versus 3s well it turns out it's on this
side of the hyperplane there so it's
probably a tune okay and so you can do
stuff like that and this one's a little
more complicated than one versus rest
because one versus rest as soon as you
find out which side of which hyperplane
it's on you pretty much have your answer
but even then you're going to have cases
where you've got to have it's going to
have to kind of go through this chain
where it's definitely not a one but it
could be a two or a three or something
like that
so anyways most of the time you're going
to do a one versus rest but one versus
one is still valid as well it usually is
a little bit more processing to do one
versus one but it's more balanced so if
one versus rest is not working out for
you you can check out one versus all and
just kind of see okay so those are the
separating you know basically one more
concept that we had to draw out
basically but how did how to work with
more than two classes if you if you have
them and most or many real-world cases
are going to have more than two classes
so you might as well figure it out and
really the support vector machine really
came bad that's most recent rise was due
to character recognition so you know you
might have you at least got you know
zero through nine so ten ten numbers or
you might even have like 26 letters in
the alphabet times two for the casing
and all that so anyway obviously we need
the support vector machine to be able to
classify more than two groups so moving
over I'm going to bring up the SVM
documentation so this is just if you
google SVM SVC you can get here you can
also just kind of follow along as I run
through this so I'm going to run through
each of the parameters and then we'll
talk about attributes really briefly at
the very end of that but I'm going to
come down here and run through them in
this order so the first parameter here
is C so this should tell you right out
of the gate I mean it says it right here
and also in the first sentence but this
is a soft margin classifier if you want
a soft margin classifier to be hard
margined you can do that right you can
you can increase or decrease C however
you want right and we already kind of
explained how that would affect and just
for the record let me pop back over to
our
or tutorial where we talked about see
right so that see is this see here right
so how important is the impact of the
slacks or the errors or the violations
or whatever you want to call them how
important is that and that's kind of
what C does is it impacts that value in
that entire minimization function as you
minimize C it means the errors matter
you know less so that's C then you've
got a colonel and colonel is kind of
what we've been talking about in the
previous tutorials in the default is our
bf4 radial basis function but you can
also put poly in there for polynomial
you can put in the linear kernel so just
straight up linear kernel if you wanted
you can have sigmoid pre-computed or
even a callable and you so you can have
custom kernels should you desire to go
down that route degree this is just
basically for that yeah the polynomial
kernel what what kind of power do you
want to raise that kernel to gamma don't
touch it it's just it's that gamma value
in RBF and all that and if you want to
mess with it go for it I would leave it
as Otto and it even tells you that's
just going to be one divided by the
number of your features if you want to
mess with it you know go ahead but I
would highly suggest you learn more
about radial basis functions before you
go messing with that coefficient 0 this
is that independent term in your kernel
function again I would just leave that
as 0 but you can mess with that if you
want probability so this is kind of
where we first remember we were talking
about the K nearest neighbors algorithm
and one of its cool features is that not
only can you have a total total model
accuracy you can also have a degree of
confidence in whatever vote that you you
might cast the support vector machine
doesn't inherently have something like
that but it can be done so we can do
probability estimates if you did want to
check that out you can come down to the
scores and probabilities and you can
learn more about how they're actually
implementing cross-validation to figure
out what the probability score of any
prediction might be you do have to call
that prior to training if I recall
somewhere on here oh yeah yeah so prior
to fit rather but training so so just
keep that in mind but you you can do it
it's just going to cost you a lot and
again the real only downside to the
support vector machine is that that
large size data set is just going to
perform poorly on the SVM bringing us to
our next topic which is shrinking
whether or not to use a shrinking
heuristic the default is true so
shrinking if you recall where we were
talking about SML or sequential minimal
optimization and that was that we pulled
up that paper by John Platt from
Microsoft
many years ago over a decade ago if I
recall I think was like 2004 or
something anyway pull up that paper I
could be totally wrong on the date sorry
anyway the shrinking is involved with
sequential minimal optimization so if
you it's it's just on by default and you
honestly should leave it on there
because it's gonna things are going to
run much quicker in the way that it
doesn't is it basically decides which
feature sets can basically just be like
ignored from the optimization algorithm
because they're deems to probably not
have any impact next we get to tolerance
so if you remember when we did the
optimization how I was how we're the
question was okay how do we know when we
have actually found first how did we how
would we know that we've been we've
actually reached optimization so in fact
let's see if I can bring that up so this
was our code I guess from 28 was the end
of it and if we scroll on down basically
to here was our requirement right we
were saying if if we found that all of
our data was basically where you had Y
sub I multiplied by X sub I times W or W
dotted with X sub I plus B if everything
was greater than or equal to one great
but then remember how is telling you
what you can do before you go optimize
down another smaller step like here what
you could do is you could find out you
could query do we have does both do both
classes have feature sets that when when
run through this basically are basically
when we printed it out do both classes
have feature sets
can't remember if we did it yeah we did
it down here at least that where that
value is very close to one or even equal
to one but it's very unlikely that
you're ever going to reach a perfect one
now with the support vector machine
generally you're going to actually throw
in that -1 value so so it's actually
going to be more like y sub I times X
sub I dotted with W plus B minus 1 is
greater than or equal to 0 ok it could
go either way but in the support vector
machine with scikit-learn it's using the
the equation of y sub I X I W plus B
minus 1 greater than or equal to 0 so
that tolerance value that tolerance
value is where it's going to check to
see if both sides have a value that is
either 1 e to the negative or 1 to the
basically name so like point zero zero
one okay if both sides have a value of
that or less it's going to say yep we've
we found our optimal numbers and we can
move on so you can you can edit this
like if you wanted even more optimize
you could make that rather than 3/8 or
something like that but this is more
than fine next you have cache size this
is just the size of that kernel if you
remember when we were showing the kernel
in using that pretty sure the previous
tutorial that gram matrix kernel that
becomes your grande matrix kind of that
has all that data that we're running
through the kernel that can obviously on
a large data set get pretty large so you
can you can limit the size there but I
probably wouldn't unless you had a huge
data set but yeah you can run into
issues if you're going to decide to do
something like that you've got class
weight I would probably just leave this
alone but obviously you're going to have
you might have classes that shouldn't be
weighted the same there's could be a
time where that happens but and if that
is you can you can modify that verbose I
wouldn't touch that because you're
hopefully are working in the
multi-threaded context if you need it so
I wouldn't worry about that
max iterations this is another kind of
like going back to our tutorial here our
max iterations were what
let me find them here right
we had three max iterations basically
this is just going to go through here
here and then here and that's it that's
as many calculations it's gonna is going
to run this and however many so it's
like Thresh lates more that's not how
many iterations we run we're actually
running quite a few more iterations and
that so in iteration I suppose would be
maybe I guess would be the while loop I
don't know probably for the four steps
and step sizes would be so for however
many for loops actually ran you would
have maybe that I don't know anyway
because these are the step sizes and
then it's like so each of these would be
three times while not optimized however
many steps we might take so anyway you
could remember if I was I was telling
you with tolerance each time you decided
you hadn't reached that tolerance you
could take smaller steps right so you
could take smaller steps but that
process could go on till infinity
because actually finding a time where
you actually are you have both your data
sets have a value that's equal to zero
on both sides for those support vectors
is just unlikely to occur I mean it
might occur but it's very unlikely and
so you have tolerance to kind of say
okay if it gets you know but on this fun
this this value then okay we're done but
then also maybe you don't reach that
maybe you're trying to do a hard margin
on something that ought to be soft
margin you might actually never reach
those values therefore you can set max
iterations to say okay we're going to
only run basically this question here
for I on self data you know we're only
going to do this a million times after
that we're going to be like okay we are
done okay so so you can do something
like that but the default is actually
there is no limit but you can set one if
you want decision function shape one
verse one 1 verse rest okay or none
the default is none and pay attention
there's a couple of changes down here
new in version point 17 decision
function shape OVR
is recommended that that's the one that
you use deprecated decision function ovo
and none are being deprecated and then
apparently the default of none
well currently behave as ovo but it
raised deprecation warning but will
change - I think they're saying this is
going to change - OVR and 0.18 but yet
they're saying they're deprecating none
so I'm not really sure exactly what that
means but I'm pretty sure the default
will just simply be OVR I'm pretty sure
that's what I'm reading
but yeah anyway then random state this
is just for a random seed if you want it
if you're doing that probability
estimation I don't know if I really said
it but the the probability estimates are
going to cause really heavy load on your
processing so unless your data set is
relatively small like we've been working
with very small data sets so it really
probably doesn't matter to use
probability but just understand that
doing that on a large realistic data set
is going to likely be hard and then we
come down to attributes this is mostly I
mean you can check like for example the
number of support vectors so again if
you've got you know let's say you've got
500 total samples and you've got 400
support vectors with your soft margin
classifier with the RBF kernel that's
that's not very good right you don't
want that so it's always a good idea to
kind of check the number of support
vectors and compare that to the number
of samples also you can get the
locations of your support vectors and
you can get the things like W and B and
why might you want W and B well if you
want to visualize it right and same with
the support vectors you could grab those
to actually visualize the data set with
matplotlib or something like that so
that should be everything if you do have
questions comments concerns whatever
feel free to leave them below otherwise
we're going to be moving on into flat
and hierarchical clustering and then
after that neural networks and deep
learning and all that really fun stuff
so anyways questions comments leave them
below otherwise till next time