SVM Training - Practical Machine Learning Tutorial with Python p.26 Video

what is going on everybody and welcome
to another support vector machine from
scratch tutorial as well as part 26 in
our machine learning tutorial series
where we left off we are building our
own support vector machine and we are
currently going to be working on the
fitment section which is the training
it's also the optimization problem like
basically the entire support vector
machine is an optimization problem where
all all we have to do is figure out what
W and B is and then from there to
classifies we've already written it is
that classification right there it's
just the sine of X dotted with W plus be
super easy where X is the unknown
feature set so we're going to work on
the fitment and that's the optimization
part and I just want to say right at the
gate this is not gonna win you any you
know Emma Stone petition or anything
like that so it's just meant to be a
really simple example for how you could
go about solving a convex problem but if
you really want to know more about
optimization there's other sources you
should check out besides me I can't
teach you optimization to any huge
degree luckily I do have some sources
for you I'll just pull this down here so
first of all we this is a convex
optimization problem and Stanford has
this lovely gigantic book 730 pages PDF
but you can just save it or whatever and
it's all on convex optimization so you
can bebop around here and learn a bunch
of cool interesting things and
specifically you'd be looking for
quadratic problems as well as that
lagron
dual problem moving along I also want to
draw your attention to the s mo so we're
going to be what we're going to be doing
is extremely rudimentary and it's one of
the earlier kind of ways of solving the
problem but as you'll see here and maybe
by the end to the larger your your your
data set the longer it's going to take
to do the fitment process because you've
got it you've got to check every
variable to see does it suit the whole y
sub I X I W plus B greater than or equal
to one question that has to be applied
to every single data sample or at least
that's the theory we're here with s mo
the ideas that you can kind of do it in
a sort of batch system
but anyway I just want to draw your
attention to that this is some
information also I think this is the
maybe the original research paper on
that question anyway you can check those
out if I forget to put these in a
description someone remind me and I'll
add them there also hopefully we'll have
them in the tutorial itself the text
version of this tutorial
finally I just also we brought this up
already too but the CVX Optima Joule
this is the page and stuff like that
specifically for convex optimization
sure enough yeah convex optimization and
what you'd be looking for here is
something dot QP probably
anyway we're not going to be worried I
think it's convex absolutely anyway just
drawing your attention to there are more
sophisticated ways but the module of CVX
op they encourage you to download it and
look at it yourself the source code and
see what you think
so anyway moving this stuff out let's
get go ahead and get started on this
optimization so first of all to begin
training or fitting and optimizing all
that fun stuff we're going to first grab
our data so we're just going to say self
dot data equals data so now the entire
algorithm can reference data which will
be important when we go to visualize it
so so if that data equals data and then
we're going to have we have this opt
dict and I kind of already explained
opps ticked we called it I think mag or
mags maybe anyway the optic is going to
be basically the following so you've got
your dictionary and it's going to be the
magnitude of W will be our key and then
our values will just be a list where
it's W and B okay that is how this
dictionary will be populated it's going
to start off empty though I'm going to
get rid of this pass so now what we're
going to do is we're going to define
transforms and let's go ahead and just
define them and I'll explain them but
you probably figure it out before we get
done if you've been following along so
is this in this okay so the transform
and we'll just do negative 1 negative 1
and actually we'll do them in the same
order so negative 1 1 then negative
one negative one and then one negative
one okay so these transforms are what
we're going to apply to the vet yeah the
vector of W as we like step each time as
again we've this has already been
explained so I'm not going to spend too
much time on it but each time we have a
vector we want to transform it by these
are basically get the product of these
all of these values same thing and and
at eat for each step that we step down
because for magnitude it doesn't matter
if it's negative or positive because the
great neutralizer of squared but when we
go to get the dot product of something
with a negative and B or negative let's
say in X when we do that dot product
will that negative make a huge
difference absolutely just like the
transform itself makes a difference now
what we're going to do is kind of write
a dirty function or a dirty for loop
here I'm trying to think of a better way
that could possibly do it but we'll just
we'll do this for now this is going to
be the reason why we're going to do this
is so we can get maximum and minimum
ranges so both for our for our graph but
then also for where we're going to start
when we value when we start with a value
for being a value for W and start
stepping we're going to use this
information if you can think of a better
way I'm all ears this is just the best I
come up with on the fly for now so
anyways we're gonna say for y I in self
dot data so I'm just gonna I'll try to
use as close to the fundamental names as
we did before but why I is just your
class then we're going to say for
feature set in self dot data Y I so
we're reference st. whoops ye Hawkeye so
now we're the feature sets now we're
gonna say for feature in feature set all
data dot append feature and then we're
just going to say self dot max feature
value equals max
all data and then we're just going to
basically copy this paste that and then
rather than max now we're at min don't
forget to change this to min and then we
have our values and then we'll go ahead
and all data equals none just so we're
not holding that in memory again you're
going to have to reference every single
data point as we go through so you don't
want to have all data now we maybe what
we could have done is iterated through
the dictionary and somehow got the
values without creating a new list I
really I don't sure how you would do
that honestly so anyway if you could do
that like if you could get the max
values without creating a new list
somehow that would be important but
again this is mostly meant to kind of
give you some clarity's on the whole
optimization problem itself not so much
that I really think anyone's going to
take this code and and do a support
vector machine at scale so anyways
move it along now we're going to step
sizes so what are going to be our step
sizes and what is this well let's first
run your first right one self dot max
underscore feature value times 0.01 so
these if you recall the beautiful
pictures I drew for you we you know you
had that you and then you take big steps
first right and then once you've figured
out what the minimum of all those big
steps were you'd go for a range around
that minimum and then you take smaller
steps and smaller steps so you're not
taking pointless small steps so first
what we're going to say is our step size
is point nine point zero one point one
so big steps here so it's just whatever
is that maximum feature value that's the
sahte times basically 0.1 that is the
size of those original steps we're going
to take so pretty big steps but then
once we find a good value I'm going to
go ahead and just copy this line copy
and we'll paste two more paste paste
once we find that value where we'll take
1% steps and then once we find that
value even smaller steps ok and we good
you can continue stepping down there
after this point though like after this
point you're really a
it's starting to get pretty expensive
and we'll talk about that as we go but
here is a point of expense I'll just
I'll try to mark some of them as we go
on but you'll see where we can tweak
quite a few different things to either
get a more accurate response or a
quicker response depending and again
that's mostly just so you can understand
the problem of the SVM optimization
problem not so much that I wrote like I
said I don't think anyone's going to
actually improve this one enough to be
truly great but anyway that's one area
for improvement also I'll just note each
step can you thread these steps
simultaneously like could you
simultaneously run the step function
here and then the step function here in
here think about that but we'll get
there I guess so think about that for
now and then because you know how that's
going to happen so could that be
threaded or multi processed or whatever
now after step sizes we're going to set
a couple more kind of step in other
variables so first we're going to set
set V range multiple and we're going to
say that's equal to five I'm just gonna
put a space there so people complain
about my white spaces I kind of like to
cook just clump things I'm sorry I think
it makes it easier to read anyway so
there's my white space but feel free to
not to have white space if you don't
want B arrange multiple equals five so
this cost here I'll say extremely
expensive so I think we'll all it may be
explain this one a little more as we go
but yeah I guess I'll explain that more
but basically B doesn't need to take a
small of steps as W does it doesn't have
to be as precise it doesn't have it's
not as valuable to get it as precise
basically so we're going to multiply
this step a bit but not only does it not
need to be as precise it's also
extremely expensive to to add that one
although we could make it cheaper and
I'll explain I'll explain why as we when
we get into the thick of things but for
now we'll just leave that there and if
you're kind of fuzzy on that don't worry
hopefully it'll make more sense when we
actually start using it now we're also
going to do B
the underscore multiple and again that
one will set to five and this one I
think well I will leave this one alone
for now but we'll probably add more to
it in a little bit and then we're going
to have latest optimum optimum optimum
and that's gonna be self dot Max
underscore feature underscore value
times ten
okay so latest optimum is just going to
be that starting basic list we'll call
it the first element in vector W so
you'll see the first place where we kind
of cut a corn a major Koerner is going
to be here where our vector W is going
to have multiplied the same value for
everything within the vector or the
matrix or whatever you want to call it
that's all going to be the same number
it's actually still pretty darn accurate
even though you're going to do that but
that's the first major Koerner that
we're going to cut is going to be is
right there and really as far as the
whole algorithm is concerned I actually
think that's probably the major the only
major corner we're cutting mostly you're
just going to save a lot of processing
doing that but anyway so we'll leave
that so basically once we find the
largest number we multiply it by ten
we're going to say vector W is equal to
that number comma that number right so
anyway again you'll see what I mean once
we get there so those are the starting
values and then what we're going to do
is begin the actual stepping process so
we're going to say for step in step
sizes what do we want to do well first
we're going to start off we got to start
somewhere with the W so we're starting
at you know basically the top of that
bowl and we're about to drop the ball
into that bowl we're gonna say that W is
currently NP array and it again is
latest actually let's I'm just going to
copy and paste it it's too easy to type
out that latest optimum lazy stop doing
so again major Koerner is being cut
there so just keep that in mind so next
what we're going to do is we're gonna
have an optimized value so we should say
optimized and we're going to set that
equal to false and what we're going to
do and basically optimized will stay
false until we basically have known
more steps down to take so again this
this is the crux of why we can do this
because convex right this is why the
convex problem is so valuable as we know
when we've been optimized so that's
really good otherwise if you don't know
you you might want to take ten extra
steps just to be safe or you might want
to take even more huge steps or
something like that so anyway optimize
false and then what we start is well not
optimized optimized and I think we will
pause it there so I'll say pass and we
will continue on in the next tutorial
where we should be able to finish this
this this entire function actually or
method rather so the next tutorial will
actually finish it up and get this thing
optimized and then probably after that
we'll be able to graph it and maybe
finish it so we might have two more or
maybe three more to finish up this
entire SVM so anyway if you have any
questions comments concerns up to this
point feel free to leave them below
otherwise as always thanks for watching
thanks for all the support and
subscriptions and until next time