Mean Shift Dynamic Bandwidth - Practical Machine Learning Tutorial with

what's going on everybody welcome to
part 42 of our machine learning tutorial
series we've been talking about
clustering with the mean shift algorithm
and in the last tutorial we built our
very own custom mean shift algorithm
that appears to be working pretty well
like we can run it real quick and like
we've got our clusters chosen
automatically we feel like we're pretty
darn smart
unfortunately this really only works
when the radius is 4 or about 4
otherwise if you do something bigger or
something it doesn't work the way we
intended so basically we've reduced this
to a hard coding issue where we've had
to ourselves implement some sort of
radius that makes sense the only reason
we can choose 4 is we could kind of look
at the data set and see you know a 4
radius would probably be a good radius
for this data set ok but that's not good
because what if the data set was 30
dimensions you want to be able to so
easily look at it be like yeah the
radius ought to be about 4 or you know
or whatever it might be so instead what
we're going to do is we're going to put
on some weight and we are going to use
that to help us basically automatically
handle for finding the centroids
correctly so we're not actually going to
use a simple radius we're going to use a
very large radius but we're going to
penalize points for the further they are
from the current centroid in question so
that's going to hopefully give us much
better results and make this code much
more dynamic let's get to it so first
we're going to go radius and we're going
to set the default radius to none so if
the person really wanted to set a radius
maybe we might let them do some radius
stuff but I'm going to set that to none
for now and then we're going to say
radius norm step equals 100 that'll make
a little more sense when we actually get
to it so I'm not going to explain too
much there but we want to have a lot of
steps in the radius and like a lot of
basically band widths and then depending
on which one they're in the closer they
are to the centroid the more the higher
the weight or the value is for that data
point so that's kind of where we're
going so then we're going to say self
dot and then I'm just gonna copy those
copy
equals paste okay so cool now we're
going to pop on over to the fitment and
basically before we define really
anything we'll just add it up here we're
going to say if self dot radius equals
none what are we going to say well we're
going to say all data centroid and what
we're going to do is we're going to find
the centroid of basically all of the
data so we're going to find the center
of all of our data well and then so have
the way we do that is just M P average
data and then ax is 0 so the same thing
we've done somewhere down here somewhere
where are you there it is
new centroid okay so same thing we've
done there so we set that and then we're
going to say to all data norm equals NP
Linna
norm so this just gives us the magnitude
from the origin so data centroid so
basically from from the norm to this
data point we're just finding like what
about the averages and then we're going
to use that average to help us determine
a decent overall radius to have and the
steps and all that so anyway we do that
all data norm equals that and then we're
just I solved out radius so now we're
actually defining a radius will be all
underscore data underscore norm divided
by self dot radius underscore norm
underscore step and that'll be all for
now so now we've got a new radius it's
that that entire norm divided by the
actual radius step that we're going to
take now we're going to go into our
while true loop and we are going to
first we need some weights so we've got
let's see for I in let's before I in
centroids so this is our while true loop
we're in like that alone there so watch
your lip and then we've got 4-iron
centroids here and then basically before
we iterate through our actual feature
set we are going to define our weights
so we're gonna say the weights are equal
- I for I in range of self dot radius
underscore norm underscore step so this
is going to be a you know zero all the
way to ninety nine basically so we're
gonna have weights from zero to ninety
nine and then we're going to reverse the
order of this list of : : negative one
will reverse the list there's other ways
you can do it but this will do it
so now it's going to be 99 98 97 96 and
so on okay so that's step number one
what's going on Harrison from the future
here Oh as I was going through the
text-based version this tutorial I cut
this kind of stupid error here where
we're defining the weights within the
for loop I mean this is just me kind of
like going through the process of like
all the stuff that we needed and at this
point we're about to use weights so I'm
I'm defining weights but actually the
weights are the same for the centroids
and through each iteration so that's
really kind of dumb to be redefining the
weights every single time it's not a
hugely costly process but it costs and
there's no point to keep redoing it so
anyway
weights what it it shouldn't be in the
for loop and it really shouldn't even be
in the wild true loop it should be
defined like right above while true so
anyways just thought I would throw that
in there I'm sure I've made lots of
other stupid calculations I've just
happened to catch this one so I'm
bringing that up so we have our weights
and then for feature set in data what
we're going to do is let's see if and
Peter Lynn al norm where actually we're
going to pretty much redo this entire
thing so what we're going to do for
feature set in data so I've cleared out
everything in that for loop now we're
going to say is distance equals NP Lynn
Allen orme whatever that feature set is
feature set minus the centroid so now we
have the full distance and then what
we're going to say is if the distance
equals zero we're going to say distance
equals 0.01 okay that's all I got that
might be a little too absurd maybe that
okay so so basically as we we you can
also say like nevermind but anyway this
is just for
in the feature set is comparing distance
to itself for that first iteration so
we're just going to do this
to make it simple so now what we're
going to say is the weight index equals
the integer value so it's just going to
come out it of distance divided by self
dot radius so what's that entire
distance and then we're going to divide
that by that radius that that that
distance that we wanted to work with so
the entire distance divided by the
radius that's basically how many radius
steps we took and the more steps that we
take the less we want that weight to be
right so if the way it index is zero the
weight would be 99 if the weight index
is 3 so if there are 3 radiuses in that
distance that's further away we would
want that weight index to be 3 and that
would correspond to 96 right or I don't
know anyway some are in the mid-90s
anyway oh now we're going to say if
weights underscore index is greater than
self dot radius underscore norm
underscore step minus 1 we're going to
say the weight underscore index equals
self dot radius underscore norm
underscore step step -1 what's happening
here okay so if if we if the distance is
just greater than that maximum distance
or if it's more than 100 steps away
we're just going to say the weight index
is that max element which in this case
would be in theory like zero so so if
it's outside of the all of the bounds
we're just going to say that's the
weight value and then we're going to say
to underscore add and we're going to say
that value is weights weight index to
the R squared or to the power two so
we're squaring that times the feature
set
and there we have that one thing I'll
add is I'm not really sure how to
implement it very well in this code
because without making it probably too
tedious but for example one way that you
can do like averages without actually
doing what we're doing because in theory
we're multiplying the size of our data
set this is going to be huge in theory a
huge array or list in this case right
it'll be a huge list of features and
then we're going to average that huge
list where one thing that you can do is
you can say okay here's a number there
are a hundred of these numbers and then
that makes us one big number and then
you could say that big number divided by
there's you know one hundred of these
that's easier to do that make up making
an array that has a hundred of those
numbers and then calculating the mean of
a hundred of those those numbers that's
going to take up more space in your
memory so something like that could be
added to this but again this is just
meant to be a really simple example to
show case weights not to actually be
super efficient but anyway keep that in
mind basically at this point when we
create this we're making a very large
list in theory it's a hundred times each
feature possibly so you're multiplying
your data set quite a bit here plausibly
so keep that in mind so any way to add
and then what we're going to say is
we're just gonna say in underscore
bandwidth plus equals to add so you can
when you have like two lists this is a
list plus a list okay so that's why you
can do plus equals there okay so now
what we've got is we're going to come on
down to your centroid let me think here
let me make sure so MP dot average in
bandwidth that's fine we get the new
centroids the tuple cool uniques
that's fine and so yeah basically
between uniques and precent roids we
need to add a bunch of code so we're
going to say to to underscore pop it's
going to empty list and then we're going
to say for I in uniques so for each
element in
and then for I I in will do this we
think this is legit I've for I and
uniques and therefore I I in this is
just kind of like a way that we're gonna
sorry I'm trying to think if I'll get
away with this
I think you could do for I and uniques
and then for I I in uniques that should
work if that doesn't work we'll come
through I can't imagine a reason why
that won't work anyway if I equals I I
we pass okay so if they're identical we
pass because it's going to be evil to
itself but we've already the ones that
are totally equal to each other
we've already we've already identified
those when we did the sorted list set so
if I equals I I
it's literally itself ok so so we're not
worried about that and the whole reason
for this new code that we're doing is
because we've made this size much bigger
we're in theory incorporating a lot of
other data points so there's going to be
centroids that are like really close to
each other but not perfectly equal to
each other and we need to be able to get
rid of those because those are useless
right we don't need them like if they're
you know point zero zero one percent
different from each other why would we
have those that doesn't make any sense
so we are going to get rid of those and
this will be a sort of tolerance ish
kind of thing anyway if they are equal
to each other we pass because what I
just said we've already got the set so
that would be purely itself L if NP l NL
norm of NP array I and then NP whoops
minus NP r AI minus MP dot array i I
let's see if again so if the distance
between these two um if these the
distance between these two arrays or
vectors or whatever you want vision them
like if that distance is less than or
equal to the self radius so they are
within one step one radius step of each
other we're going to say these need to
be converged to be the same the same
centroid so if they are within that st.
one step of each other and keep in mind
we how many steps do we have we have a
hundred steps in the data set so that's
a lot of steps so if they're within just
like that one original step what we're
going to do is pop it out so we're going
to say to underscore pop dot append PI
and then we're actually going to break
here so we I'll show you what happens if
we don't break there later but anyway to
pop append I I now once we have the two
pups and just real quick why can't you
just pop PI right away you can't modify
the list as you iterate through it okay
so that's why we have to kind of do it
this way now we're going to say for I in
to underscore popper in a lot of cases
you'll get an error sometimes you won't
get an error I think you'll get an error
for sure in a dictionary I don't know in
a list but it's not a good idea to ever
try to modify a list as you iterate
through it for I and two pop we're going
to try to unique start remove I and in
fact we should we probably don't need to
have a try there because that's that's
why we added that break but anyway what
if we don't have the break there we'll
probably add like twice because there's
going to be like one versus each other
so we'd probably add twice if we didn't
break there so anyway we probably need
that try to accept but anyway we'll
we'll show an example of that whenever
whenever we go through it so now let's
see now what we're going to do is self
that centroids you go centroids so
basically all the way up to this point
um the code is done we should everything
should work let's go ahead and run it
and just see what we get at this point
because I think that's all the major
code changes we wanted list uniques dot
remove I I not let me go ahead and add
I'm pretty sure for in2 pop to pop a pin
let's go ahead and just have the try and
except here real quick I thought that
break would save us but anyway except
pass for now let's just see what happens
okay so we worked out in the end
wonder why there would be two cases of
the same pop hmm anyway I have to think
about that later and really look at the
code
but anyway so we've got the the clusters
it's not as perfect of a cluster as we
would want most likely because we popped
this one looks pretty good this one
looks in this one but if we aren't
actually classify these clusters by the
cluster Center then that would be a
successful one also this is a very small
problem so so because of that very few a
number of data points that could skew
the the centroids a little bit so anyway
that's that now let's work on adding
classifications and then actually
allowing this code to scale out a little
bit to bigger data sets and we'll show
an example there
so now after let's see self-taught
centroids equal centroids great we're
gonna say self dot classifications and
that's going to equal an empty
dictionary for now and we're gonna say
for I in range of the Len self dot
centroids and we're gonna say self that
classifications by equals an empty list
for now so we're going to classify by
you know closer zero one two we've kind
of done this I'm pretty sure this is
similar code from the k-means and then
we're going to say is for feature set in
data we're going to say the distances V
equals make this a list of Lists NPL NL
norm of the feature set - self dot
centroids centroid and then that would
be for centroid in self dot centroids
it's never enough to screen for such ok
good so those are our distances and then
we're going to say the classification is
that minimum distance right so that
would be classification equals distances
index and then it's that min of the
distances so whichever centroid has the
least distance that's the classification
that'll be the classification index
basically and then
we're going to say self dot
classifications classification right so
that's the classification of that
feature set so we're gonna say that
classification now gets an append of the
feature set now we know it's a part of
that class okay so once we have that
good also we can take this here and
paste that under predict so distances
classification self dot classification
or rather actually we just need
distances in classification and then we
return classification so now our per
dates class is completed now we should
be all set so then what we're going to
do is we're going to pop on down - we've
got centroids pill to us scatter great
now we're going to do is for seeing
centroids will leave this code here but
then we're going to get rid of this code
and we are going to say for
classification in CLF classifications
what do we want to do well we're going
to say the color is equal to colors
classification so we're just gonna use
that as an index Lang make sure we
multiplied that list of colors where's
my bar there it is
yes we did okay coming down
classification fix that and now we're is
a for feature set in CLF classifications
classification okay PLT dot scatter and
then this will be feature set features
at zero feature set one and then we're
gonna say the marker will be will do X
here and then color will be just color
because we've already color redefined
color here color is equal to color s
equals 150 and that's probably all we
really need
and then PLT our show we're done whoops
great let's look at that real quick make
sure that works ooh we need to change
the line with below skirt here let's say
line wits equals five great okay so we
got the clusters that worked out bap so
now we're going to do is we are going to
pop on to the top here and we are going
to get make blobs from s kala so from SK
learned on datasets dot samples
generator import make underscore blobs
now what we're going to do is let's
comment this out before we forget we are
going to do let's say X comma Y we're
not going to use Y but it always when
you make blobs it will it will classify
the blobs for you just for a favor
because when you make blobs you might be
doing SVM or you like you might actually
be doing SVM and training a classifier
or you might be trying to do close your
range so it can do both anyway get you a
blob that can do both make blobs equal
or no pink blobs n samples and we'll
start with a 15 will say centers equals
3 and then we'll say n features equals 2
and then what we can do actually let's
run this one first and then we'll we'll
play with that second so say everything
looks good let's do it
with wound okay so this one and it gave
us only one centroid even though clearly
price should have one there there in
there let's try again let's try again
before they all mess up okay so this
one's a little different uh this one
probably wanted you to be there but it
did at least get this cluster in this
cluster hopefully I'm hoping it's gonna
give us like something like totally
obvious so we can be like yeah good job
yeah so this one's totally obvious and
we can say yeah good job to our
clustering algorithm it figured out yet
that
correct that's correct that's correct
awesome okay so that looks good let's
increase samples again as usual if
you're on like a smaller computer or
something you might not want to get do
as many samples and if you did like you
could change the radius step smaller so
that's gonna make a huge difference on
your on your processing time Wow are you
serious
I'm very angry at that one I'm going
back to the hundred steps so we get the
proper looking classifications I don't
think the hundred steps would have made
that mistake all is taken forever to
come on while we wait for that we're
going to import random and then for
Centers let's say is going to be I
cannot believe this not coming on must
have come up and now it's under here see
Oh didn't get it oh my goodness
this is embarrassing here but I'm ready
to freeze these samples here let's do
let's do 50 hopefully it'll be a little
quicker yeah second one really should
have come through it's embarrassing well
we might just have to keep number of
samples like 20 sisses takes forever I
just popped up hold on at least this one
worked I don't understand that's pretty
confusing how some of them work totally
obviously even this one's like pretty
close like I wonder why some of those
ones where they're closed like this
usually it's like got a cluster Center
right there and then you've got this one
wonder what what causes that but anyway
there that one worked so what I want to
do is um will say centers equals random
dot R and range and let's do between 2
and 8 and we'll just randomly centers
and then let's uh yeah let's run let's
just see let's see what we think only
see it
and we'll see oh my goodness haha this
is messy as heck oh my lord one alright
let's see so looking at just the S is
kind of hard to see you let's say you
got one cluster here probably one here
so one two three oh this is hard
four five six six total clusters I think
that's what I think we have there and
then let's let's come over here print
for Centers sinners oh seven I want to
what they were oh wait let's see Center
no I can't get it now but okay um maybe
we probably need more samples if we're
gonna do eight let's go up what will be
willing to go up to five yeah this one
probably has one two three it might have
five or four but unfortunately we
clearly didn't get them all actually
that one only had four I wonder what
those ones were but as you can see
there's some significant standard
deviation going on so you might want to
tinker with the standard deviation
that's probably throwing us for a loop a
little bit but anyway um so I think
that'll bring a wow yeah like this is
another case where you probably have too
many uh too many
are too few samples rather but our code
is pretty darn slow so that for sure
even though we were maybe a little
faster than scikit-learn with k-means
with mean shift we are definitely the
losers here so anyway that's all for now
this is a really simple stupid example
of mean shift but as you can see it's
it's a big deal either hard coding the
radius is just lame and stupid or having
to choose that value yourself there are
various ways that you can choose the
radius and automatically find it you can
probably think of a better way to do it
than ours I'm again I'll just say that
most likely the largest hang-up in our
code is this part here or multiplying
and we're creating a theory a huge
vector that might take a while but then
again I mean that was taking a lot
longer and I would have expected even to
have let's say 50 samples times 100 I
mean that's still that's like 5,000
samples so why is that taking a long
time out I don't know
that shouldn't take that long but anyway
this was supposed to be a nice custom
version that just gives you an idea of
how it works not necessarily so you can
have something that's going to win new
competitions so anyway that's at if you
have suggestions about how to improve
this if you can figure out a much faster
way or a more reliable way I'm all ears
there are various things you can do like
you can modify the weights you can
modify the step and you can modify here
especially we're squaring the weight so
oh so it's not it's not 100 times 50
it's we are squaring that value so
that's probably let's let's just just
for kicks it's gonna make a big
difference in accuracy but let's uh 50
sample let's do yeah we'll keep 50
sample to see how quickly that comes up
yeah so that's like really fast and then
even though it did affect our accuracy
so it's much quicker if you if you don't
square it which makes sense because you
know let's say the weight is 30 30
squared is big number so that's what's
doing it but again if if there was a
decent way to add them to the list and
say these were worse so like maybe maybe
make a weight dictionary and then add
everything to the weight dictionary that
had those weights and then when you went
to go do the average you would just have
one big number and you would say okay
this big number constitutes X data
points X number of data features or
whatever that would be a much easier
process on the on the machine to do so
anyway this is clearly the area for
improvement if I just multiply by two
now we got an empty sequence there
that's not good enough anyway that's no
surprise that's a pretty surprising
anyway now I'm just embarrassing myself
this is not the best code you've ever
seen I don't know what is but anyway
that's good we'll end on this note
anyway questions comments additions
changes I'm all I'm excited to see if
anybody can come up with something
better I'm sure somebody can so feel
free to share if you come up with
something good anyway now we're going to
be moving on to a recently much more
popular topic which is the neural
network which has kind of evolved into
deep learning and all that so we're
going to be changing Gators now talking
about neural networks which are actually
extremely simple at they're their base
form but then they get very complex very
fast so anyways we're going to be
transitioning into the neural network
questions comments concerns whatever
leaving below otherwise until next time