Mean Shift with Titanic Dataset - Practical Machine Learning Tutorial with

what's going on everybody and welcome to
part 40 of our machine learning tutorial
series in this video we're going to be
continuing to talk about the mean shift
clustering algorithm and to start we're
gonna revisit code from part 36 and
basically we're going to take everything
down to the fitment or fit X up to the
top copy that code and paste it in here
if you don't already have this you can
either go to the text-based version of
this tutorial or you can go to part 36
doesn't really matter we're gonna change
a couple of things first we're not using
k-means we're going to be using mean
shift so whoops where's my mouse there
we go mean shift also we are going to be
making a copy of the original data frame
which we'll do basically as soon as we
read in Titanic XLS we're gonna and
before we drop body and name and all
that we're gonna make another addition
and that's gonna be original underscore
yes it's gonna equal PD done data framed
copy yeah again we can't just say
original DF equals DF because as you
modify DF it will also modify original
DF so we do this to copy that data frame
and the reason why we're doing that is
first we want to feed it through mean
shift which will cluster our data for us
but then after that we want to look at
that information we want to see what it
found and I don't bite you but I prefer
to read things and text for more there
that are used okay so we could get by
you know using zeros and ones and twos
and threes but for like where they
departed from where their cabin was
where they male/female and all that it's
just easier if it actually says the word
for us so we're gonna do that and then
we can kind of reference these by index
later on once we've made the
classifications so we do that also means
shift for the classifier down here no
longer are we doing k-means it's mean
shift
and we get rid of n clusters great so
that should be all we have to do for now
or at least you know to modify the code
we're done that's well really that is it
because they actually you you you fit me
shift and we're done right but no okay
so you guys I can hear you laughing from
here now labels is he left labels
underscore so that just gives us the
labels and then we're gonna say the
cluster centers equals CL f dot cluster
Center so nothing really new there and
now we're gonna add a new column to that
original data frame so we're gonna say
original underscore Kiev and we're just
gonna say this is clustered group so
it's just a new column and for now that
column is just empty it's gonna have
nodded numbers in it and P nan no prob
so now what we're gonna do is we're
gonna iterate through the labels and
populate the values in of this column as
those labels so what we say is 4i and
rain the land of X what do we want to do
well we're gonna say the original
underscore GF cluster group dot I'll oak
i equals labels what the hell am i doing
okay so what happens here is we're
saying with I Lok we can reference
basically the index of the data frame
but really this is the the row on the
data frame so I'll oak and let's say I
is equal to 0 that will be the 0th row
right that first row in your data frame
this that's what this is gonna reference
and say okay in that first row of the
original data frame under the column of
cluster group we're going to set that
value equal to whatever labels I is and
again labels is just right in order of X
as well there's no shuffling or anything
going on here so that's why we're doing
that so now the first thing right out of
the
that everybody's gonna want to know is
okay
so we've created cluster groups the
question is natively remember with
k-means when we said we wanted you to
classify these into two groups we said
hey we're kind of expecting these to be
survivors and dyers and sure enough to
about seventy percent accuracy that is
what it did but here i'm going to show
you what the power of mean shift and
k-means possibly could have found
similar results but it's less likely but
anyway here is the the power so first
let's say survival rates and of course
the question is here obviously it might
make two groups might make one group
might make five groups ten groups
whatever but then when it makes those
groups who can kind of start diving into
those groups so you'll see what i mean
here in a minute but for now survival
rates this would just be the key will be
the cluster group index or the cluster
group number followed by the survival
rate so just really simple there and
then what we're going to say is for i in
range n clusters n plus underscore i
believe do we even define and we
probably better define n clusters so
we're gonna say n clusters equals the
length of n p dot unique labels so just
however many unique labels we have
that's how many clusters we have so then
for i in range of those clusters we want
to first we're gonna say temp DF equals
original DF and this is an original DF
where the original DF cluster group is
equal to the float of i so this is just
a conditional statement that just says
we're creating a new temporary data
frame where the original where within
the original data frame the cluster
group was equal to this so it basically
it's a new data frame where which only
contain
let's say that I is zero for first where
so this temporary data frame is the
original data frame only where the
closure group is cluster group zero okay
so that's just your quick temporary data
frame now we're gonna do is we're gonna
say survival underscore cluster equals
temp underscore DF and again this will
be another little conditional data frame
where temp DF survived equals one so the
survival cluster data frame will be
where the temporary data frame which is
the one word that's just specific
cluster that we're curious about where
the survivor or survival rate with or
the where they survived one was that
they survived so that's the survival
cluster and then we can just do survival
rate equals the Len of survived survival
cluster divided by the Len of 10 TF
there's a few other things that you
could do here but later on you might
care to actually have that survival
cluster data frame and the temporary
data frame but there are a couple other
ways that you could get just a survival
rate this is just one way so feel free
to do a different way if you want
so that's your survival rate and then
what we're just gonna say is survival
rates my is equal to whatever that
survival rate whatever that rate
actually is so now at the end of this
we're gonna print survival rates now
we'll just say because the mean shift
has a slight degree of randomness you
might not get the exact same results I
get and that is going to be true when
you actually process real data you're
gonna get different results different
times so I'll go ahead and run this but
don't be freaking out if you get a
different different result so first
we're getting that that warning about
the numbers stuff that I'm ignoring
and then we come on down here's another
this okay so so we're setting values on
a copy of a data frame again I'm not
worried about that so in this case we
get three major groups group zero one
and group two and we can see in group
zero their survival rate is 37% not
horrible but not great group one has an
84 percent survival rate which is
actually really good and then group two
has a 10 percent survival rate now I'm
gonna go out on a limb here and I'm
gonna wager with you'll find individuals
that group 0 is mostly second-class
passengers group 1 with the 84 percent
survival rate is probably mostly first
class passengers and then group 2 is
let's see sorry I'm getting confused
already 0 cluster second class cluster
one first class cluster 2 third class
with the worst survival rate that's one
thing that's gonna happen here so we
can't really leave this this would be an
example where all the people wishing I
used ipython this would be a little more
useful but we can continue working in
the interactive shell from this point
you could also have this code over here
but again each time you run this it's
gonna be a little different okay but
we'll continue on with this example so
the first thing that we could do is we
could say something like this we could
say we want to see print
well first we can do this we're first
we'll say print original bf let's see no
print original DF where the original DF
cluster group is equal to unless let's
visit group number 1 I believe these
will be our first-class passengers but
we'll see
coming up here so this is the group okay
sure up this is the full group every
single individual in this group is
indeed a first-class passenger nobody
saw that coming okay
there are also mostly female so a couple
of things to note again for just because
their first class I mean this isn't an
issue on rich people always win
necessarily but first class was just at
the top of the boat so they were more
likely to actually get a lifeboat
because they were on a level that needed
a lifeboat conversely as we're hopefully
most likely to find group two is going
to be mostly third-class individuals so
we can do alts P and look at group two
but group she's probably gonna be bigger
but let's just it that's not so bad the
short enough group to is all third-class
individuals only one of these people
actually even survived but but again
hundred percent that they are
third-class individuals and then it's
likely that the group two is going to be
well group two is gonna be a huge group
but we're gonna wager that it's mostly
second class but obviously this isn't
all of the third-class passengers and
obviously this is not all of the
first-class passengers
okay there's probably more so anyway
we're gonna come down here now and then
what we're gonna do is we're gonna do
all P and I think we're curious what
okay so group 0 we're thinking that's
probably second-class obviously that's
gonna have everyone else basically so we
could look at group 0 and we're printing
original DF and then we're gonna do dot
describe okay so that'll give us a
little a little more space to deal with
okay so here we're working with 1280
people the mean of the class is 2.3 so
sure enough pretty close to class 2 in
general but we do have everything we've
got we've got some first-class
passengers and we've got some
third-class passengers so we have both
in this group we can see some more
information what the average age of this
group was and then we could do the same
thing like we could get the average
information like with described this
just gives us stats on
dataset I think first was that first
class group yeah so again the average
age here is 37 so there are older people
in this group and then we could do the
same thing with forget already what the
other group was so 0 1 so then 2 and
again we can see that this group
actually a little older - 39 ok so then
you might start getting curious so for
example if we look at this is group 2
which I'm gonna go ahead and print the
survival rates against already forgot
what they were so print survival rates
right so for example with group 2 group
2 is the one that basically it most
people died right this is your third
class passenger interesting thing to
look at is maybe look at looking at fare
how much money did these people actually
pay so here we can see the minimum was
29 and the maximum was 69 and this is in
British pounds if I recall so that's the
survival rate or I mean that's the the
fare for group two and then looking at
group zero which is mostly second-class
passengers as we believed we can see
again going to the fair
we can see that the I'm so lost the
minimum is 0 apparently someone got on
there for free I don't know what's up
there and then the max is 263 so this is
0 to 263 for the fair and then if we do
the same thing again only this time on
Group 1 and and keep in mind 263 this is
including some first-class passengers
because this this data set did include
or this this clustered group did have
some first-class passengers but now
looking at this group of first-class
passengers we can see the minimum was
247 pounds paid and the maximum is 512
pounds with the mean being actually 312
so the the people in this group appeared
to still be
a slightly more elite group of
individuals and the same is true here if
we look at the the mean of the fair here
which is actually pretty but which one
was this one this was our this was our
second class so the mean of the fair was
29 and then I wonder if we looked at the
mean of the fair for group two fair the
mean well actually these people paying a
little more money that's interesting I
wonder if it has something to do with
that zero fair so I wonder someone
actually got on for free or if that's
like missing data it's probably missing
data anyway so interesting and then with
that missing data from group zero I
wonder if that would affect the fair but
I would wager probably not so and the
other question we could ask is something
like what would be the survival rate of
the the first-class passengers in this
group zero so for example in cluster
zero remember the survival rate was
thirty-seven percent but what about what
if we asked what was the survival rate
of first-class passengers in in this
cluster zero for example so you could
ask you could have something like this
you could say
let's see cluster zero and you could say
we could say cluster cluster is zero or
possibly let's say cluster
sorry I'm trying to decide what I want
to do so first we could take this here
and then I could say you know cluster
zero equals this only this we actually
want I think group zero if I recall yeah
so we that's the what cluster zero is
and we need to close this off let's see
so here here okay so that's cluster zero
and then we're gonna say cluster 0 FC
equals and this would be fluster
underscore zero V equals oops plus or
zero we're within cluster underscore
zero and then this was peak class places
Wow okay this would be the last tutorial
of the day as well apparently cluster
zero P class equals one so first class
passengers okay and then we would say
cluster underscore zero underscore F C
dot describe and this will actually give
us the survival rate as well so here
survived the mean of this is 60%
so 60 percent of the first class
passengers in cluster zero survived so
that kind of gives you a little bit of
information on you know still the upper
bounds of the survivors within cluster
zero we're first class passengers and
then of course you would be you'd have
to ask like is that survival rate you
know you could do the same thing for
rather than cluster zero first class
what about second class third class
passengers
stuff like that what was their survival
rate in this group but for the most part
it appears to me that the group is
largely classified based on the fair but
there's probably some other information
in here of like what these groups are
actually classified into but still at
the end of the day we can we see these
groups are classified into P class but
also to some degree their survival
sort of survival rate so anyway now what
I want to do real quick is remember this
created three groups and then this was
the you know the survival rate right so
0 37 84 and 10% let's just run it one
more time and then just see if we get
more groups and then right we actually
wound up with an extra group here so we
still got 37 this is probably the second
class the bulk of the data most likely
and then this one actually has a group
that survived 100% which is pretty
interesting so we could take see I
wanted see this let's take this copy I'm
just gonna come down here paste and we
could look at Group 1 and I don't want
to describe I want to see this is so we
have some people basically on two
different tickets but otherwise these
are all together people these are their
fares and the other interesting thing I
wonder is with the fare that it is like
we can see that this is a ticket for one
two three four people so their fare is
higher possibly I don't really know so
it might not even be the case that these
are the truly elite people but it turns
out that that seems to have a pretty
decent impact on their survival rate
like maybe it's how many people were in
their group I really don't know but you
can keep digging into it and find out
but anyway these people had 100 percent
survival that's pretty crazy
interesting so in
I think there's anything else I really
want to point out and then really in
k-means we kind of already figured out
that that male and female for whatever
reason didn't really seem to actually
play a huge impact in that in like those
two groups and then when we took away
like the lifeboat and all that the two
groups were still relatively consistent
so pretty interesting so anyway that's
it for this tutorial if you feel free to
like dig around you can probably find
some more interesting things in this
data set it's a pretty popular data set
people use it for all sorts of analysis
and stuff like that because it's kind of
common that you'll get a data set that
just has a bunch of information on
people and you don't really know you
might not are not just information on
people but information on just things in
general so you can use clustering to
kind of group those things together and
then you kind of start going into those
groups and see like what makes up these
groups why why are these being
classified as groups because in this
case now we have four total groups and
also if I was to bet if we do two so
private
third class individuals sure enough yeah
so you can see it can it's continuously
giving us these these similar groups
where at least the least likely to
survive or the cheapest people in third
class and so on the most likely to
survive apparently are the people that
pay the most for their ticket so
interesting so anyways that's it
questions comments leave them below till
next time