K Means with Titanic Dataset - Practical Machine Learning Tutorial with Python p.36 Video Lecture | Machine Learning with Python - AI & ML

what's going on everybody and welcome to
part 36 of our machine learning tutorial
series in the last video we talked about
how to handle non numerical data which
is useful for not just clustering but
really just machine learning classifiers
in general and now we're actually going
to feed through the titanic' data and
see what groups it classifies the
passengers into we're going to say
obviously this flat clustering so we're
going to say two groups so we're kind of
assuming in / hoping that the two groups
are like survivors and nonsurvivors or
close-ish to that so it's going to be
very hard for it to know beforehand who
the survivors and who the dyers are it
has no prior knowledge that the ship is
even going to sync but we're thinking
that something about these passengers -
their name in their body but where they
came from where they were staying how
much did they pay siblings on board name
actually not named sex age in their
passenger class we think that's going to
have some sort of impact on their chance
of survival should this ship sink so
really this is prior knowledge you could
have ran this before the ship even left
port and the question is would we know
what the likelihood of a passenger
survival would we know the likelihood of
their survival based on these
characteristics and then we can we can
actually now we know whether or not
those people survived so we can see like
did that work so anyways interesting
let us begin the first thing we're not
going to actually be using cross
validation that's kind of dumb so again
clustering you can predict with like
after you've trained and fit a
classifier we don't have it yet but
after you've done it you can't still use
dot predicts and predict future values
based on those clusters you more likely
want to use something like a support
vector machine or maybe a neural network
down the line but you you could continue
to predict using the clustering
algorithm just thrown that out there but
we are not going to need cross cross
validation because that's actually not
we don't need that because because there
is no you're not this is unsupervised
machine learning so we can actually test
against all of the data because of that
so I'm just going to delete that cuz I'm
not actually using that and I'm going to
comment out the printing of the head and
let's begin so this one will be pretty
quick we're going to go ahead and and
run just simple capital X equals NP R a
and then we're going to say DF drop and
we're going to obviously drop the
survived column we're going to compare
to that in a moment
but we're going to drop that because
including that would be probably
cheating so anyway yeah drop that and
then we're going to say dot as type
floats so this just converts that data
framed float so just in case for
whatever reason down a line someone
doesn't know the superior method for
convert objects because this is nice
because it converts the actual data
frame so if you wanted to continue
working within that data frame great so
I like to have that I'm not sure why
they got rid of that maybe I really
don't know but anyway for these for this
tutorial if no one has posted a solution
and that's just giving you a straight up
error not a warning you you can just use
as type float and that'll that'll be
fine now what we're going to do next is
we're gonna say y equals MP array of the
data frame just the survives column now
what we're going to say is the
classifier CLF equals k means and again
we're going to say n clusters equals two
and then c left fit X capital X and then
now we're going to compare the groups
we're saying hey k means we've got this
data minus the body in the name we've
got all this other information basically
all of this without name and without the
body identification number so that would
be also including lifeboat information
which if there is no lifeboat it was
filled in with a 0 with that information
so actually I said
for the boatlift dock or port any way we
could figure this out so hopefully if I
remember well we'll drop boat and see if
that makes an impact because probably
whether or not that person come out on a
lifeboat matters but but the clustering
algorithm has absolutely no idea that
we're going to hope that we're hoping it
classifies into the survival column it
has it doesn't no that's our objective
so boat hopefully doesn't have a huge
impact on whether or not it that's what
it does anyway Magic's about to occur
correct is zero and for i n for i in
range of the Len of X what do we want to
do well we're going to predict me equals
MP dot array let me think here because
we can we can do the prediction or we
can just simply iterate through the
labels it really doesn't matter because
it'll keep the class that it was already
but we could just go with the labels
from the that previous tutorial I mean
we pull up that flat class I was going
to go through and do the prediction but
that doesn't really it should be the
same answer but it doesn't make logical
sense for me to do that so hopefully
that was this tutorial if I recall the
labels gets labels underscore so you
could be able to do that take so so
predict me we should be able to say
labels equals that Wow gosh this is hard
I kind of want to do that because then
we'd have to convert probably survival
although survival is 0 or a 1 2 it
doesn't matter well stick with predict
me I'll show you both ways that you
could do it because in the previous
tutorial the other one with part 34
clustering with k-means we said labels
and we actually compared basically we
just you would compare labels to the
survived column in either case the
number might be flipped so for example
if we get let's say twenty percent
accuracy we actually probably got eighty
percent accurate
or if weak but if we get like you know
63 we can't be too confident which way
we actually were correct but if you get
like consistently 20% or and you'll
probably see like 20% and then like 80
and then 20 and 80 if you keep running
it just know that you probably actually
got 80 anyway predict me so even if if
you had a classifier that consistently
got 20% accuracy right you you could
just do the opposite of that classifier
right and you'd be fine anyway just just
say no but these numbers are arbitrary
in the first century that it finds it's
going to say that's centroid zero right
so it's not going to matter if that
centroid is not going to care that our
first r0 is whether or not they survived
or not it's just going to arbitrarily
choose 0 1 2 and so on anyway
well we'll keep with this one and we'll
just do X i dot as type floats and then
we'll go ahead and we'll say predict me
equals and this will be well first well
actually reshape this predict me dot
reshape negative 1 to the Len of predict
me and you fix this and then we'll say
the prediction equals CL f dot predict
whatever predict me is obviously again
you would never use this and supervised
learning because with supervised
learning you told it what the right
answer was whereas here it's just going
to make this prediction based on the
location of that centroid so this isn't
going to do anything different but in
theory you could also just do that CL f
labels compare the two labels to each
other if they're identical
great okay so prediction equals that and
um anyway if prediction zero equals the
y I so if they're the same thing we're
going to say correct plus equals one and
then finally at the end of all this
we'll just print what the correct number
was over the Len of X again if you get
like thirty percent or twenty percent or
ten percent it's it's the resep
so long as it's a consistent 10% because
you'll probably get 10 and 90 then 10
and 90 and so as long as it's a
reciprocal one of those two reciprocals
you can be confident that it's going to
be the larger one because you could
always flip it around and that those
choices were arbitrary inclues
clusters try again okay so this one gave
us a 49 percent accuracy so we are
somewhere between 49 and 51 let's run it
one more time
51 I told you so
so this in this case so far we're really
not finding any any value here but what
are a couple major concepts that we've
covered so far that might be important
well one thing we're not doing is any
pre-processing other than converting the
numbers but we're not scaling any of the
information so the first thing that we
could do is we could say X now is going
to be pre-processing dot scale X and I
just want to make sure that we're gonna
yeah so that's still going to go through
the prediction it'll be scaled and all
that okay good so we'll scale X let's
see if that does any major change and
sure enough that has made a seemingly
initially very major change and again so
as you can see we're fluctuating between
you know 70 27 28 this this means that
you would have the flip side hopefully
that makes sense if that doesn't make
sense to you post below and I'll try to
reiterate that again but it just the
clusters are assigned totally
arbitrarily so the cluster zero might be
actually in reference to your 1 and
their 1 your 0 or it might be 0 0 1 1
it's just random which one gets assigned
so when you get a like something like
that you can be confident you're
actually 72 percent accurate for example
so actually we have quite a high
accuracy here just by adding in that
scale so you know somewhere between 70
and 7
three or four percent accuracy there's a
couple other things we could do like for
example like does their ticket number
actually have that much value so one
thing we can do is just right here we
can do DF drop then we'll drop the
ticket and then we can also drop let's
just drop ticket first and see if that
makes a big difference
ticket not contained in the axes really
did we already dropped ickett no maybe
you got dropped for some reason in this
he'll non numerical data what happened
to our date a ticket column I'm looking
at the ticket column you're full of crap
do I Titan I didn't type it oh yeah
that's same error before I swear this is
not like age old um we made a few few
hairs here one in place equals true okay
yeah so probably the last tutorial for
the day making too many mistakes here
okay so the ticket column dropping it
doesn't actually appear to help us if
anything it hurt us a little bit our
averages are kind of yeah
it's weird I'm surprised apparently the
ticket number it matters it seems to
actually have like a little bit yeah Wow
you'd really have to run this like you
know a million times but it almost looks
like the ticket number actually has some
significance okay
alright anyway so you can play with you
know adding and removing some of the
values there but as you can see when
pre-processing the oh one thing I
definitely want to drop is boat because
I feel like boat might be cheating
because it could be separating everybody
into groups that got into a boat or not
so it might be like totally contingent
on that boat but sure enough it doesn't
look it has an impact but on a very big
impact about as much as that ticket and
actually you know what we need to Ono
ticket is already there
was thinking I told you this gotta be
the last tutorial for the de mostest I'm
also hungry okay so we're getting like
consistently about 69 percent accuracy
even remote removing the boat column so
um so let's just print the head after
the boat home and let's see yeah okay so
again like 68 69 so that is with so
we're to about let's say a little call
it 70 percent accuracy without knowing
or telling the machine learning
classifier that we want you to separate
these people in the groups of likely to
survive or likely to die without telling
it that's what we wanted to do it is
with 70% accuracy given this data
telling us and it just knows that yep
based on the clap flips based on the
class all well obviously not the
survived Congress we're dropping that
poor training it the sex age siblings
parents ticket number fare cabin where
they embarked from in their home
destination it just automatically
somehow is knowing the two groups now
one final thing I'll run real quick I
don't know the result of this but if you
remove the sex column so women and
children were indeed uh first no wrong
button again anyway so what happens if
we remove the sex column right because
that was really bounded to whether or
not someone survived or died like I want
to say like I think like ninety percent
of women and children survived whereas
like twenty percent or something of the
male's died survived anyway so we'll
keep boat there and then let's also just
drop sex just to see what happens and as
you can see I mean it you know it's a
little weaker again 31 means actually 69
percent or 68 percent in that case give
me the 60s thank you so yeah 68 percent
so again excluding the boat and the sex
so again sex had a huge impact it did
have a huge impact on the boat or on the
ship rather and
so would whether or not they got onto a
lifeboat but removing both of those
things again we have it is just the case
that based on their class their age
siblings ticket fare cabin where they
embarked from in their home destination
these people are clearly about seventy
percent into a group that is likely to
survive or die and the classifier just
knows that right just based on centroids
and moving them around I think that's
crazy
but anyway in the next tutorial what
we're going to do is actually build our
own k-means our own k-means classifier
but we will not be leaving the titanic
data set for the same reasons if you
recall before with the whole amat let's
say you're the Amazon data analysis an
ELISA your boss thinks there's only two
groups but in reality there's a spectrum
of groups and that's what you would use
a hierarchical clustering algorithm for
so we won't be totally leaving this
behind we're going to revisit this with
hierarchical clustering and see if we
can find some more interesting data or
information from this data set but
anyway right out of the gate I think
that's pretty impressive that we can
find out this much information clearly
these are two classes of individuals or
being grouped into and it just so
happens those two classes highly relate
to whether or not they survived or died
anyway I think that's very interesting
so if you have questions comments
concerns whatever feel free to leave
them below otherwise as always thanks
for watching thanks for all the support
subscriptions and its Hill next time