Handling Non-Numeric Data - Practical Machine Learning Tutorial with Python

what's going on everybody and welcome to
part 35 of our machine learning tutorial
series where we're talking about
clustering specifically flat clustering
with the k-means algorithm so in this
tutorial we're actually going to be
using this titanic dataset I don't
really know if the original true
original origin of the Titanic data set
but you can go to this link here I'll
try to remember to put in the
description if I forget you can go to
the text-based version of this tutorial
and it will be there as well and you can
also remind me and I'll put in the
description if I do forget hopefully I
won't now i also i've kept all the
imports from the previous suit like the
tutorial we just did so i kept all those
there and then i went ahead and added
some of this data again i probably this
will be in the text-based version i
don't think all i might throw in the
description but i might not but anyway
the titanic dataset as you might guess
contains data from the passengers on the
Titanic which sunken all that and was a
very emotional movie and so this is the
data that is contained within that that
dataset and basically this has
information on what was the class of
that passenger so like first class
second class third class okay so like
planes right first classes is the best
second-class and third-class also first
class was higher on the ship they're
closer to lifeboats and all that third
class is at the bottom of the ship where
things flooded first anyway so there you
have that then you've got whether or not
this passenger actually survived their
name you've got their sex up rung but in
there you've got their age their
siblings so basically a number of
siblings are spouses on board you've got
the number of parents and children and
then you've got the ticket number which
I'm not really sure that's going to have
any true value but you've got it you've
got the fare how much did that passenger
actually pay which will be very
interesting that we'll see down the line
whether or not how much they paid has
any impact we expect class to have an
impact but then the question of what
about their fare after
their class does that have any impact
the cabin that they stayed in which
obviously is going to matter in
depending on where that cabin was
located but it probably matters more
what what what their class was where
they embarked from again that probably
just going to have more to do with how
much they paid for their ticket and what
their class was if they got on a
lifeboat
what lifeboat they got on to their body
identification number and then they're
wherever their home or their destination
was so we have all this information and
our interest is whether or not we can
find any insights from this data but
then also going back to our like our
Amazon data analyst example we should
we're assuming that at least some of
this data is valuable data and
determining whether or not someone would
would survive or die
in the sinking of the Titanic so we we
have this data set and this is a popular
data set because people assume that
these are valuable traits and
information on some people to determine
whether or not they would survive a
sinking of the Titanic and so the
question is what if we used k-means and
we tell k-means hey or any other flat
clustering algorithm but in this case
k-means we say hey k-means separate
these people into two groups and what
are we going to learn about those two
groups and the first question we can ask
is hey did they survive or not in like
of these two groups like what's what's
the survival rate of those two groups or
even even more simply how accurate are
these two groups at deciding survival
and death so can we can is k-means going
to figure out is k-means basically to
separate these people into groups of
survivors and dyers basically so that'll
be an interesting thing to to find out
so of those imports we are actually
going to bring in we're going to say
from SK learn import pre-processing and
cross-validation we're going to import
pandas at
as PE and I think that's everything
we're actually going to use so now we're
going to come down and again if you
don't have this make sure you get it put
it in the directory you're working in or
specify full path mine's in the same
directory as a script I'm working on so
I can just say the name basically DF is
going to equal PD read XL is or isn't
pandas just lovely I love pandas anyway
Titanic dot XLS we read that into a data
frame we're done we can also let's go
ahead and just real quickly we can print
out the data frame head so we can just
kind of see some of this information and
one thing I'm going to point out and
what this tutorial is about to be
covering is well these are numbers right
passenger class number survived as a
number name not a number but do we think
the name is going to have actual like
significance there's probably some
natural language processing that we can
do based on the name in determining
whether or not that name is prestigious
or not but we'll leave that alone we're
not going to go down that bad next we
have sex this is one that's a really
important characteristic yet it is not
numerical and basically the rest of this
stuff all the way up to cabin again
cabin is probably pretty important but
it's obviously not numerical form then
we've got where they embarked from again
we think that's important but it's not
numerical data and then the destination
may be important we think it could be
important and we'd like to leave it in
but again it's not miracle so the
question is immediately what do you do
with this data because the all machine
learning is going to require you to have
numerical data so so what do you do well
I don't know what everyone else does but
I'll show you at least what I do when
doing this sort of thing so that so
generally what you're going to do is
okay let's let's take let's take the sex
column so you're going to take you're
going to take the sex column and then
what I do is I'm going to create I'm
going to take the list of the sex column
I'm going to take the set of the sex
column which is just going to be
basically the unique
values and then what we're going to do
is so like the set of the sex column
values is going to be female male right
that's going to be the set so we could
do the same thing with like home desk if
we just did this these five rows if we
took the set of the home destination
column we'd actually have two values we
have st. Louis in this Chesterville I
don't know that Ontario or something I
don't know I really don't know anyway
oh yeah must be I don't anyway so so
that would take the set and then from
there we'll just assign we'll say okay
female is zero male is one right and
we'll just keep going down the list like
if there's five elements in the set the
first is zero seconds one and so on
that way we assign a unique ID to the
value in that column now if you have a
lot of values in that column though you
might find that you're getting outliers
right and as you've already seen
outliers are likely to cause some
trouble so if you have you know a small
number and then you end up doing
pre-processing anyways you'll probably
be fine but just keep that in mind but
this is how I'm going to handle this
comma but honestly I've never really
looked into like how people translate
text data into numerical data but I'm
sure people do something similar so
first what we're going to do is let's go
ahead and we also had a lot of missing
data in there so we're going to fill
that in as well and there's got two
options we have here and we'll have to
see down the line which one matters but
I'm probably I'm more inclined to
actually not I'm getting ahead of myself
it's like I'm thinking I've already made
my decision and I'm already disagreeing
with myself and you're probably like
what the hell is this dude talking about
so first we're going to say do you have
touch her up and we're going to go ahead
and drop a couple of columns we're going
to drop want to drop that body and then
we're going to drop the name column
because we we I'm thinking you know the
body identification number if there's
even a number there it's good I'm
assuming the body identification numbers
for dead people I don't know I actually
pull this up really quick
oh we're not printing head anymore
anyway
what it was the air here so this is not
contained one in place equals true and
then let me print if that head
so budding in in and n survived zero huh
so maybe at least the survivors that we
have don't have body identification
numbers and then probably some people
don't have body identification numbers
because their body was never found
I'm just totally making this up on the
fly though so I'm not really confident
that that's true but I'm thinking that
that's the case so if they even have a
body identification number they probably
died and so our purpose here is to see
if this data is descriptive of survivors
and dyers or not so I don't want to have
that there because I feel like that
would taint the whole thing so we're
going to drop that we're also going to
run a DF convert objects converts to
merit equals true and then we're going
to do a DF L an a we're going to fill a
zero in place equals true I'm going to
go ahead and wager that pandas is going
to yell at me for this convert objects
pretty sure to change that yeah
PDD to do Merrick unfortunately that
wants us to use like a series at a time
I don't really want to do that I want to
be able to convert the entire data frame
if you know how what the real
replacement to this because like this
basically when we say this we're
converting all of the columns to numeric
but if you try to use PD converting
numeric or something like that I think
yeah
PD to numeric if you use that on the
entire data frame you get a problem so
I'm not really sure what the true
correct way around that is and it might
not even be an issue in this data set
but I'm gonna throw it in there for now
anyway but if you get an error feel free
to comment that out because probably in
the coming months that will be totally
deprecated and removed okay
so we've got the data and basically at
this point we still have that like non
numerical data so actually going to
handle for that now service a handle non
numerical data for the data frame
this is going to pass the data frame
here and then we're going to say the
columns here are gif columns values okay
so these are just the every column and
then we're gonna say for column in
columns what do we want to do well we're
going to say the text digit Val's equals
and we're just going to say an empty an
empty dictionary for now then we're
going to have a quick function here
that's just going to be convert to int
Val and then it's going to say return
the text digit bowels of the index that
is that value so what's going to happen
here is like I was saying you know say
well from female will be index 0 so this
dictionary will be like 0 colon or
actually I'm sorry it's most like it'll
be more likely this it'll be female : 0
right and then male and so on so then
converting to an int it's going to say
ok we're going to reference text age of
us what female what's the value 0 okay
simple as that that's how it's going to
do the conversion so now what we're
going to do is we're going to say if the
DF column column dot d type is not equal
to NP dot int
in 64 and the DF column dot D type does
not equal NP dot lo 64 if neither those
are the case then what we're going to
say is the column underscore content is
DF column dot values to list and then
we're going to say the unique elements
here unique underscore elements equal
sets column contents so it's just we're
just grabbing like I was saying before
that's just going to give us all the
unique non repetitive values there then
we're just going to iterate through or
and say x equals zero and we're going to
say four unique in unique unique
elements if that unique is not in text
digit valves whereas a text digit valves
unique the equals x and then we're going
to say x flips x plus equals 1 and then
we come over and then we're going to map
once base a at this point we're going
through each column right I'm pointing
on the screen like you can see that
going through each column we've defined
this a little stupid function and we've
emptied out the dictionary and then we
are basically asking if that column is a
number or not if it's not then we're
going to convert to list get the set of
that list we're going to take the unique
elements populate that dictionary and
then still within that if statement all
right because we don't want to do this
if it's not within this if statement
that it's not a number we're going to
say DF and now make some space DF column
whatever that column is is going to be
equal to the list map and then we're
going to convert to int and then we're
going to apply that basically to the
values within the DF column so we're
basically resetting the values of DF
column by mapping this function here
to to the value that is in that column
and if you want to know more about
mapping functions you should check out
the pandas tutorial in the data analysis
section on Python programming document
so we've done that and then finally at
the end of this function we simply
return DF oops hey more sleep or coffee
or both return DF now DF is now going to
equal handle non numerical data DF and
then what we're going to say is print DF
n whoo what a doozy okay so we're done
we're still getting this stupid little
error if you know the solution to that
feel free to comment below because
although actually this is just a warning
convert objects is deprecated
interesting anyway so if you know the
solution of that feel free to post below
and help others out hopefully whenever
this hits the fan and no longer works
so anyway coming down here as we can see
sex 1 0 and everything is converted to
numbers
beautiful beautiful ok so that's how you
can handle non numerical data there's
probably better ways to do it so for
example here you know converting the
column you know making it a list
converting to a set really I mean this
is not the most inefficient code in the
world but again if your set was very
large obviously you wouldn't want to
apply this to like a text document most
likely although I don't know any better
way really to assign values to two words
for example I usually use this for text
documents too but anyway there's
probably a much more efficient method I
just don't simply know it
anyway that's that now we're going to do
in the next tutorial is actually apply
clustering now flat clustering and see
the groups that it creates and then see
if those two groups are indeed survivors
and nonsurvivors
so if you have any questions comments
concerns whatever up to this point feel
free to leave it below otherwise as
always thanks for watching thanks for
all the support and subscriptions until
next time
you