K Nearest Neighbors Application - Practical Machine Learning Tutorial with

what is going on everyone and welcome to
another machine a learning tutorial in
this tutorial what we're going to be
doing is continuing along with
classification and specifically right
now we're talking about K nearest
neighbors which takes any new data point
and just compares the quite literally
Euclidean distance between that point in
the other data points to determine which
which class it happens to belong to by
comparing its kind of distance from one
class to another class basically so
we're going to be using scikit-learn to
do this and then we're going to use a
data set from from the UCI edu data set
so here we go so this is this is for
University of California at Irvine I
believe is UCI okay so this is the
website if I happen to forget it's
relatively simple archived ICS that you
see ided you /m l slash data sets dot
HTML they have a ton of day sets here
they've got them
I'll even scroll up here they've got
them separated by tasks that you would
perform with this data set like
classification regression clustering
other they've got attribute types data
types
area that kind of environment right like
Sciences engineering social sciences and
so on how many attributes and then how
many instances so probably samples here
anyway there you have that great site
for practicing machine learning and the
data set that we're going to use is the
breast cancer data set we're going to go
with the original one I've already got
that up though so that would bring you
to a page like this and then you'd be
interested in the data folder which
would open up this page and then if you
click both of these you'll get this and
this so the first one is the actual data
that corresponds to the breast cancer
set so we'll right click and save that
and I'm going to save it in the folder
or the thing that we're working in which
is this right here and then the second
one actually I'll just save it there the
second one is we don't really need to
save it but you want to read these
because this gives you a little bit more
information on the data set itself
primarily first of all you get the
citation request
also you have coming see pass usage you
can learn about like who's used it and
stuff like that and then basically we're
looking for ya right here so the dot
names right we didn't know what which
one of these look like what anything was
but if you come down here you can see so
the first column is just sample code
number basically an ID the second column
is clump thickness and uniformity of
cells size shape and so on you keep
going down the list and then finally the
11th attribute is actually the class
which is either benign or malignant for
the tumor so benign or malignant is an
attribute but that's also what we're
actually trying to predict for so we're
going to actually make that the class or
the label so to speak so anyways that's
our data we've downloaded the data I
have it now let's see if it's yeah down
here so open that up and so here's the
data and what I'm going to go ahead and
do is add a header row here we could do
this in pandas but it's probably easier
just to type it in really quick here so
we know the first thing is ID clump
underscore thickness and you'll
definitely want to be doing this along
along with me and then I'm going to say
unif cell size UNIF cell shape and i'm
going to scroll like this here or
something let's do that then this okay
unif cell shape and then what do we have
next we have Marge adhesion single Epis
cell I think that's saying sighs I guess
bear whoops
bear nuclei bland whoops bland kromm
we'll just stick with that norm nucleoli
my toe says and then finally class and
so hopefully we have the right number
here so I D two three
or five six seven eight nine ten eleven
okay cool so the thing that to pay a
little bit of attention to will save
this is that eleven is a class two means
the tumors benign four means it's
malignant so you'll actually see later
when we go to code this ourselves you
most machine learning algorithms don't
want the class to be a string right it's
supposed to be a number this is how you
can convert it but later on I'll show
you you can kind of get away with using
a string but anyway for the most part
you're going to be using like numpy or
some other really optimized library that
does number crunching and that's going
to want numbers not a string so the
other thing to pay attention to is this
says it does indeed have missing
attributes there's 16 missing and if
they are missing it's denoted by a
simple question mark also you get the
class distribution here which just says
basically we've got 458 benign tumors
and 241 of them are malignant so it's
not totally a balanced data set but it
might actually be more realistic in
terms of how many are benign or
malignant I'm not really sure but
hopefully more people have benign tumors
than malignant so anyways that's the
data set we can rename it now let's go
ahead and load it in to what we're
working in so just remember that name if
hopefully it's saved in the local
directory so let me move all this stuff
over I'll put this down here and so the
first thing we need to do is handle for
the missing and bad data but first let's
go ahead and make some required imports
so import numpy as NP from SK learn
we're going to import pre-processing
cross-validation and neighbors so you
should recognize pre-processing already
we've already covered that
cross-validation already covered that
neighbors is new but you can probably
guess what that's going to be used for
and then we're going to finally import
pandas as PD again we've already kind of
been using that so first of all we're
gonna load in that that CSV or actually
it's just a text file but we'll treat it
like a CSV cuz it's basically what it is
we load that into a data frame so PDR
Eid underscore or CSV and the name is
breast cancer Wisconsin data so I'm just
going to copy and paste that that's the
name we read that in and we already know
out of the gate we have some missing
data that is denoted by a question mark
so let's go ahead and replace that
information so we're going to say D F
dot replace and we're going to replace
all question marks with a negative
99,999 and then we're gonna say whoops
comma comma in place equals true so that
just modifies the data set or the data
frame right away and then remember
before I was asking is there any you
know you want to watch out for useless
data and in this case what might be
useless data well when looking at the
data set here the first column is ie D
does ID have any implication as to
whether or not a tumor is benign or
malignant the answer of course is no so
we don't want the ID column that's a
completely useless column and it's going
to lead our algorithm stray unless I
happen to forget will we're going to get
rid of it for now but later we'll bring
it back just so you can see how much
havoc that might wreak on on our
algorithm here especially K nearest
neighbors other algorithms might handle
for outliers better but you'll see how
bad K nearest neighbors handles for
outliers so anyway DF drop and we're
going to drop what we're going to drop
um then in brackets the ID column pretty
sure you can get away with just string
but I know brackets will work anyway one
there and then in place equals true so
we're dropping the ID column now as you
can remember before we're going to
define our X's and Y's also I believe
I've already explained the negative
99,999 the whole point of that is just
so most algorithms recognize that's an
outlier and we'll treat it like an
outlier as opposed dumping the data
again a lot of real-world data sets have
a lot of missing data just just a ton of
holes to the point where
if you were to drop everything with
missing data you would maybe sacrifice
50% of your data set or something like
that it's you don't want to do that you
don't have to do that so anyway we do
this to make it just a huge outlier
basically at least for that specific
point now with K nearest neighbors this
really doesn't do too much try to
envision like if you just had two
dimensional data plus the class try to
envision what negative 99999 would do
that's really significant so the only
time you would ever identify probably
with a class that had it if you if your
current one or your current feature set
has this would be comparing it to
another feature set that also almost
certainly has one of those but anyway
we're going to have that there so maybe
in this case you might want to actually
say zero or something like that you just
kind of keep that in mind depending on
how you want to do it will see that this
probably won't have a huge effect anyway
also we only had 16 so you could in
theory rather than replacing the
question mark you can do DF drop na in
place equals true and just get rid of
those so anyway that's really it comes
down to opinion and also your specific
data set and the algorithm that you're
going to use but anyway we're going to
do it this way
so now we're going to define x and y so
I've got x and y again capital x is for
your features y is for your labels or
your class or whatever so x is going to
be NP array and we're going to do
basically the exact same thing that we
did before when we were using linear
regression so the features are basically
everything except for the class column
class do you have to drop class comment
one and then the label is basically just
the class right so NP array DF class so
now we have our features and our labels
now we're going to do the
cross-validation
so again X train X
test why train why test equals
cross-validation dot train test whoops
train test split and we're going to
split what x and y and the test size
will be zero point two twenty percent
feel free to change that if you want but
this line again we've already covered it
that's just how you can quickly shuffle
the data and separate it into training
and testing chunks that meet our our
hopes so now we're going to define the
classifier so you've already seen this
before so ah
I fit yet classifier equals neighbors K
neighbors classifier K neighbors
classifier okay I think I spelled that
right so we've defined the classifier
now we can fit so CL f dot fit and then
we fit what well we fit the X train and
why train again all this code is almost
identical to when we did the linear
regression code that's what's really
just so nice about scikit-learn is even
with different machine learning
algorithm I hate today classes but
really they are classes right before it
was regression and regression is totally
different from classification but as far
as scikit-learn is concerned it's
basically the same thing
so we fit the data and then of course we
can immediately test it with will call
this accuracy equals CL f dot score X
test Y test and again we're calling this
accuracy I may forget to change it in
the written version but we call it we
want to call it accuracy and not
confidence like a lot of times I want to
call it confidence but really it's the
accuracy because we can also measure
confidence as I already kind of
explained in the theory part with K
nearest neighbors when it comes down to
the vote basically so there's our
accuracy and then we'll print accuracy
and let's go ahead and save and run that
it's taking a while to pop up but there
it is let's pull it down here but um
it's telling me the file does not exist
sure it does
liar maybe it's done oh I know what it
is so I I downloaded that data but it's
actually I bet it's dot data dot text
let's try that yeah interesting okay so
with just with the defaults and all of
that our accuracy is 0.9 64 which is
96.4% that's huge accuracy now of course
like I said before with the self-driving
car and all that differentiating between
a baby in a blanket and blobbit are on
the road I don't know about you but I
don't want to be told that I do or don't
have cancer even with a 4% inaccuracy
right I want like 99 plus percent but
probably you know I would go off a test
that was ninety six point four percent
accurate now that now I'm thinking about
it let me comment out the dropping of
the ID column and let's just see what
this does
BAM massively we're almost to flipping
Queen status just by including the ID
column that it that's just that's huge
okay so finally the next thing that
we're going to go ahead and do is we're
going to make a prediction so let's say
you've trained this classifier in most
cases especially if you trained on huge
amount of data what we would pickle the
classifier immediately we just save it
to a pickle but I'm not going to do that
every single time so if you want to know
how to save classifiers go back to that
one this just trains in tests so fast
it's not really that necessary
so anyways now let's make an example to
predict so let's say example underscore
measures equals and this is going to be
an umpire a and let's just make some up
so let's say it for two one one one two
three two one
so first thing we're going to do is let
me pull this over so what what
that predict go so keep in mind this is
the first few - the ID and also - the
class so it would be like these numbers
here so a four so automatically like
looking at this it looks like this first
column is kind of higher for what I
believe to be malignant I think for was
malignant if I recall right anyway it's
a four class every time that's pretty
high that's a four so you can look at
that but also the other numbers seem to
be in line with the two class also I
mean you can make up your own if you
want just go ahead and search the
document and make sure that does not
exist for two one one one two three two
one right that set of attributes does
not occur anywhere in this document
therefore it's unique it's never been
seen before so we're going to test on
that so there you have that and now to
do a prediction prediction equals CL f
dot predict you've also seen this
already with linear regression so again
it's just so it's just the same no
matter what type of algorithm you're
doing which is it's just nice and then
print we're going to print the
prediction and we're probably gonna get
either a warning or an error here but
I'll explain that next I don't want to
add that to the list up I'll see example
measure okay so example measures I got
an error because of that murmur okay so
we're actually getting an error that I
didn't expect to get oh let's see
let's see it's easy um what we're doing
I forgot to uncomment out the ID column
so let me go back because I am confident
now that will get our prediction short
off there we go okay aha so what tipped
me off was that our accuracy was still
56 percent okay so now our accuracy is
ninety eight point five percent and we
did get our prediction but you'll see we
got this this deprecation warning that
it wants us to reshape our
our data before the problem was we were
passing an incorrect number of
attributes because we we had one more
because of the ID column okay
anyway fixed but now to get rid of that
value error we would do this and then
run it and then now you don't have the
stupid deprecation warning so that makes
things maybe a little more complex so I
just wanna before we zip away we're done
with K nearest neighbors but now we're
talking about working with scikit-learn
and also numpy as far as reshaping and
shapes are concerned and i think it's
important to understand how this works
because it's going to come up so so in
this case we had one sample so we had
one negative one so what if we had two
samples like what if it was like this
like copy this let me think here I think
we can get away with it's got to be a
list of Lists so it would be a list of
list will do comma paste and now I've
got two and let's change this to like
two and then this - two - and so we've
got well basically we've got two samples
so noting the directions we had before
this was for one sample so we're going
to change that to two right so let's go
ahead and save and run that and you'll
see we got our two predictions - and it
- so it was able to do that but then
what happens when when you have an
unknown number like you might have an
unknown number of predictions maybe this
week you've got 15 patients maybe next
week you've got 45 patients right that's
going to be very common and you want to
actually you don't have to hard code the
shape every single time so what you're
going to need to do is rather than when
you go to reshape at least program wise
I don't know any better way like make
converting like a list of Lists or
making a list of Lists based on a
dataset super easy converting to a numpy
array super easy I'm not sure how to
properly shape it automatically to the
shape that scikit-learn wants
what you can do is once you've made this
list of lists once you're ready to
convert you've converted already to
numpy rate once you're ready to reshape
all you have to do is you've got
measures dot reshape here we're going to
say rather than reshaping two to
negative one you just replace two with
the Len of example measures right and
now example measures can be any length
right will still run it and we'll still
get a report back that is two two and
that is how you can program it in to
just automatically predict on any number
of predictions and again this is not K
nearest neighbor specific this is any
any of the scikit-learn classifiers and
in general just how to properly reshape
your numpy array for a variety of
reasons but that's just one of the
reasons why you would is so you can feed
it through scikit-learn okay so there's
your real world example we found that we
can predict based on you know somewhere
between 94 and 98 percent accuracy which
is pretty impressive especially now that
you know the intuition of how simple K
nearest neighbors is and now what we're
going to do is actually write our very
own K nearest neighbors algorithm and
then what we can do is we'll we'll run
it on these exact same data and then
we'll see how we compare to scikit-learn
now just thinking logically here knowing
how simple K nearest neighbors is in
exactly how it works already should we
perform accuracy wise similarly or exact
or what compared to scikit-learn so
think about that and in the next video
we'll actually start writing our own K
nearest neighbors algorithm from scratch
so if you have any questions or comments
leave them below otherwise as always
thanks for watching thanks for all the
support subscriptions and until next
time