Testing Assumptions - Practical Machine Learning Tutorial with Python p.12

what is going on subscribers and others
welcome to part 12 of our machine
learning tutorial series in this
tutorial what we're gonna be talking
about is testing our assumptions so up
until this point it's been I would say
rather hand wavy in the sense that I
have just said hey these are the
algorithms and whatever they output
these are the answers to those
algorithms and we have done linear
regression in r-squared and all this and
so the question is we need to actually
kind of test all of these assumptions so
we've got actually two major algorithms
one is the equation for the best fit
line and the other one is the r-squared
or coefficient of determination so we've
got these two major algorithms that are
also comprised of many other algorithms
as we even saw just a few videos ago the
MIS placement of a single parentheses
changes everything and completely ruins
the entire thing so we need to be able
to test to make sure things are working
as intended so in the world of
programming this is there's a similar
kind of field and structure called unit
testing where we you know test each
little small unit basically that we can
in a program and this kind of helps us
from getting into trouble now this is
not going to be unit testing but the
idea is fairly similar we've got a lot
of ideas we've got a lot of inner
working parts and we want to at least
test them to make sure the easiest way
we can do that is by working with sample
data and by saying sample data that we
have the power to change so that we can
create a data set that is a more linear
data set and or at least the
relationship is more linear and then we
can test to make sure is r-squared
better higher right and and then also
just test our best fit line but for the
most part we're actually going to be
testing r-squared and if if if the data
is not more linear we can make it more
spread apart our squared should be lower
and so on so anyways let's go ahead and
do that and we can also confirm visually
that the best fit line is is indeed
working just by looking at it and seeing
whether or not
is indeed a best-fit liner what looks to
be a best fit line so first what we're
going to go ahead and do is import
random because we're going to be using
random numbers everybody the obligatory
pseudo-random if you don't say it's
pseudo-random someone absolutely feels
the desire and urge to comment and say
but it is not real random so anyways
pseudo random there you go
you uh nitpickers okay so what we're
going to do is just right under here
let's we're going to say define create
underscore dataset and then here we're
going to have we're going to pass some
some parameters first is how much like
how many data points do we actually want
to create here and then we're going to
say we'll pass variance and this will be
how variable do we want this data set to
be then we're going to pass step in step
will just be how far on average to step
up the Y value per point and we'll
assign a default value there and then
finally we're going to do correlation
and this is where we can just pass a
value and say we want correlation to be
positive negative or none and what we're
going to do here is correlation or hold
on so correlation will either be true or
false and then if it is true to get it
to go to be a positive correlation step
we'll just be some positive number right
because that's changing Y and to be a
negative correlation you would just
change this to a negative number right
so and in fact another way we could do
it is we could actually say correlation
is positive or negative and if it's
negative you do a multiplication of the
step and that's actually probably a
better way to do it either way would
work but we'll do that way actually so
the first thing that we're going to do
is well we would want to be able to at
the end of this I always like to build
the skeleton function first so at the
end of it what is the objective and that
would be to return the numpy array of
the
and for now again we will specify the
data type so we don't forget this later
on because it's probably going to be
useful later on so we'll say float64 so
that returns the x's and then we also
need to return y-values so wise and then
D type equals NP float64 okay so that's
the objective that we want to do and
then now what we want to do is create
some start creating at least some random
values so the first thing we're going to
say is we're going to start with Val
equals one so that's just going to be
the first value for y basically and then
we're just going to say why is this an
empty list and then we're going to we
could say something like for I in range
of and how many how what should this
range be well should be hm for how many
right so for range and HM what are we
going to do well we're going to say y
equals the vowel plus random dot R and
range and it should be random dot R and
range from the negative variance to the
positive variance so some range in there
is what we want to do first and then
we're going to say wise dot append that
Y so here we would just be in through
the range using that how much variable
and then we're just appending that
current value plus a random one so this
would give us data but really no
correlation if we actually wanted that
data so then what we would ask is so
keep in mind that Y is literally the Val
so it's just that starting value and
then our variance from that starting
value so this would be pretty worthless
at the moment it would just be somewhat
varied but not by much well it depends
on what you said the variance was anyway
so then what you could say now is if
correlation and correlation equals
positive
what we could do is Val plus equals step
which would in this case default to two
and then Elif correlation and
correlation equals negative what do we
want to do well Val minus equals step
finally at the end of the day what all
we're going to do is now we've got the
Y's so we just need some X's so you
could say something like X's equals and
we'll just do a one line for loop I for
I in range of what the land of Y's
that's good enough where you could do
how much for that matter anyway so now
we've got what we need and we're
returning some X's and Y's so to create
a sample data set we could do something
like and for example let's we can leave
this here for now but we're going to
comment it out just so we know that
we're working with our new data instead
so underneath this you could create a
new data set but I guess we'll create it
will create down here underneath all
these other functions so you could say
something like X's y z-- equals create
data set and then let's say we did a
recall that it's how much variance the
step in the correlation so let's say we
said we want 40 data points with
variance of 40 the step will be to and
correlation will make that positive so
now we have X's Y's we can print our
squared and all that fun stuff and let's
go ahead and run that real quick and in
fact la are we still pro we're still
graphing that prediction so let's we'll
get rid of the prediction we could
actually leave the prediction that might
be kind of interesting for now that we
might run in trouble I'm not really sure
if we're gonna get in trouble for that
or not but we'll just do that and
let's run it and see we might have to
change something else but I think that
would be everything we would change
awesome so here's our data set and sure
enough there's a nice best fit line for
us and we see that we would kind of
agree with that visually let's go
and graph that other plot though that
one and this will be a G prediction I
don't even see it it was 4 x equals 8 I
guess it would be right on the line and
then we're plotting the regression line
so I'm guessing the line is just going
right over it probably it's just being
drawn over it still not seeing it
however it was x equals 8 right so it
should be that it's probably this little
plot right here I'll zoom in it's there
I don't know if you'll be able to see
that on the on the video but it there
isn't need a plot there and in fact we
could do something like I think with
scatter it'll be s equals and then let's
try 100 so this is like for the size and
indeed there is a huge green there so
okay anyway so there's our prediction as
you should expect it's perfectly on the
line so we're going to close this out
and so now how would we attest our
assumption well recall that we've got
how much and then variance so if I said
if I took variance which is currently 40
and we saw that it was like 0.5 I think
for R squared
let's look at again well since it's
random data this time I was point 6 okay
so in theory if we if we decrease the
variance what should happen well what
should happen is that number should go
down pretty significantly so long as we
decrease variance significantly so let's
do it
let's do 10 we can save and run that and
as you can see it's much tighter
everything's there and sure enough the
coefficient of determination is very
very strong it's 0.92 much better than
before what if we change this to an 80
now it should be less than 0.6 and sure
enough it is less than 0.6 and so what
you can begin to do is automatically
write a program that's
simply calculates the coefficient of
determination for for just a sample
dataset and you would just make sure for
example that you'd start with 40 save
that number and then you would change
that to 10 and hopefully the coefficient
of determination was less than this
initial number and then if you went
greater it should be greater and so on
that would be a way to test just that
we'll call it a unit in theory you could
build a unit test out of this but this
isn't quite yet a unit test but anyway
so you can test that and then sure
enough the other thing you could do is
while you we had a positive correlation
if we change this to false we should get
quite an ugly data set sure enough we do
and the coefficient of determination is
almost zero which is absolutely not
surprising because that almost looks
like a completely flat completely flat
line and sure enough this data is
completely nonlinear so if you did have
a data set and you were trying to run
linear regression on this data set and
you came back with an r-squared that was
this number that's like point zero zero
zero seven you would probably be smart
enough to decide hey my data is actually
not linear we can't quite do linear
regression with this data that said you
can do other forms of classification
with the data or not just classification
but you know other forms of machine
learning I'm thinking classification
with your data doesn't necessarily have
to be linear and in fact class a lot of
classification is should be linear in
some way but we'll get there anyway um
that's enough for now I think but just
kind of keep in mind that when you
create big big scripts like we have here
in big programs that are kind of based
on a lot of things you want to make sure
that it's about right we could check the
best fit line ourselves kind of visually
but R squared we could not really
totally test that but you could
definitely program something that would
go through and like I was saying check
to make sure R squared was acting
according to our assumption or our
knowledge of how it ought to act so
we're basically done with regression but
I want to make a quick edit to this
video to cover two pretty important
things one is a fundamental aspect to
machine learning that
be getting overlooked using the really
simple example that we that we've used
here and then to I made an error that I
think is bad enough that we want to
cover it plus I think you can learn a
little bit from from the mistake that I
made so let's let's pop over to the code
and address these two things hopefully
pretty quickly so first of all looking
at the data I'm going to change this to
from 1% basically due to 10% now we're
going to run that and we're going to see
that it's basically an exact copy of
like the data leading up just shift it
in price a bit right so coming over here
it's basically the same
this version is squished up a little bit
and that's just because the blue line is
the prediction line that plots even on
the weekends and holidays whereas over
here the stock price only occurs during
Monday to Friday and not on holidays as
well so anyways basically an exact match
just higher in price and the reason is
kind of twofold one we've created a
linear model that is going to attempt to
do this but then also we've made a
mistake as so so we'll address kind of
both but anyway the first thing is in
the biggest mistake actually there was
two mistakes one I noticed in the video
just going back over it I'm pretty sure
it was here there was also a colon at
the end of the X I don't know why that
was there no one actually brought out
that one I just happened to see it right
before filming this one anyway that
basically is x equals x right all that
says is X up to forecast out and then
finish the whole thing right that
doesn't do anything so that was just a
typo but then you get to this point and
we're still kind of in a world of hurt
because X what we were intending to do
is say ok X is the first let's say in
this case it's 10% yeah so the first
we're saying X is the first 90% of data
this is the stuff we're going to Train
against and then we're saying X lately
and our objective here was to say X
lately is the last 10% but instead what
we've done is we've sliced X and
redefined X here and then sliced X after
it's already been redefined so this is
actually minus forecast out of the 90%
so obviously simplifying things a little
bit this is the basically up to 90% and
this is the
10% of that 90% it's a little bit more
but anyway so that was just that's a
failure in logic okay so really the fix
that you just cut that paste it there in
there you have it now this is still
going to create a model that's
relatively AK and and very similar to
what we've already seen and again this
is because we're using linear regression
it's going to create a linear model that
resembles what we've already seen so
again you've got some jagged then you
got to jump up and then price it's a
little different but it's very very
similar okay so anyway that's that's
just given what we've done and how we've
trained it that's going to happen so now
let's talk about the last thing which is
the fundamentals of you know what kind
of features should you train against so
what was the objective here first of all
let me just say the reason why why we
did it this way is just for simplicity
sake we're just trying to do a really
simple regression example but let's say
you know regardless of whether or not
you're interested in stock investing
this problem is every machine learning
problem is going to likely be a somewhat
complex problem so you have to think
pretty logically about the features that
you choose to use so looking at this
which of these things hinges directly on
on price or will directly impact price
right well obviously adjusted close well
what about high minus low percent
doesn't matter what the price is not
it's a percent right it's a normalized
value so that doesn't have anything to
do with price how about percent change
no right these may be volatility right
may be magnitude same thing with how-
low percent volatility in like direction
maybe but not price what about volume no
not price right this is just magnitude
kind of fluctuation maybe stuff like
that volatility so the only thing that
really hinges on price is just a close
to illustrate that despite training on a
future value that is indeed price what
we can do is we can actually drop
adjusted clothes from the features and
what do you think when we drop this what
do you think is going to happen before
we graph it is that going to create a
similar line that follows price is it
going to be a falling price upward price
flat line what's it going to create for
the prediction so think about that we
run it
it quick and we'll get our answer and
the answer is not going to be probably
what we were hoping for right it's just
more of a flatline and why do we get
this right well the high - slow percent
you probably had very similar high - low
percents back when price of $400 $600
$800 right not big differences the only
thing that might be sort of impactful is
the adjusted volume since probably less
people are quickly flipping an $800
stock as opposed to a $50 stock or
something like that but regardless these
just aren't the greatest features so
thinking about your problem
in this case it was stock investing what
what is it what is a stock price
indicative of it's indicative of the the
entire company's value let's think of
Google for example like 500 billion
dollars I think why is Google worth 500
billion dollars is it because of the
adjusted close high - low percent
percent change adjusted volume no come
on be logical about it you know that's
not the case there are people who
believe in pattern recognition and stuff
like this but or at least you know chart
patterns in stocks and sorry but it's
been tested but there's plenty of
research done that doesn't work but
anyway some people still believe it but
but fundamentally why is Google worth
500 billion dollars it's not because of
this stuff fun fundamentally Google's
worth 500 billion dollars because of
things like its quarterly earnings it's
price to earnings as price to earnings
to growth it's Book value and so on
these are the things that value the
company so if you wanted to predict
stock price you would use features that
attempt to predict the company's overall
value and then from there you can divide
that by outstanding shares and get a
specific share price for the company but
anyway this was just meant to be a very
simple example if you want to see a more
complex example of doing investing with
features and fundamental features of
companies I do have a tutorial series
out for that it's like 30-something
videos if I recall or maybe 20 or
something but it's kind of tedious
because you got quarterly earnings which
is that recorder then you've got things
like price to earnings to growth which
you could measure all the time Book
value price to book you can measure all
the time and so on so a lot of these
things and also just the the
entire company's values that you know
changes as the day throughout the day so
anyway it's it can get really complex
really quick so we just wanted to use a
really simple example but if you if you
are looking for a more complex version I
do have one but anyways that's it with
regression hopefully you can learn from
my mistakes down here you'll all
probably continue making mistakes and
you'll probably make mistakes too and
that's just that's just like part part
of it honestly so luckily we could
visualize this and we could catch it but
a lot of times you're not going to be
able to visually catch it so you want to
like read and reread and all that your
code but still you're going to make
mistakes so unless you're a robot or
something
so anyways hopefully you can learn from
my mistakes otherwise we're going to be
leaving regression behind now and
traversing into classification so stay
tuned for that as always thanks for
watching