Bias Variance Dichotomy Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to our lecture on bias
variance dichotomy this is a conceptual
lecture so you must be right now going
through lectures we're learning many of
the machine learning tools and
techniques and this is not one of them
so we're not teaching you a new
technique in this lecture but we're
introducing a very important concept
that's that's prevalent in machine
learning and therefore it applies to all
the techniques I would say that you are
learning and I will go and so far as
saying this is probably one of the most
important concepts that you should
understand initial learn okay so let's
dive into what the core concept is the
concept can be said in very simple
sentence the idea here is that while
adding complexity to a model well if you
if you want to keep adding complexity
complexity to a model you might improve
the fit of the model right so you might
find out that you're better describing
the data but that need not improve the
predictive accuracy of this model when
you compare it to new data that you get
okay
and this concept is true for whatever
type of model that you look at so you
can take an example in linear regression
and you can add complexity to it by
either adding more input variables or
even with one input variable you can add
more complex transformations so for
instance one way of just adding
complexity to a simple problem where you
have one inter variable one output
variable is missing I'm not really
interested in looking at it through the
standard interface but I'm also I would
like to look at it as a polynomial so
what is the model when you have X X is
the input variable but you can also take
square X cube and so on so you can have
a more complicated fit between y and X
because then simple regression you've
always seen Y is equal to MX plus C but
what if we had y is equal to M 1 X plus
M 2 X square plus C so you can add
complexity that way you can add
complexity by adding more variables you
know this is again not confined to the
regression framework
almost any method that you take you will
typically find that there's some way of
getting more and more complex so for
instance you might have already covered
trees right classification regression
trees you can add more complexity by you
know creating more and more and more
branches to the point where it has such
a complicated tree you have such a large
tree where this terminal node or leaf is
a single data point that you're using in
your training set so you have really
there's no complexity but kadian is
named was the choice of K could could be
a same complexity with noodle networks
something that you will learn in the
future the number of layers in the
neural network can mean to us and this
idea of complexity will become more
clear as we you know talk through some
example but the idea is that you can
make the model more and more come to us
the model that you are going to use to
create this relationship between the
input and output weight input variables
in the output variable that can become
more more more complex and in the more
complex it becomes you will do a better
job of fitting the data that you have
but as the meaning you are creating a
better model and the answer is it might
not right because while you may fit the
data better that you have tomorrow when
you need to predict using this model you
might not do a better job of predicting
alright and we are going to see how that
can happen the campus area now you this
code concept
machine learning is also sometimes
referred to as Occam's razor that's more
of a mathematical concept and it's
definitely used prevalently in machinery
and the idea there is not well at what
register SPECT idea there is quite
simple which is that if there are two
models with equal predictive accuracy
then you prefer the model that is
similar that is less complex so that's
not what about them but you see how
these two highly related concepts with
the concept of biased feelings dichotomy
yeah essentially questions saying that
oh I can add more complexity to the
model and it will look like it's doing a
better job of fitting the data but am I
getting better predictive models okay so
let's let's actually you know understand
this through an example in this
particular model in the two tables that
have shown you here is modern one which
is your good old linear regression that
you know and here is the data set okay
so use the same data set which is the
same X on the right hand side and the
same y right but I say that I don't
necessarily believe that this is the
right model that is model one is the
right one what if the relationship
between x and y were more complex so I
add an X square term and X square is
nothing but it's really simple right
it's just I take X and I square it and I
create a new column and like that I keep
on adding columns till X to the power 9
okay and finally I have I have all of
these as potential input variables as
described by this model
so Y is some function of all of these
parameters and I do a multiple
regression so how does this work up this
is your here is an illustration here is
my linear model okay here are the data
points that you saw before in the
plotting and here is the line that you
see is the fitted line
the statement seems fairly standard and
and so this is what you get
how does still what I'm so the
polynomial though the one that I created
before is actually a ninth order column
right so how does that look
well it turns out that it does a really
good job of fitting all the data parts
as you can see this ninth order
polynomial described by this black curve
goes through every single data point and
that should not be surprised you have
ten data points and you're using a ninth
order polynomial to fit it there is
enough flexibility in the model in the
coefficients is enough complexity in the
model where you don't you can actually
go through each data point here in the
first model it's a straight line right
even if it if it wanted to it cannot
even if I have and I'm using the
flexibility to put this thing wherever
you I want right I can move this name up
and down I can rotate this line but the
best job that I can wind up doing is
actually the line that you see on the
screen so I did a simple linear
regression which does try to you know
fit as many data points as possible and
this is as best as it could do now with
a ninth order polynomial when it tries
to do that it does a really good job
fits all the data points so you're
sitting there going like wow so maybe
maybe my system is neither a polynomial
but it's not I'll let you in on a secret
this actual data between X&Y
was created by an actually a linear
system with some amount of noise right
so you true equation the true
relationship which because I'm under the
guard okay I'm the one who's actually
creating this these data form so I
the Oracle I'm letting you in on the
secret that I actually created this data
points through some y is equal to beta
naught B naught plus b1 X and not X
square and X cubed but there was some
error some noise right like any regular
system
so given this less it looks like this
the night or a polynomial still did a
better job of sitting right the fit is
great but let's look at how this more
these two models compare when they have
to predict another effects right here is
the fitted model the same graphs as
above in the linear model but I create a
new data set right I create a new data
set and I see how well my line does as a
job of predicting what's going to happen
next and the answer is it doesn't go too
bad right so this is the predicted line
the blue line is the predicted line that
I got from my training data now when I
go get new data it looks like I missed
up in a couple of times and I probably
should expect that given what I know
right now make that that there is some
amount of noise in the system that is
irreducible but what this basically
means is that tomorrow when you come to
me with saying that a 4.5 what do you
predict I say I'm going to predict this
value and in reality and wind up seeing
this man okay the the line in red is
what I wind up seeing in the field when
I make a prediction and the prediction I
make is where I wind up what I wind up
using as ruler okay now what do you see
in terms of predicting with the ninth
order polynomial you actually do quite
terribly bad I mean look at this data
point right so this data point where
it's one point five my prediction at one
point five would be something close to
minus four minus 13i or whatever right
so
fine if I were to use this ninth order
polynomial as my fitted model I would be
predicting - potty but look at what I
actually got right I got plus 50 so I
was almost off by I was almost off by 55
units whereas I'm never off by that much
with the linear model and you're going
to see the same kind of weird practice
here in fact while at something like 0.5
this model just goes through the roof I
can't even fit it inside the drawers so
clearly the ninth order polynomial while
it's doing a great job of fitting does a
very bad job of predicting right and if
my goal more often than not I should say
okay my goal more often than not is to
really come up with good predictive
accuracy so most we've seen many
problems you're just not trying to fit a
model to the data that doesn't buy you
too much but you want to pay it the
model that can be generalized that can
be generalized to other situations
so tomorrow indicated intercept because
what are you going to do with the model
you're either go to predict early
they're going to interpret the model in
either case you need to acknowledge that
what you have the data that you have is
nothing but a sample and if you take in
another sample if it turns out that you
would have told a completely different
story then maybe the way you're doing
things is not really correct my core
point is that for instance if you had
seen the red star data right in the
linear regression model you might have
created a slightly different line maybe
the line would have looked a little bit
like this right but think about what you
might have done with the complex
polynomial you might have created a you
know completely different polynomial
function that look then again we'll go
through all that I data bound but if for
one sample you create one story
for another sample you create a
completely different story then maybe
the way you're doing things is not
really active so here's what we showed
you the system where it was truly a
linear system until the whole using a
linear model made more sense than having
a more complex model now what happens
when there is more complexity to the
system let us for instance say that my
true model was quadratic okay that's
what I'm going to show you I'm gonna
take a quadratic model and then see what
happens when I try to reuse it okay so
here is a quadratic model this is the
truth that the machine learning
algorithm does not know that in the
world I will never know that the true
system is quadratic the only thing that
I have is data substantial showing you
the data I'm just letting you in on the
secret that for today's exercise
I created this data so this is the data
this is the model that I created this
data using this model and adding some
amount of noise or uncertainty okay so
this so the model that is creating this
data is actually us you know this
moreover Y is equal to beta naught plus
beta 1 X 1 X this is because there's
only one X and beta 2 X square so I'm
using some model like this with some V
naught B 1 B 2 plus some amount of noise
okay obviously if you knew that this was
the model then this is the model you're
going to try and fit right I mean if you
knew this was the model then this is the
model you should use with the truly
known b1 b2 so you don't even need to do
any kind of machine learning or
statistics and scissors much sadly
you're only given the data and you're
not told which monitor it is now let's
see what happens if you had this data
and then you try to fit a line right
this is one fit but at the end of
give you a feel for what happens when
you do as many times I generated another
set of data points using this model so
I've shown you only one set of blue dots
blue small many circles but affect me
another set of blue mini circles and
then fit in line
I fit another than but one thing you
should notice is look all these lines
are in general more or less the same
they try to do the same job and in
general they might have failing
chronically in certain cases they always
wind up under estimating in this region
because they're always under the truth
they always wind up overestimating so
each time I do this exercise it looks
like chronically off in certain areas
but I'm fairly consistent each time I do
this exercise I wind up kind of creating
the same line you know what happens when
you have a ninth order power what
happens is you're still not doing too
great and the reason you're not doing
too great it is because this is a
quadratic system when you are trying to
fit something so complex
all right so you're still overshooting a
lot but there are a couple of things to
notice none in general as expected as
shown in the previous slide you're not
always telling the same story one time
you're telling one story the next time
you know we are predicting vast either
through this kind of extreme readings
from one kind of prediction to the other
it's not seen in the linear fit but take
another look we are not always being
chronically off in certain areas yes
this is only five such fits but imagine
if you had five thousand such fits right
even if you had five thousand fits in
the Linea model you will always be
overestimating some billions will always
be underestimated some things what's the
nuts called bias whereas in in the ninth
order polynomial the idea is that yes
there's so much variability but if you
were to do many many many fix the
average you might not be off from the
blue line and that's useful in concept
it's not useful in practice because in
reality you're going to get only one
data set and you're going to try one fit
so if it's off its off whether it's
because of whatever reason but it really
helps to understand why it's all right
here in the linear fit it's off because
you may be chronically always going to
be off because you're trying to fit a
very rudimentary model a very simplistic
model for something that's a little more
in reality more complex the model the
reality is more complex okay so this is
more complex because it's a quadratic
alright and the moderate right the fit
is too simplistic because it can even so
the model can maybe be a line so it's a
line which is you know simple whereas so
here what you're going to witness is a
lot of some agency way to be always off
but is out here in the ninth order
polynomial this is so much variability
because you're just getting fooled by
the pure ones it's the same thing that
you saw in the previous equation nothing
is really in the previous slide nothing
is really change it's still trying to
fit something that's a model that's
overtly complex to a system that is not
that complex right if we went from a
linear data source to a quadratic data
source but that's still doesn't warrant
a 9th order polynomial fit there's a lot
of variability and this is the point
that gets captured in what is often what
is really described is a bias variance
dichotomy this graph is taken from the
history intentionally in Freedman book
elements of statistical learning and it
captures what we've kind of tried in the
state in the last two slides which is as
model complexity course increases it
looks like you are doing a very good job
of fitting the data this is nothing but
the fit
you take the data and look your error
keeps you want very well so as you keep
on making more and more more more
complex model you're going to go through
more and more data points right but at
some point your ability to predict so
this is prediction on you Nina right the
red line is prediction on new data the
blue line is the fit on the data that
was given to you for training so it's
called the training sample and the
prediction is down on the test sample
your prediction keeps on getting low to
a sweet spot of model complexity and
then after that it goes on and what is
that sweet spot you the sweet spot would
be attained if you were able to match
the exact complexity of a system so if
you had a quadratic system for instance
and you use a quadratic model that would
make them in the sweet spot and so if
you have a complexity that's perfect you
know N and what we are going to do is if
you don't you in the individual well you
know what the true model is right so how
do you figure out what this complexity
should be and that is going to be
covered in our lectures of invalidation
right how do you validate a model or how
do you fine-tune some parameters of a
particular model again it can be nearest
neighbors it can be trees in the neural
network we support vector machine it can
be a simple regression but if there is
some kind of tuning parameter that can
increase or decrease complexity how do
you go about increasing decreasing
complexity is seeing what works best and
then choosing the appropriate one that
we capture in validation the lectures on
navigation
when this lecture we want to create an
appreciation then as model complex
increases the fit
becomes better
the prediction you need to find that
sweet spot of moral complexity and
that's the core idea the other thing is
also the other idea here is that yes
when modern complexity is low you don't
do too well in terms of your prediction
but the reason for that is because there
is high bias meaning that in that linear
regression if you remember right you
were always off when you were trying to
fit that linear regression to that
quadratic function you will always often
certain regions so you had a high bias
and Louise what we mean by that and I'll
go back to that slide is you were biased
in certain regions so these are regions
where you have bias all right you're
always going to be off right because of
the nature of you've tried to fit a line
through a curve but you have very low
variance the variability between many
such fits is a loner right so on any
given day you get any given data set
it's not like able to come up with a
completely new equation so that's what
we call as high bias and low variance so
I bias
and lo Bayless okay now you step over to
the ninth order polynomial sure the bias
is not that high it's not like I can
tell you that you're going to
chronically be under predicting or over
predicting in some regions so this has
low bias right so I can't tell you up
front that you're going to be always off
in one direction but it's got high
variance right what we mean by that is
on any given day if I take any given
data set I did I take a sample and I try
to fit this follow me I don't know which
line I'm going to get I mean this line
predicts some astronomically high value
up here where is this curve predict some
astronomically low value up here so this
is like such high variance on a given
data set I don't know how I'm going to
be predicting and therefore if I
couldn't have such high variance it just
means that I'm probably not going to do
a very good job of predicting right I
don't even know what I'm going to be
predicted so that's the concept between
the bias phase dichotomy which is that
when you go for lower model complexity
you get high bias and romanians
I mean you go from I am waterproof XD
you get low bias in five minutes and
this is a very important concept to
internalize with respect to machine
learning and it is going to be extremely
important even in terms of applying more
advanced statements so I hope the
bias-variance dichotomous clear thank
you