Model Assessment and Selection Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to our lecture today
on model assessment and selection so
this lecture is in many ways similar
both in style and content to our lecture
on the bias-variance dichotomy it's
similar in style in that we're not going
to be teaching you a new technique but
we're teaching you a concept in machine
learning that's really important and so
like the bias-variance dichotomy it is
really applicable to almost every
technique that you could be using in
machine learning
I should not just say every technique in
machine learning more narrowly in every
technique in supervised learning right
so you're regressions your neural
networks your cards I mean your decision
trees your SVM's
you know so whatever whatever techniques
you're learning right now and whatever
you have learnt this is a very very
important concept it's also very similar
in content because in the bias-variance
dichotomy lecture you really learnt
about how it's really important to
understand that you can have highly
simplistic models which would have a
high degree of bias and low variance and
therefore not really good and at the
same time you can have highly complex
models which tend to over fit the data
and therefore have a lot of variance but
very low bias and and therefore you will
we introduce a problem there saying that
well you it's not a good idea to go to
either extreme in today's lecture we
will be answering coming up with
solutions for that problem and not just
that problem for many other related
problems but somewhere the solution is
in looking more carefully towards how do
you assess a particular model and
therefore how do you go about selecting
between multiple models okay
so jumping into the subject in terms of
when you have data and you're trying to
relate that use the data towards models
right there's one part which is actually
using the data to train the model okay
there's another part which is you could
use some of the data not
strain the model but to choose between
models or to fine-tune the model so
using the laser to train the model is
model training model selection the word
selection is used kind of loosely here
could mean selecting between multiple
models or it could mean you know and
meaning you could use that to select
between completely different models or
at the same time you could use that you
can fix the model and it could just be
like parameters that define the model
and you could go about figuring out what
value to set it to by using by using
data in the model selection and finally
you have the concept of model assessment
which is once you've fixed everything up
right if you wanted to get an idea of
how good this model is so that you know
that when you take this model to the
field this is the performance you would
expect that has to do with model
assessment and corresponding to these
three goals or objectives you could
break up your data and this is something
that you might not always have the
luxury to do but you can if you if you
have a data rich situation where you
have enough data then you could
potentially break up the data under
training validation and test where the
training goes towards model training the
validation goes towards model selection
and fine tuning and the test goes
towards model assessment and there's no
formula exactly as to what this break up
should be or what is enough data that's
a very that that's that's a hard
question to answer
it varies from case to case but at the
surface you know typically if you have
enough data you tend to break it up as
either 50% 25% 25% and finally 25% out
here 50/25/25
but you could also have you know 60 20
20 and again even for this you can't you
can't say one is the right one is wrong
or something is perfectly right or wrong
I'm just giving you some values that are
typically seen you can also see in the
picture that have the the the
representation I have below that
sometimes you might want to do training
and validation together and we will talk
about that more specifically but if you
this sometimes comes up when you don't
have as much data where you can't you
don't have the luxury of just taking one
25% and calling it validation so
somewhere you know you just use the same
data to perform training and validation
and how you'll do that is something that
we're going to talk about in good detail
in this lecture but you still try to
keep some percentage for model
assessment that the final say on how
good the model will be if you take it
out to the field if you if the model
were to completely look at new data and
try to predict how good will it work
okay so what we're going to do next is
we're going to take each of these faces
and talk in detail about what what
really we can do for each of these
objectives the first phase which is
model training on is one where you take
the data and you train right what we
mean by training out here is really that
you've fixed everything you've fixed all
your steps you've fixed or you've chosen
a particular model and you've chosen all
the parameters associated with this
model so this phase you're not doing any
model selection in phase one you're not
doing any model selection no selection
you're also not you also not trying to
decide on what values that some of the
modeling parameters should be what I
mean by that is let's say you're doing a
Ridge regression in Ridge regression you
have a complexity parameter if you go
back to our lecture and rigid regression
you are doing this optimization where
you're doing the ordinary it's it looks
like the ordinary least-squares but you
have another
right which this is the term associated
with regularization where you penalize
really large coefficients so for that
you need to set a value you need to set
a particular lambda you've already
decided that in this phase right when
you go in for data for training you fix
the model and you fixed all the
parameters associated with the model if
you're for instance doing a neural
network training you're not using this
data to decide how many intermediate
nodes should be there that's not what's
happening once you've finalized on the
exact model and its details you are just
using this data to figure out what the
model should be meaning take the case of
the linear regression right you fixed
everything you fixed what the input
variables are you fixed what data you're
going to process with respect to a
simple linear regression or a multiple
regressions there are no other
parameters to fine-tune so it's you just
plug in the data and you get the betas
all right so here you might just be
getting the betas so so that you've
created the model you're essentially
creating the model in this phase if
you're doing decision trees then let's
say you you you need to have fixed on
the algorithm for the decision trees
need to have fixed what the input
variables are you need to have fixed
other parameters that you could probably
play around with in decision trees and
we're going to talk about those
parameters soon but once everything is
fixed you just plug in the data and out
you get R is a tree structure tree
structure or network in the case of a
network so this is essentially the last
leg once you fixed everything you're not
doing any model assessment you're not
doing any model selection you already
selected your model you've already set
the parameters for your model you're
just plugging the data into this
finalized version and getting the actual
model out of it so that's what you're
doing in this data for training what
happens in the validation phase in the
validation phase you can you you're
essentially using this data so this is
separate data right you use some data
for training
think about this you using separate data
and you're using this data to to make
some decisions you could be using this
data to select between multiple methods
right so you might say hey I have this
data I could do a linear regression on
it I could do a regression tree on it or
a random forest or something I could do
a neural network how do I know which to
do why not do all of them and see which
one does better right and how do you do
see which one does better in the
validation phase what you'll do is you
use the same training data that is you
use the same training data that you saw
we discussed in one all right
and you'll fit these three completely
different approaches you create the
beta's for the regression you'll create
a tree for the regression tree you
create a network for the neural network
now you go apply these three alright you
go apply these three to the validation
data what I mean by that is that you go
take the input data in the validation
and apply these three methods and you
get some predicted outputs like each of
these methods are going to give you some
completely different I mean might not be
completely different but they're going
to give you different outputs different
predictions compare these predictions to
the actual output that you have in the
validation data right any when we say
data whether it's for training
validation or test it means you have
input output pairs right so the if you
have five input variables you'll have
values of each of those five input
variables one data point is nothing but
the vector of all values of the five
input variables and the one output
variables so whatever value that is so
in some sense when you have these input
output pairs what you can do in the
validation phase is you would have
created the model the models from the
training phase and what you're
interested is in comparing multiple
different models so you might have
completely different methods and then
you apply the input of the validation
phase to get predictions from these
methods and come
these predictions to the actual output
and see how well they did and you can do
that by both in a regression context or
a classification context meaning that if
you predicted 13 and it's 13.5 I can say
are you missed by 0.5 if you predicted
that it's going to be a class like I'm
predicting that there's a 70% chance
it's male and then it winds up being
female you can say you would you can use
a different kind of a metric to measure
that performance like Miss
classification index or entropy or
something but essentially the idea and
the validation phase is to choose is to
make some decisions we spoke about how
you can use that to choose between
methods but more often all right what we
see is that it's it's used extensively
to fine-tune parameters for a given
algorithm so let's say I go about and
I've decided right at this stage so
we've already talked about let's call
this a B and C so we've already talked
about 2a okay in 2b what I'm really
talking about here is let's say a fixed
I've decided that I'm going to use a
decision tree okay so I'm going to use
the cart algorithm now the cart
algorithm might have a lot of parameters
that I need to set one potential
parameter could be the minimum number of
leaves in the terminal nodes what that
means is the minimum number of data
points in a terminal node of the tree
right so in a cart algorithm you can
keep on splitting the tree into branches
and branches still that the terminal
nodes just have one data point the idea
is where do you stop and one way of
choosing where to stop is to have a have
a limit you put in a limit saying don't
create new branches if they're going to
result in the final nodes having less
than let's say five thousand data points
or we call them five thousand leaves but
what should that be should that be five
thousand should that be five hundred
should that be 50
right what you can do is you can run all
of them you can create three completely
different trees one with each version
and again validate that use the
validation data set to see which does
better right and for pretty much any
method any complex enough method you'll
find that the method the method itself
will have some parameters that you need
to finalize so in the case I guess you
guys have finished you might not have
started neural networks yet but you must
have finished so we finished a rigid
regression right so again let's let's
think about rigid regression in rich
regression we know that what what rich
regression does is it does this process
of regularization by introducing a new
parameter called lambda right lambda is
what forces is lambda is kind of like
the the way in which penalty is applied
towards that optimization problem and
constraints the size of the betas so if
you have a very high lambda that's like
a very harsh penalty towards large sized
betas or coefficients in that linear
equation so Ridge regression goes ahead
and applies this penalty now if that
lambda which is this parameter is very
low it's close to zero then there is no
penalty the Ridge regression becomes
like the ordinary least-squares
regression right it's it's a it's
actually identical if that lambda is
exactly set to zero but then the
question comes about what should I set
lambda to I understand the concept I
understand the concept that I might want
to apply some amount of penalty to
really large betas and that was what we
discussed in the course on
regularization saying you know I don't
you know I can if I have
multicollinearity in my data I might
have two betas that are of two opposing
magnitudes that are becoming really
large and so on and so forth right but
what should I set the lambda to be in
the ridge regression in the process of
rigid regression I need to set the
parameter lambda to some value and what
do I set it to is there a right answer
to it
the answer is you might not know that a
priori as to what to set lambda to but
you can use this validation phase right
you can set lambda to 0.25 set lambda to
0.5 set lambda to 0.75 and create three
different Ridge regression equations
three different predictors now take the
input data from the validation set and
up and apply that on all three
predictors all these three predictors
will come up with some predictions three
different sets of predictions of the
validation data compare the predictions
of each of these predictors compare
these three predict sets of predictions
to the actual output data of the
validation data set and you might and
you can then compare these three and say
oh it looks like when I set penalty
parameter lambda enriched regression to
0.5 I do better than when I set that
parameter to 0.25 okay so I get this you
can go to the back I know you finished a
subset of techniques and you're still
going to learn for instance neural
networks and some other methods but what
you should take you know commit to
memory in some sense is that almost any
machine learning technique that you
adopt you'll have to choose some
parameters and this validation can help
you choose the parameters Oh another
example I can think of is for instance
the K nearest neighbors what should your
value of K be right am I supposed to be
looking at five nearest neighbors six
nearest neighbors ten nearest neighbors
one nearest neighbor what should in a
careless neighbor algorithm that we
discussed in our class where we compared
regression to K nearest neighbors what
should the value of K be and you can use
this validation to figure that out okay
the last use case in some sense of the
validation phase is one that can also be
used to choose the correct number of
input variables and to also say which
ones those input variables should be
such choosing the correct number and
also choosing those which so I can say I
need five input variables are not six
but which five should they be so in that
sense and this kind of really links up
with best subsets
right where you're trying a whole
combination of different inputs again
the idea could be that I can have a
fixed model I could say I'm going to use
rigid regression like I'm going to use
regression or I'm going to use trees
with this parameter but which inputs
should I be using should I use all my
input variables should I use input
variable four seven and nine you know so
that decision also sometimes like we
spoke earlier about regression your R
square will just keep on getting higher
and higher as you keep on adding more
inputs that does not mean you're getting
a better model right so choosing those
inputs is you can choose a model with
inputs ABCD another one where you use
inputs d e f g and then and you can
really compare the performance of these
two different models on the validation
data set now there are some techniques
already there that we have discussed
which helps you choose what the input
should be and we've spoken about that
using metrics that are more complex and
r-square like for instance i adjusted
r-square which you know goes ahead and
you know penalizes more complexity and
there are a whole bunch of blanket
approaches to penalize models but so the
spoke of adjusted r-square there's
something called the key key information
criteria AIC and then there's the
bayesian information criteria so there
are a whole bunch of metrics that will
just go in and say blanket I'm going to
penalize you for adding more input
variables but we don't need to do that
well that might be computationally
convenient and it is you could if you
have the luxury of creating this
validation data set a simple thing that
you could do is you can just try these
different combinations of inputs for the
chosen method of doing the training and
so on and apply them and and see which
approach does does better so you you
don't so you could just ultimately wind
up using prediction error instead of say
something like I just add a square
you can use a prediction error because
you're not just blindly looking at how
good the model fits the data that you
train the model on you're now taking
that data that the model has never seen
when it trained and you're using that to
decide which what what your input
variables should be or how many input
variables you should have so the
important thing is with the validation
itself and this whole process of model
selection is not just one thing you
could use validation really to choose
between multiple methods you can use
that to fine-tune the parameters of the
algorithm and you can also kind of use
that in some way to choose what your
variables should be okay
finally you have the test phase now
we're at number three right the test
phase and we had three and now we're
talking about three the test phase the
test phase the purpose is different it's
not to fine-tune the model it's not to
help you come up with the best model you
could have come up with a good model you
could have come up with a horrible model
the test phase says I don't care my job
the job of the test phase is fairly
simple the job of the test phase is to
give an accurate idea of what the
performance of this algorithm is going
to be right and you can turn around and
say why don't I just use you know
something like my prediction error that
I got in the validation phase and the
answer is you cannot because you have
hoped or he'll I mean you can go even
one step further and say why can't I
just look at the predicted the error the
residue will sum of squares in some
sense the error that comes from the
training right you cannot do that for
the simple reason that you are
explicitly fine-tuning the model
fine-tuning the coefficients fine-tuning
the parameters irrespective and this i'm
loosely using these terms because it
applies to any approach you take whether
it's decision trees whether it's
discriminant analysis whether it's
support vector machines whether it's
noodle networks whatever approach you
take there's a whole bunch of
optimization going on these approaches
right here you're sitting there in rank
make this model make sense out of this
data that process itself is going to
make you do very good on this data now
to get a true picture of how well you'll
do when you see new data you don't want
data that was used to make this method
good in the first place
right you use some data and the training
data to figure out the coefficients you
use some data in the training data to
construct the actual tree or the actual
network you use some data and the
validation data set to fine-tune what
your parameters should be why so that
you do a good job of fitting now you
don't want to use the same data to tell
me how good I will perform when I go
when tomorrow someone comes to me and
says ah here is this input data can you
come up with a prediction for me all
right to do that you want to really look
at data that has never been looked at
essentially this is you you should think
of this as data that's been kept in the
vault and never looked at and you're
bringing it out in the end after all the
modeling has been finished you've used
you've used the training data to
construct the trees you've used to
select the validation data to fine-tune
it
you now have end product the end product
in a regression could be a set of
coefficients like the betas then product
in a tree could be the tree structure
whatever it is right you basically have
a finalized end product and you want to
now see how good the end product works
now pull out this data pull out this
data provide the input data from the
test data set ask the model to make
predictions compare it to the actual
output and you're just going to report
that you're not going to use this data
to make any more decisions in terms of
which method to use or you're not going
to use this data to fine-tune anything
about your model okay because if you're
doing that you're using the data to make
the model better but you're not giving
any longer an accurate impression of how
successful
you would be if you were to completely
see this data afresh and you were to
make predictions you're not you're not
giving an assessment of the model okay
so in order to give an assessment of the
model pull out this data that has never
been seen by the model in the end and
the only purpose of this data is to see
how good the model is not to construct
the model or not to fine-tune the model
okay so that's the test phase now we're
going to talk about in the last part an
approach called cross-validation and
it's it really ties to this case that I
showed you in the first slide where the
training and validation are kind of put
together this is an approach that really
kicks in when you feel like you don't
have enough that much data for a
separate validation set and you want to
kind of squeeze more out of it okay
so a simple approach there is you just
have the training and validation set as
one large data set and you can break it
up into n chunks so I'll cure of broken
it up into five chunks going between
five to ten is typical and the approach
is called actually k-fold
cross-validation so it's K fold where
you have to choose a particular K okay
and here I've chosen five and so
basically broken the data set into five
chunks obviously very important guys is
that this breakup isn't it is random so
if you're given the data in some order
you don't want to very you know
conveniently just break it up into five
chunks and because there might be some
kind of you know implicit order maybe
the data was for instance
chronologically ordered right you don't
you don't want to create training data
set one of the early chronology later so
you you kind of want to shuffle the data
up and then break it up into five bins
and the idea is to use any four bins to
train for to train and one to validate
so you might sit there and say hey you
know this just sounds like you're
breaking up the training and validation
at the later stage then at an earlier
stage but answer is no there's one more
step once you do that shuffle them
around meaning now
in this particular graph I've shown you
let's call them 1 2 3 4 & 5 right in
this graph you're using 1 2 3 & 5 to
train and you're validating so this is
strain and you're validating with for
now permute oh okay in the next step do
1 2 3 4 and validate on 5 next step
permute 2 3 4 5 and validate on 1 and so
like that you keep going till you've
done all the 5 combinations where you
would a validator and separate and you
take the cumulative validation results
so you'll take the cumulative validation
results to see how well to make
decisions in terms of validation and so
this is called k-fold cross-validation
so one question that comes up fairly
frequently is what should the value of K
be in a k-fold cross-validation given
your tentative idea that it's typically
somewhere between 5 and 10 but again
that's not something that's cast in
stone and it's more important that you
understand what it means to choose high
value of K versus a low value of K in
the simplest sense a high value of K in
a k-fold cross-validation leads to a
model where leads to an assessment I
should say which is of very low bias but
of high variance okay and a very low key
so okay of like two or three years
so essentially could have low variance
but a higher bias and one extension of
for instance the the cross-validation
where you have a very high key right
where you like need to have an unbiased
assessment but an assessment with lots
of variance is the case of the leave one
out cross validation the idea here is
that if you have n data points I'm going
to use n minus 1 data points to do the
training and I'm going to validate on
that one data point that I left out and
what I'll do is I'll keep just like in
k-fold cross-validation how you change
the training training sets and the
validation sets how you shuffle them how
you permutate them the same way you're
going to leave one data point out for
validation train on the others and then
predict on this and then keep doing that
iteratively so obviously with their when
if if it's essentially like k is equal
to n then the cross-validation is almost
completely unbiased but can have high
variance because the n training sets are
so similar to one another right and some
an approach like that is also
computationally fairly cumbersome but
but if you have that approach then then
the leave one out cross validation is
also something that you might consider
so I hope that gives you an idea of
cross validation and more broadly the
use of you know the idea behind
validation as a means of model selection
and this whole thing of breaking up
training for creating models validation
for assessing and kind of you know
selecting models and tests for pure
assessment but no no further selection
and decision
thank you