15 Validating a Model Video Lecture | Rapid Miner: Learn and Master - Data & Analytics

In the video on 'testing your model' we showed you a simple way to use the same example-set for training
and testing, in order to measure the performance of the model.
But there we tested the model on the same data on which we trained it,
which is much like giving a student all questions of a test before testing them.
Measuring the performance of that student this way is not going to help us, if we want to know
how well they will do when they are asked to answer questions they haven’t seen before.
After all, that is what we are really interested in!
What we could have done instead is to give that student 80 example questions of our 100 available ones
and then also test them on 20 new questions which they have not seen before.
Or even better, if we want to make sure that our selection of questions has
no effect on their performance, we should do this multiple times.
Of course using different students would mean changing the boundary conditions and hence
we would need to reset our single unfortunate student’s memory before each run of the test.
The two methods described are called split-validation and cross-validation
and we will talk about the latter one – cross-validation - in this video.
On the screen I have the process '04 testing the model' open and displayed.
If you want to follow along you can download the data file and processes which we are using, and import them.
The 'Data loading process'-video will show you how it is done.
Now let's go ahead and get started on the cross-validation.
First let's get rid of everything after the 'filter examples' operator by dragging
a rectangle over them and then just hit 'delete' on your keyboard.
Then let's come over to our operator panel and search for 'cross-validation'.
So let's drag that over here. And let's connect the ports.
Before we dive into what we need to do with this 'validation' operator,
let's talk a little bit about what 'cross-validation' does.
If I look at my parameter panel over here, I have something called number of validations,
the default is '10'. So what does that mean?
Well when I have the default of '10' it will take my data-set and break it into 10 smaller subsets of data, and
then it will go ahead and build a model on 9 of those, keeping one of them for testing and then it will iterate.
Sounds confusing doesn't it - well let’s take a look at the image here
which will help us to clear up some of the confusion.
In this particular case we have a dataset that has '5' cross validations.
So you can see here '1', '2', '3', '4', '5' iterations and so what the cross validation does, is, it breaks the dataset
into five subsets - trains and builds the model on four and measures the performance on the fifth.
Then it iterates it selects another 4 - different ones - and selects another separate test-dataset and then builds
the model, measures the performance and finally begins to average that.
So it'll do that over the entire five or ten or however many validations you set
and give you an average performance of the models.
So let's switch back. Can you see the little blue hierarchy symbol?
Whenever you see that on any RapidMiner operator it means that there's something
going on underneath the hood, it's a sub-process typically.
So double-click on that.
Ok, now we actually have two new windows.
Do you remember how we talked about 'training' and 'testing' our student on different subsets of questions -
in the sub-process of this 'validation'-operator I have a section for 'training' and I have a section for 'testing'.
So the left window represents what we have in our 'testing the model' video in the upper branch,
you have also seen this brief when we started this video.
Since this is the training area we clearly need to put our 'decision tree' model in here.
I'm going to pass my data, my 'training data' to this 'decision tree' and build a model from it.
So I´ll connect the 'MOD' to the 'MOD' port.
On the right hand side we will do our testing.
And for that we need to apply our model to our 'testing' subset and measure the performance.
Let's pull in our apply model operator and also the 'performance' operator.
So before we make any of these connections let's talk a little bit about them.
Here in the 'training' window we create 'the model'.
Since we want to apply it we need to feed the information about the model
into our 'apply model'-operator.
Now we need the 'unlabeled', 'test' data to which we want to apply it.
In our '04 testing the model' process this came initially from the 'multiply'-operator.
So where does it come from in a cross validation?
The copying or example selection of the test and training subsets is done automatically
and represented through the two windows.
And we can get the test example-set here from the TES port, so let's connect that and then send our
- now labeled examples - to the 'performance' operator.
And finally, since we are doing multiple iterations in a cross validation the PER port of our Performance
will go to the PER port to build the average of the different model's performances.
Great, now we are almost ready to run the model.
Let's go up one level, outside of the sub process, come back here to the 'validation' operator
and connect the PER port to the results.
But before we run the process why don't we save it first.
So let's go to our repository to the left.
Highlight process - right-click - store process here - and let's call it '05 cross-validation'.
Now we can click on the 'Run Process' button.
Ok - these are the results - well hmmm.
What you see here is a 'confusion matrix', so no wonder we are a bit confused - ok that was a cheap shot!
But once you know how to read it, then it's quite simple.
Let's start over here with the 'accuracy'.
The overall accuracy of all predictions achieved by the 10 models - each of which made a prediction of
90 examples, after being trained on the other roughly 800 - is, nearly 84%.
That is the percentage of correct predictions.
The 'plus', 'minus', is one 'standard deviation' which is calculated from our 10 individual model accuracies.
The lower this number - this is a critical, important item to note when you're evaluating your model
- the smaller the one standard deviation value is, the more stable is your model.
A stable model - given that it has a high enough accuracy - is good as it will give you a smaller range
of best and worst cases which you have to consider for the quality of you predictions.
Now let's take a closer look at the table.
‘Pred. loyal' stands for 'predicted loyal', 'pred. churn' stand for 'predicted churn'.
Now if you look down the columns of the table, you can see what labels we had in our example set,
and then if they were predicted right or wrong.
So we had a total of 509 plus 69 'loyal' candidates, ok?
And when we ran the algorithms through, they predicted 'loyal' 509 time, which were actual ‘true loyals’.
And 69 times it predicted 'churn', when in fact it should have been a loyal.
In total we were able to capture 88% of our 'loyal' cases.
For the 'churn' cases we achieved 76%.
A really handy way to look at is, to look at the diagonal which shows the 'true loyals' and the 'true churns'
- that were predicted correctly.
And now you can see how the overall 'accuracy', which we first talked about, is calculated.
It is the total of correct predictions - 509 plus 246 - given as a percentage of all 900 predictions.
The last thing to look at is the 'class precision', which shows the percent of predictions that 'were correct'
for each predicted label - 'loyal' or 'churn'.
So if you want to calculate that for our 'Churners', you simply add 69+246
and then divide 246 by that sum, to get 0.78 or 78%.
Now if we interpret the table with focus on our 'churners', then we can say that,
we are currently managing to predict 76 out of 100 true churners.
On the other hand in 22 cases out of 100 churn predictions we label someone
as a churner who is in fact a loyal customer.
And that's it - that's how you can effectively validate the model and see how good it is,
before you take next steps to improve it or decide to put it into production and deploy it.
Thank you very much for watching.