14 Testing a Model Video Lecture | Rapid Miner: Learn and Master - Data & Analytics

Hello and welcome back.
In other videos we show you how to load data, create a model and how to apply a model.
In this video we are addressing the topic of training and then testing a model
so you can find out how accurate your predictions will likely be.
If you want to follow along you can download the data file and the processes which we are using,
and import them.
The 'Data loading process'-video will show you how this is done.
So let’s get started.
First we are going to reuse one of the processes which we created in another video.
We are going to our 'Processes' folder and we will select '02 build decision tree'
which is from the 'Creating a model' video.
So let’s double-click on it to load it up and let us take a look at this process.
First we load in some customer data, and in that customer data we have examples that have either
a 'churn' or a 'loyal' label but there are also some with 'missing values'.
Since we want to first build a model we only want to use examples with labels.
We need to filter out the data with missing values before we build our 'decision tree'.
So what this process does is, it creates essentially something called a 'training'-dataset.
In the 'applying the model'-video we show you how to 'score' the data, which means we actually take
the examples without labels and apply our model, to predict our new customers' loyalty
- putting or segmenting those unknown customers into the 'loyal' or 'churn' buckets.
But the issue with that approach is that we can’t really assess the performance of our prediction,
as we don't have any information as to if our predictions are actually right or wrong.
So in order to test our performance, why don't we build and train our model first on our example set
- with known labels - and then we can try to predict those same known labels, with our model.
So let’s go ahead and get an operator called 'apply model'.
We need to connect the 'decision tree'-MOD-port - stands for model - to the 'apply model'
operator's input-MOD-port.
Ok - excellent.
Now the second input port is called UNL - which stands for unlabeled data -
so it is requesting the data on which we want to apply out prediction.
In our case as mentioned this is going to be the same examples set which we used to build the model.
So, let's drag this 'example' port over to here to the 'unlabeled' port.
And like in the 'applying the model' video we will get a little icon to multiply the data.
This will allow us to use the example-set for training and testing, so let's click on that.
Excellent and now let's connect the labeled output to the results.
So what did we just do here? Let's take a moment to recap.
We loaded in the data, we filtered out the unlabeled examples, that don't have 'loyal' or 'churn'
and thereby we created a training dataset.
We fed the training-data into the 'decision tree' and built the model, but we kept a copy of it to drive it down
to this branch here, where we took that same example-set and 'applied the model'
to it to create predicted labels so we can finally compare them both.
Alright then let's see how our model does and let's hit the 'Run Process' button.
So now let's take a look at the results and compare the label columns.
'loyal', 'loyal' - good the prediction was correct!
churn','loyal' - hmmmm the prediction was not correct here.
loyal', 'loyal' - prediction was correct.
‘loyal', 'loyal' - prediction was correct .
Now to get an overview it would be best to compare all the previous existing, with the predicted labels
and add them all up, and from there you could get a percentage, on how accurate your model is.
Well you can do this by hand or you can let the computer and RapidMiner do that and get the insight much faster.
The key is to measure the performance and as you have certainly already guessed there is an operator for that.
So we will now switch back to our 'design' view.
When you search for performance you will see a whole list but we will use the simple 'performance' operator
and drag it to the very end of our process.
Before we run our updated process we might want to take a moment to save it.
I will call it '04 testing the model'.
Now let's hit F11 to re-run our process and we see that our accuracy is 85% - not bad - not bad at all!
But wait a minute - is this really a reliable measure for the accuracy of our future predictions?
If you think about it - what we just did is, we've provided 100 questions to a student,
and then tested them on their answers.
But did we really assess our student - our model - on the right challenge?
No - the only thing we tested now is, how well can this student reproduce answers to known questions.
We just looked at the so called 'training error'.
But what we really want to know is the 'prediction error', which describes, how well will the student
- or our model - do with unknown questions or an unseen data set.
Well - we will show you how you can achieve that in the 'validation' video.
Thank you very much for watching.