17 Optimization of the Model Parameters Video Lecture | Rapid Miner: Learn and Master - Data & Analytics

Hello and welcome to this RapidMiner Tutorial.
In the video on 'model selection' we left off with a cliff-hanger about optimization.
Now we don't want to leave you hanging any longer, so here we will show you how
'Parameter optimization' can be easily achieved in RapidMiner.
So, let's get started.
We will repurpose the process created in the 'model selection' tutorial which you can see displayed here.
In case this is the first video from our series which you are watching,
then make sure to download the process and the data used here from our website.
In our 'Data loading via a process' video you can see how to import the processes and the data.
Now back to 'Parameter optimization' and our process here.
In the cross validation, we have a 'decision tree' model for which we want to find the optimal parameter settings.
We will optimize our accuracy by identifying the best combination of the 'criterion'
to be used for splitting of the attributes and the 'minimal gain'.
The 'criterion' selection defines the criterion by which attributes will be selected for splitting and
the 'minimal gain' governs the threshold which we are requesting in order to allow the next split.
The optimization is done via one of RapidMiner's 'optimize parameters' operators.
We will use the grid version.
Before we do that let's first save the process in our repository as '07 Parameter Optimization'.
And while we are doing house-keeping let's disable the "Compare ROCs"
and the "multiply" operators since we won't need them here.
Now let's just cut the cross validation out, connect the optimization instead.
Then paste the 'cross validation' as a whole into the 'optimize parameters' operator.
After connecting the in- and output of the 'cross validation' operator
we are now ready to define the parameters which we want to optimize.
All right so let's select the 'Edit parameter settings' Wizard.
As mentioned we want to optimize the model's parameters, so we select the decision tree first.
Second step is to select our first parameter, the 'criterion', which then populates the rest of the dialog.
We can leave all the defaults as we just want to loop through all the criteria available.
We see information displayed that we are now creating 4 combinations,
which is intuitive given the number of criteria chosen.
The second parameter which we want to select for optimization is the 'minimal gain'.
This parameter controls how soon splits are being made, so the lower the threshold value is,
the larger the tree grows.
Now that the dialog has populated further we can define how we vary the criterion.
To find out if 0.1, which is the default, is a good choice we will just test with
a higher and lower value through 100 steps.
We have configured 404 parameter combinations, and given that we are having a 10-fold
cross validation in each loop, it means we are really producing over 4000 decision trees.
As you can imagine this can become pretty computation and runtime intensive
so you want to be careful about selecting the number of parameter and the granularity of your optimization.
Now we can close this dialog, but before we start the process, which will deliver us the optimized performance
and the parameters combination used for that, we will add a 'log' operator so we can take a bit of a closer look
afterwards about what happened during the optimization.
We could specify a file name to create a file-dump of the log but we just want to have
a quick look at the log so let's move on to edit the list of content for logging.
We certainly want to see the parameters we are optimizing for so let's add 'Gain' and 'Criterion' first.
Be careful to select the right instance of the 'Decision Tree' here - the one with the number '2'
in brackets is the one in our currently disable 'Compare ROCs' operator.
Further let's add an iteration counter and of course we want to know what
the performance of each loop was, so let's add that as well.
For both of the latter two list items be very careful to retrieve the information of the 'Cross validation'
and not from the model or even the performance operator because if you do that you are logging values
inside the cross validation which are generated using the subsets in our 10-fold cross validation.
OK, so we apply a count to the cross validation and then we log the performance value of the Cross validation.
Let me repeat this: don't log the performance of the 'performance' operator as that will only output
the value of the last fold of each iteration going through the cross validation but you are usually
only interested in the average performance over all folds of the cross validation.
Ok now let's hit 'F11' to run this.
Nice, our performance is roughly 85% with our optimized parameters as shown here.
As you can see the minimal gain is close to the default of the decision tree which was set to 0.1
but the criterion 'accuracy' seems to yield a higher accuracy.
To find out more let's take a look at the log.
We can see that iterations were performed for each combination of 'Gain' and 'Criterion'
and the log shows the performance for each.
If we briefly create a chart displaying the parameters we varied and the performance for the
different combinations, then we see that it is not useful to define a "minimal gain" parameter higher than 0.3.
And, if we zoom in to the area with the highest performance, then we see that
the 'accuracy' was clearly the best split criterion to use.
Alright, this concludes our tutorial on 'Parameter Optimization' make sure to check out also the model
selection module and the one on automation which shows you an optimization using more than one model.
Thank you for watching.