18 Automated Model Selection and Optimization Video Lecture | Rapid Miner: Learn and Master - Data & Analytics

Hello and welcome to this RapidMiner Tutorial.
In this video, we will show you how to fully automate your model-selection and parameter optimization.
We will start with the process we created in the parameter optimization video.
As always - if this is the first video from our series which you are watching,
then you can download the process and the data used here form our website.
The video on 'Data loading via a process' will show you how to import the data and the processes.
Now back to our process here.
In this process, we are optimizing the parameters of a 'decision tree' algorithm, but during the tutorial on
'Model Selection' we saw that also 'Random Forest' and 'Rule Induction' are performing well on this example-set.
If we were to optimize the parameters for either, then one of them
might actually perform better than our decision tree.
To try that out we could now swap out the 'decision tree' here in our cross validation manually
but that would not be elegant as we would need to repeat this and
reconfigure the optimization each time for any of the algorithms we are testing.
So, a much smarter approach would be to leave this one as is and use copies
to create one 'parameter optimization' with a 'Random Forest' and a 'Rule Induction'.
Through the magic of video editing we will jump right to the end of this copy and pasting.
As you can see we removed the 'log' operators here and have also updated the parameters
which we will use to optimize.
For 'Random Forest' for example we chose to iterate over the number of trees.
For both the 'Random Forest' and the 'Rule Induction' we chose only one parameter to be changed.
We specified only 10 steps for each, as we want to keep the time for our demo process to run very short.
In your real case, you should consider doing a wider and finer selection,
but keep in mind that the process may take some time to finish.
By the way since version 7.3 we have started to introduce parallel execution for some operators,
which gives you a lot of boost as it uses multiple cores on your computer at the same time.
Before we move on let's just briefly save our process here in the repository.
I'll just give it a new name and…all set.
OK now, if we want to perform all three optimizations on our data we could achieve
that by using the 'multiply' operator.
But then we would need to look up the performance for comparison by hand, which is again not very elegant -
after all it is our goal to find the best model after optimization in an automated fashion.
To automatically return the model with the best results we will use yet another optimization.
As varying parameter, it will have to iterate over the three parameter optimizations
which we can execute using the helper operator 'Select Subprocess'.
Ok let's do this.
We drag in the 'Select Subprocess' operator from our 'Operators' panel into the 'process' panel.
Then we cut the 'parameter optimizations' and then we paste them into the subsections
using for example the right mouse button.
Since we have three optimizations to trigger let's add one more 'selection' window.
Next step is to connect the operators.
And now we can embed this nicely into another 'grid optimization' operator which will then return
the highest achieved performance of either of the embedded parameter optimizations.
To make sure we are not only getting the 'performance vector' but also the info on
which model was selected we can also connect the 'parameters' port, here, to the 'results'.
Of course, before we now run this automated model selection, we need to specify what to optimize.
As mentioned before the varying parameter is this time a sub-process and not a parameter per se.
Therefore, it is only '1' to '3' here as we have three optimizations as sub-processes,
and we will do those in three steps.
All right now as I select 'run process' you can take a look at the lower left here
to see how the optimization iterations are performed.
Good - the process is done and the 'results'-view is displayed.
The best performance achieved by one of our three, parameter optimized models is roughly 85%.
And the model with the best result was in the sub-process 1 which is the 'decision tree'.
If we look at the ROC curve of the performance-vector, then we see that it is really nicely getting close
to the upper left corner which is where the perfect model would be.
Now if this is the first tutorial you are watching in our series then we recommend to have a look at
the 'model selection' video to learn more about the ROC curve and its interpretation.
For those of you who have already watched it you may remember that we did mention in there
that we were only looking at unoptimized ROC curves.
So as last part of this tutorial we want to show you a way how you can change that.
Alright let's go back to the 'design' view.
The current setup only outputs the information which is the best performing sub-process but what
we are looking for are the optimized parameters for each model so we can plot ROC curves for comparison.
We already generate the parameters for each of the individual optimizations with our current process,
we will just need to make RapidMiner remember them for later re-purposing.
And so 'remember' is exactly the keyword we can use to find our next operator.
Now let's drag and drop it three times and give each of them a name -
I'll use abbreviations here to keep it short and sweet.
We also need to make sure we select 'Parameter-Set' as the definition of what we want to remember.
With that, we are ready to go back to the main level of our process, where we reactivate the 'multiply'
and the 'Compare ROCs' operator which were sitting there now for a while.
First, we connect them and then we take a quick look if the sequence of execution is right.
Yup - OK all good.
Now we have asked RapidMiner to remember all the optimal parameter settings, hence it is not a surprise
that the next operator to use is the 'recall' operator while we execute our 'compare ROC' operator.
We'll remove the models which we don't need any more and then drag and drop in a 'Recall' operator.
The name we use to configure it needs to be exactly the same as the one we used
for labeling 'remember' parameter before.
It was 'DT' for 'Decision Tree' which we used and we want to recall a 'Parameter-Set'.
And in order to apply this to our 'decision tree' model here, we will finally use the operator 'Set parameters'.
We need to establish a link by using the right name of the specific instance of the operator
so we'll change that for easier understanding.
And now let's paste that it into the settings of our 'Set Parameters' operator.
Again, we need to check if the execution is happening in the right sequence - oops, here we need
to bring 'recall' and 'set parameter' to the front so they are available before the Decision tree is executed.
Now I'll repeat all the same steps for the other two machine learning algorithms
but I am not going to make you watch all this so let's fast forward.
As you see we are all set, ready to run this process.
…and finally, after a few seconds we get our 'ROC Comparison' graph showing
the curves for our optimized models.
It shows that the "Decision Tree" is the best model, which we already knew from our previous run
but now we can see that 'Rule Induction' and 'Random Forest' are a close second
depending on the threshold setting for the classification.
As mentioned, please refer to the 'model selection' tutorial to learn more about
the interpretation of the ROC curves.
With this we conclude our video on more advanced optimization, in which we showed you also
how to select sub-processes, and how to remember and recall certain elements in a RapidMiner process.
We thank you very much for watching!