Getting Started with Orange 07 Model Evaluation and Scoring Video Lecture

In the previous tutorial we've made some
predictions on fruits and vegetables,
but you've been probably wondering how do we know which classification model works best?
Today I'll show you how to assess
the quality of various prediction methods.
This time we're going back to
our good old iris dataset.
First, we will train a model with one of the simplest classification techniques -
a logistic regression.
To avoid overfitting, we are
first building the model on the training data
and then testing it on a separate
test data.
We want to repeat this procedure multiple times
and then report on average accuracy.
This is what cross validation does
and it can be found in the Test and Score widget.
In our case cross validation splits the
data into ten subsets,
uses nine for training the model,
and the remaining subset for testing it.
It repeats this procedure nine more times,
each time using a different subset for testing.
Cross validation results are reported on our right.
What do these numbers mean?
The simplest one to understand is
classification accuracy in the second column.
It reports on the proportion of
correctly classified data instances.
Looks like classification accuracy was
96 percent.
So what are the four percent of the data that were misclassified?
We can check this with confusion matrix.
Looks like our model had no problems
with classifying iris setosas.
Still, it got a bit confused with versicolors and virginicas.
We can easily observe misclassifications by selecting misclassified instances
and sending them to the data table.
Just by looking at this table it is
still hard to interpret the results
and say much about misclassifications.
Instead, let's visualize them in scatter plot.
Most misclassifications are at the
border area between the two classes -
the red for iris versicolor and the green for
iris virginica.
We expected something like this, right?
Logistic regression is of course not the only
classification method we can apply on iris.
We could use, say, random forest.
This is a more complex classification
technique,
but often has better accuracy.
However not in this case,
where the accuracy of logistic regression
is already quite high.
Besides accuracy, Test and Score widget
reports on for other scores.
We really like the area under receiver operating characteristics or AUC in short.
But that will be the topic of some other tutorial.
We can also check where misclassifications lie for random forest.
But I'll leave this to you.
Today we've learned how to use cross validation in Orange
and how to compare several classifiers on a single data set.
I've also showed you how to select misclassifications and visualize them.