Weka Tutorial 24: Model Comparison (Model Evaluation) Video Lecture | Weka Tutorial - Data & Analytics

all YouTube in this workout tutorial is
very important I'm going to demonstrate
how you'll be able to compare the
performances of the classifiers
generated by multiple numbers of
learning algorithms from a single data
set or from multiple data sets now to
begin with you should realize that when
I'm talking about evaluating the
performances of multiple classifiers
that means that we're not able to use
the Explorer because in Explorer you
want you're not allowed to use more than
one classifier at a time so to
demonstrate the idea I'm going to click
on the experimenter if you don't know
how to use experimenter then you can go
back to two 1212 where I try to describe
the buttons the functionalities the
advantages and disadvantages of the
environment in that tutorial very
briefly so to begin with I'm clicking on
new because I'm trying to do a new
experiment here and and then I'm going
to click the add new on the data set
options and I'm going to go to the data
sections of wake up installation folder
and for this particular tutorial I'm
going to select the segment challenge
ARF so that's our data set we are going
to play with that particular data set we
are not using multiple data sets but
remember that in experimental
environment you'll be able to play with
multiple data sets at a time for
algorithms we're going to the algorithm
sections and we're using three
algorithms for this tutorial I'm
clicking add new my first algorithm is
going to be an eye base algorithm it
will generate a classifier from the
segment challenge our file so that's our
first learning algorithm our second
algorithm let's say we're going to use
and SVM algorithm it can be found under
functions and you can select leave SVM
and there are lots of options for SVM's
but we are going to stick with the
default ones so we click ok so that's
our second learning algorithm for the
tutorial or tarde algorithm is going to
be for example it's going to be a tree
one and my favorite for from the tree
one is a random forest it really worked
well for me for several occasions ok so
here we have the three algorithms these
three algorithms will generate three
classifiers from the features of segment
challenge our file so this is going to
be a classification test so this radio
button is on the number of false will be
10 but here I'm not going to use 10
times 10-fold cross-validation so I'm
taking it back to 1 so as we are done
here we go to the run tab and there is a
button called start and we click on
start
so the experiment has been started at
10:14 a.m. you can see it on the log and
on the bottom bottom here you can see
that now currently as VM is walking on
the our file an eye base has already
finished it's working on the our file
and you'll see now it has been changed
to random forest and it's done because
up to 21 seconds you can see that it's
finished and there were no errors so to
start the analysis of the performance
evaluation by the three classifiers you
go to the analyze tab and there is a
button called experiment we click on it
everything is set as default and you
have to you have to be very careful with
configured test option especially with
the testing width so there are two
options here first one is paired t-test
err corrected parity tester which is not
corrected I I believe that Wicca has
this one as the latest so this is set as
default and what is apparent
tester you have to go to your statistics
101 for research where you will be able
to know how you can do a significance
test so paired t-test is a kind of
significance test on row I'm going to
set the data set so this is how I'm
going to format the test output here to
be shown here my row is going to be
displayed as the data sets the columns
are scheme scheme option scheme version
ID I don't need the set the last tooth I
just need the scheme names which is the
name of the algorithms and I click on
select so there are many evaluation
measures you can find in the comparison
Phillip how are you going to compare the
evaluate performances of the classifiers
well it's it's default set default set
to default percent correct or accuracy
but I'm more interested in finding the F
measure or f1 score the significance
here is default to point zero five this
means that any difference found among
the classifiers generated by these three
algorithms will be within ninety-five
percent confidence interval so if you
don't know what is called a confidence
interval
what is 95 percent what is alpha equals
two point zero five alpha or
significance equals two point zero five
was a paired t-test then you have to
really go back to your statistics one
one for research where you'll be able to
find out everything about significance
test but this is pretty much standard
nowadays to report the statistical
significance of the difference between
difference among the classifier
performances the rest are just the
defaults here I'm not going to play with
that and as we are all set we click on
perform test you can see immediately
that on test output you have numbers of
things here the tester is the paired
corrector tester and we analyzing F
measure not the accuracy we have one
data sets and three result sets because
we use three different learning
algorithms
and the confidence interval is 95% or
the Alpha is equal to point zero five
which is two-tailed and now we are going
to analyze the results here on the
result you can see that this is our data
sets called data set called segments our
first algorithm is nine base so that's
that sits at number one because that was
chosen you can you can go back to the
first step to setup you can see that
naive Bayes was chosen first so that
means this is our base algorithm we are
trying to compare the other tools the
live SVM and the random forests we are
trying to evaluate those performances
against these a Bayes classifier
produced by nine based algorithm so the
night base has called 84 percent F score
okay and the leave SVM scored 68 percent
F score where the random forest has 100
percent F score so now we have to say
which one is better than other well with
naked eye we can see that yes 68 percent
is much more lower than 84 percent and
100 percent is much more better than 84
percent but there there will be some
cases where this is very marginal this
can be point eight four and leave as VM
for say say for example performing point
eight one then is that a significant
result significant difference between
the night base point eight four and
leave as VM point eight one so in order
to see that in order to report that you
have to you have to report the
statistical significance at point zero
five so in Wicker what for this case you
not you are not able to see the
significance value how significant a
naive Bayes classifier significant I
pacify performed then leave SVM and how
it was poor than the random forest
you're not you're not able to tell that
numerically quantitatively but there are
some symbols here
symbol is called V that means that means
a victory and other symbol is called
asterisk that cell loss and the middle
one is a blank one that means I cannot
tell whether it's statistically
significant or not so here our nine
based performed 84 percent F score and
leave SVM has 68 percent F score and we
have an asterisk right beside that leave
as VM score F score that means that
leave SVM performed 68 percent compared
to 84 percent of naive Bayes and that is
statistically significant difference and
this is poorer result whereas if you
compare the night bass performance with
random forest performance 84 percent and
hundred percent respectively then you
can see a V sign that means the tree or
random forest has performed
significantly better than the night
based classifier so if you if you for
example if we had 84 84 percent for nine
days and 81 percent but without anything
here neither an asterisk nor of V that
means that we cannot say that this 85%
is better than 81 percent or we cannot
say in other way that this 81% is poorer
than these eighty four percent so in
order to report the comparison with the
performance of performances of several
classifiers here at any other symbols
asterisk means significantly poor and
the V means significantly better and
it's a blank that means we cannot tell
whether it is significantly better or
poor I think that helps you a lot just
one thing you have to remember that we
have just one data set here but we could
have multiple numbers of data sets there
and the results you needed to interpret
accordingly because now you are having
two different data sets on and you are
applying three different algorithms so
the result set here
look a bit differently but the
interpretation our interpretation is
going to be the same I hope that helps
and I promise that I'll be back this
time sooner thank you