Weka Tutorial 31: Document Classification 1 (Application) Video Lecture | Weka Tutorial - Data & Analytics

hi and welcome to another wake a
tutorial this tutorial and the next
tutorial these two tutorials are going
to be very interesting because in these
two tutorials I'm going to talk about
how you can classify documents so these
tutorials are important because you know
that classification of documents are is
going to be a very prominent field in
machine learning and data mining so we
can have some facilities that you can
use to easily classify various types of
documents so in this tutorial I'm going
to first show you what filter you can
use for doing document classification
and the next tutorial in the next
tutorial I'm going to demonstrate how
exactly you can classify various
documents so to start with we're going
to wake Explorer and we are going to
open a file I have already created this
file and I'm going to show you this file
after opening this so I open that file
right so what's in that file if I go to
that particular file and I can then I'm
showing you here that the relation name
is train and there are just two
attributes the first one is called
document of type string and the second
one is that tribute call class it's
either yes or no these are the two
binary classes that we're going to
assign these documents so here you can
see that we are having a we are having
something in in quotation marks the
first one is the price of crude oil has
increased significantly now for example
this is just one single sentence but it
can have multiple numbers of sentence
within within the quote marks there so
this is an actual year text document the
amount of text you have in your document
you can put all of them into into these
into these two quote marks and they will
represent the entire
next of your document but for the
simplicity sake of this tutorial I'm
just assuming that my document has just
one single sentence that is called the
price of crude oil has increased
significantly and the class of the
document is yes so what's what what are
these years and know about these two
classes are representing whether the
texts in my document are actually
pointing out that the document is
talking about crude oils so here you can
see for the price of the crude oil has
increased significantly that if this is
the text present in my document then
it's really talking about crude oils on
the other hand for the second document
again I'm telling you okay the second
document can have hundreds and thousands
of sentences and they all can be put in
these two quote marks is nothing problem
here but for simplicity sake I just I
just assume that my document is having
just one sentence that is demand for
crude oil outstrips supply so these
document the text in this document is
also talking about crude oil so that's
classified as positive or yes
whereas might our document has a
sentence called some people do not like
the flavor of olive oil so this is not
actual these the text in this document
is not actually talking about the crude
oils or petroleum so it's actually
talking about the cooking oil which is
olive oil so these document is
classified as know as the text in the
document is not actually conveying the
information that the document is talking
about crude oil that's not true it's
actually talking about cooking oil so in
this way I have six examples in my
training file and this is the file that
I've just start up loaded to that that I
just are loaded in my way cat explorer
so we are now going back to wick X
door here we have a very nice filter is
called it's kind of unsupervised filter
so you can go to filter unsupervised
then it's an attribute filter you go
down and you can see that we have a
filter called string to word vector so
if we choose that what exactly it is
doing is we have the strings there for
example in our train file we have all
these strings so this is just one
symptoms you can have thousands and
millions of sentences within quote marks
if your document really has that volume
of text so what string to a word vector
is doing here is actually uh it is
converting all those string elements
present in my train file here is the
relation name and in the six instances
you know that we had six documents there
with two attributes two attributes one
is document and the other one is class
so all of those strings contain words
right now stringtowordvector if you do
not if you do not modify any of these
arguments here just you just using other
default arguments then what
stringtowordvector does is it is
actually creating a vector of words that
are all contained in those six strings
so you know that there are several
mechanisms you can use a java code you
can use a Python code to generate a word
vector from a group of strings but for
Waker it's already written there so you
don't have to worry you don't have to be
worried how are you going how you are
going to produce a word vector from
strings because you know that term
frequency inverse document frequency
these are very vital information or
attributes required for various machine
learning and data mining jobs so you
don't have to be worried about that is
wake up will do it for you so if we
choose
the stringtowordvector and keep the
default settings there and click on
apply you can see that it's interesting
soot to see that our class attribute is
a now situated as our number one
attribute we're in the in the in the
main file if we go back here in the
train file you can see that our class
attribute was the last attribute but up
to after applying that filter that
unsupervised filter our class attribute
is now situated or positioned at the
first as the first attribute of our
attribute list and previously we just
had two attributes here the document and
class attributes but now we are having
35 attributes what does it mean it means
that as you apply the string toward
victor filter on the entire training
file training our file its then it then
creates a vector of words found from all
the strings present in your AR file so
you can see that we're having crude
demand duh and again a small is not
capitalized si is not capitalized crude
for has in increased and so on so these
are all the 3435 minus one so we have
one class attribute 34 words that are
uniquely present there in your our file
all of those six strings that you had in
your train dot our file so if you click
on any one of those attributes we can
see that the minimum value is zero
maximum is one you have the mean and
standard deviation and so on so you can
have information for these attributes
for all of these attributes okay so this
is interesting because just by applying
an unsupervised filter what you have
done is you have broken down your
strings into word vectors and what
vectors are really important so now we
can all if we want to develop a model or
build a model using any classification
algorithm
then this class attribute needs to go to
the end of your attribute list so how
are you going to do that I showed it in
one of my tutorials how to take the
class attribute to the end so this is
how it is done you click on edit and you
can see that your head you have the
class the nominal type of attribute
called class situated at the very
beginning of your train dot our file now
after applying the filter so you can
right click on that class attribute and
you can select attribute s class so now
it goes to the end of your attribute
list so you can see that out we are
having some blue and red going on here
to denote the yes and no portions and on
the class activity you can see that we
had three SS and three notes okay
so in this way you are you have just
taken the class aptitude from the
beginning of your attribute list to the
end of your attribute list now using
this word victor of presentation of your
train dot our file you can always
generate a generator model using
different kinds of algorithms is the
procedure is the same so you go to
classify tab you are for example
choosing tree types three tight
algorithm called j48
and we're not going to do any
cross-validation here because we are
just we're just having a very few amount
of training data at six so we click on
use training set and we click on start
so this is the model that are created
with j48
using the word victor representation of
your trained on our file ok so if you
want to apply this model to any other
test set then what you can do is you can
you can select right click on on the
model the end the results a result list
and you can set the model at any place
for example we are going to save it as
word victor
Oh model okay and we click on save so in
this way you just saved your train dot R
file which is actually presenting the
document texts the texts in the document
you are having and a class of those
documents whether they are talking about
crude oil or not you have just saved a
model of these trained model of a
classifier of on these train dot R file
but you didn't use actually the string
attribute or the document attribute here
what you have just done is you have
created a word vector from all the
strings present in your AR file I hope
that helps and I'll be back with more
document classification demonstration in
the next video thank you very much