Weka Tutorial 25: Sparse Data (Data Preprocessing) Video Lecture | Weka Tutorial - Data & Analytics

hello youtube and welcome to the 25th
waked tutorial in this tutorial I'm
going to demonstrate how you will be
able to format sparse data so that
you'll be able to feed to wake up in ARF
file format to begin with let's see what
is called sparse data so I have made up
a file myself to demonstrate this
tutorial that's called sparse data
sparse dot data so here you can see the
relation name is Bart data and I have
ten real real at real valued attributes
and one class attribute that spam and
ham
so the real valid attributes are
representing here the function words
various function words for example da in
of which etc what my data set represents
here this is a very tiny data set for
the tutorial but what it is representing
here is the number of function words
present in some documents so we have two
instances here so that so that we can
say that our data set is containing two
documents and what we are trying to find
out is the number of occurrences of each
of these ten function words in those two
documents so this is a sparse data set
why because you can see that we are
having lots of zeros in our data sets
because it is not necessary that a
document should have gah inside of it so
it's zero but the first document has one
in as we are having one at the position
two for the second attribute and also we
are having some values here three but
the other function words are zeros
similar to the first one the second one
also represents a sparse nature where
you can see lots of zeros just one for
which is which means that our second
document has which appearing four times
in it but the other function
are absent in that document and it's for
just to play with the spams and hams out
the labels of the documents so this is a
sparse data so from this example you can
see that the sparse data means that when
your data is containing lots of similar
values in this case which is zero to
represent this in Wicca or to represent
is in our file format so that you can
treat that file to wake up because these
values here these are quite good-looking
so far but when you're having we are
just having ten attributes a real-valued
attributes but when you're having
millions of attributes for example in
text mining you can see millions of
attributes can be extracted from the
documents so in that case you'll have
for example ninety nine thousand zeros
and the rest are filled with values for
function Wars frequency but to see such
data set it's very cumbersome one this
is not exactly what you're trying to
look at but what I will be able to read
that because if we go to the Weka
Explorer here we are just opening that
file particular file we are going to my
drive where I have saved the file so
let's go to Program Files and we're
going down to wake a three six data and
sparse example so you can see that my
data set can be fed to wake up very
easily but this is not exactly what
we're what we are expecting to appear on
the screen here because we are having
lots of zeros just a few a couple of
values real values real real values
there so in that case to represent this
in a different way which also will be
able to be fed to Waker we have some
different ways so to write down that
we need to see you need we need to go to
the data section you can see that to do
that how many zeros are before our first
real value that is one so that's just
one zero so you see you can write down
one and then input one you put a comma
then you can see our second real value
there is three so how many values are
proceeding that three so that's one two
three four five and six so we write down
a six there space three so this means
this eventually means that we have a one
but one more digit digit is proceeding
that one so we have a three and six more
digits are proceeding that three and now
our next value is BAM which is the class
label but how many values are before us
our proceedings BAM so that's one two
three four five six seven eight nine and
ten
so we write down 10 space spam okay
so for the second instance the second
document in our in our data set we have
three zeros preceding a four so that
means we are going to write down three
space for then we put a comma and the
rest of the values are zeros then we
have the class label ham for the second
document so we are having ten in ten
attributes attribute values there and
then we are having our second value
which is not zero for that particular
instance so we write down a ten space
ham and then we can ignore these two
entries there because there they are
similar so these two are representing
representing the two lines that I just
deleted so to wrap up I need to put each
instance there
to the curly brace so I put a curly
brace at the beginning curly black curly
brace clothes and for the second
instance I'm again putting a curly brace
at the beginning and one at the end and
I save the file and then we go to our
wiki Explorer we are going to open the
file and we're going down to a sparse
example we click it and you can see that
it's now opening in Weka Explorer I hope
that helps thank you very much