Weka Tutorial 21 Merge and Append ARFF files (Data Preprocessing) Video

hello and welcome to another tutorial on
wake up in this wake a tutorial I'm
going to show you how you can append and
March to or more than two our files
together so in weka appending and
merging are two different things
though in English it may feel like that
both of the both of the terms are
denoting to the same thing appending and
marching but they're quite different in
Weka so you realize the difference when
I'm going through the tutorial so at
first we will take a look at the append
and then we will take a look at the
March technique to a pen and March two
or more than two files together but for
the tutorials sake I'm going to show you
how you can append two files together
two datasets together and you can always
you can always a pen or March more than
two files so here in my H Drive you can
see that I'm having two files called
training and test ARF
so at first I'm going to append a file
together so what's a pending well if you
if you open the test dot our file here
with the notepad plus plus you can see
that here we are having five attributes
or features outlook temperature humidity
windy and play and in this file I can
tell you that there are 14 instances so
these are the instances from line 10 to
2023 we have 14 instances here and five
attributes so I'm going to take these
instances these 14 instances all I'm
trying to do is I'm going to take these
instances from this file test dot ARF
and I'm going to put those data at the
very end of training our file so here
from line 24 I'm going to append these
information for
my tests dot our file so that's what I'm
going to do so in order to do that I'm
going to go to simple CLI option not
explorer experimenter or knowledge flow
this time simple CLI when you are
clicking that you have a window open
just like this so in that window you can
see that there is there is a there is
some place where you can write down some
instructions or command lines so I'm
going to use that part of this window so
as I'm going there I'm going to write
down some Java comment Java basic
commands so that's that starts with Java
and and now I'm going to use some of the
classes that are included in your wakeup
package so that class is Weka dot cor
dot instances remember to put a capital
idea
now the command is append space now I
have to put the path the directory path
and the filename there so my file first
file goes there training and my
directory is H then put a space and you
have to write down the second file h :
slash test arf a space a redirection
operator or greater than sign in your
keyboard then another space and H : /
append dot R so what i'm doing here i'm
going to take the data portion of test
dot r file and i'm going to put them at
the end of the data section of training
dot r file and i'm going to create a new
file appender
ARF that actually contains the data from
training and test our files so as I hit
the bottom you can see that there are
some messages like finished redirecting
output to append dot R so let's go back
to my H Drive is there anything nearly
created and
yes you can see that appended our file
is now created and if I open that with
my dog pet plus plus you can see that
previously we had 14 instances in my
training and test data now we have 28 so
the data from the second file test dot
ARF they are actually appended in in
this newly created file called append
ARF so why is this needed this is very
well-known common practice in machine
learning that you have your data set
split into training and tests and their
different data you are going to induce a
classifier from the training data and
you are applying that classifier on the
test data that's common now if you have
a very good result on the test set this
is a very well-known practice that you
combine all the data together from
training and test splits and then you
have a new new data set and apply thus
that same classifier on the whole data
set with a 10-fold cross-validation if
you have a good result and you can you
can publish the result okay so that's a
common practice so what happened here is
a my training dot ARF here it had 14
instances and SEP 5 attributes and my
test dot our file that that also had 14
instances and 5 attributes and and I
appended them together so these appended
our file now have five attributes and 28
instances so that's how you deal with
append and in Weka now the thing is that
in order to append them both both of the
files for this case is the training and
test their attribute name and the type
and also if that's a discrete attribute
of the discrete value attribute data in
that case the attribute values they have
to match with the other file with each
other exactly so if you have something
like Outlook here in this file
the test dot our file cannot have any
misspelled feature name just like this
so the name has to be the same the type
has to be the same and in case of
discrete valued attributes that value
the ranges the attribute can take they
also have to be the same otherwise you
were getting an error so that's pretty
straightforward okay so now we will take
a look at how a mark is done in wake up
so again we go to the simple CLI and now
again I'm going to that CLI input pin
and I'm going to use the same same
almost same semi similar command but
instead of a pin I'm going to use the
word called March ok so before doing
that I have to change the file so just
take a look if I if I hit the button
enter button now when I'm getting some
errors here okay why are the errors okay
so the March thing is a bit different so
we have we have our test dot our file
here and training dot our file here so
what merging is is that the tests are
the training dot up we are trying to
mark these two files together ok so now
the thing is that sometimes it can be
the case that you have Outlook
temperature humidity windy and play in
one file then you produce another set of
file with the same number of instances
14 instances in test dot R and 14
instances in training darf but their
attributes are different okay so what do
you do at that time you are actually
merging not appending so in this test
dot or file I'm going to change the name
of the attributes for example it's not
Outlook we can say that yeah it's it's
going to be Outlook but just change the
name here out Lu - maybe this is a
different feature than Outlook
temperature too
humidity too windy to play too so for
the sake of this tutorial I'm just
simplifying the facts here one file has
outlook temperature humidity windy and
play and maybe you are trying to get
another set of features on the same set
of data here 14 instances and you've got
five new attributes like outlook to
temperature to humidity too windy to and
play too so now we are trying to March
them together so that we have 10
attributes in that file
but for 14 instances ok so how do you do
that now if we go to the simple CLI and
we run the same commands from there then
you can see that finish redirecting
output to H : /i penndot our slope it's
not appending its map margin but I
didn't change the name of the file
output file so if you go to the H Drive
and op reopen the appended our file in
notepad plus plus you can see that now
our data set our date new data set
called appended ARF
it has one two three four five six seven
eight nine ten attributes here so the
attributes are marched and you can see
that is quite different than our
training and test or our files so now we
have 14 instances just like our training
and test out our files but here the
number of attributes is 10 so 1 2 3 4 5
6 7 8 9 10 so that's how March works so
appending and merging there are two
different types of things and you can
see that we've got also got some errors
and you have to realize why you have
those errors in while doing that stuff's
just just the way I've shown to you so
there are there are some subtle
differences but you can play with some
of your files and if you get get the
right thing from this tutorial I think
it's going to be very simple to append
or March to our files to or more than to
our files together
so enjoy your holidays goodbye