Data & Analytics Exam > Data & Analytics Videos > Weka Tutorial > Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing)
Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) Video Lecture | Weka Tutorial - Data & Analytics
|
39 videos
|
FAQs on Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) Video Lecture - Weka Tutorial - Data & Analytics
| 1. What are outliers in data preprocessing? |  |
Ans. Outliers are data points that deviate significantly from other observations in a dataset. They are usually considered as extreme values that are unusually different from the majority of the data points.
| 2. Why is it important to detect and handle outliers in data preprocessing? | |
Ans. Detecting and handling outliers is important in data preprocessing because outliers can have a significant impact on statistical analysis and machine learning models. Outliers can distort the data distribution, affect the mean and standard deviation, and lead to biased results. Therefore, it is crucial to identify and properly handle outliers to ensure accurate and reliable analysis.
| 3. What techniques can be used to detect outliers in data preprocessing? | |
Ans. There are several techniques to detect outliers in data preprocessing, including:
- Z-score method: This method identifies outliers by calculating the standard deviation from the mean and flagging data points that fall outside a specified threshold.
- Box plot: Box plots visualize the distribution of data and can easily identify outliers as points that fall outside the whiskers.
- Interquartile range (IQR) method: This method uses the IQR to determine the range within which most of the data falls. Data points that fall outside this range are considered outliers.
- Mahalanobis distance: This method calculates the distance between each data point and the mean of the dataset, identifying outliers as points with a large distance.
| 4. How can outliers be handled in data preprocessing? | |
Ans. Outliers can be handled in data preprocessing using various techniques, such as:
- Removing outliers: In some cases, outliers can be removed from the dataset if they are determined to be errors or irrelevant to the analysis.
- Transformation: Applying mathematical transformations, such as log transformation or square root transformation, can help reduce the impact of outliers on statistical analysis.
- Winsorization: This technique replaces extreme values with values closer to the mean or within a specified range, reducing the influence of outliers.
- Robust statistical methods: Using robust statistical methods, such as median or trimmed mean, instead of mean and standard deviation, can make the analysis less sensitive to outliers.
| 5. Can outliers be useful in data analysis? | |
Ans. In certain cases, outliers can provide valuable insights in data analysis. Outliers may indicate rare events, anomalous behavior, or important patterns that would otherwise go unnoticed. However, it is important to carefully evaluate and understand the nature of outliers before incorporating them into the analysis, as they can also introduce noise and bias if not properly handled.
Text Transcript from Video
hello youtube and welcome to the 19th
Weka tutorial in this tutorial I'm not
actually continuing a Wicca tutorial 18
second part because what cultural 18
talked about the knowledge flow
environment and I currently am busy now
to work with knowledge flow environment
but guess what I get some really
interesting stuff in Weka and learn this
very new technique in Weka so that's why
I'm thinking to give you a tutorial on
this because many people also ask me if
this is really possible in wake up to do
this stuff I'm going to teach you in
this tutorial so I think this is a good
idea to work on with the weak Explorer a
game in this 19 tutorial and I can
promise you that when I'm a bit free
when getting away from my research work
at that time I'll make a tutorial on the
second part of the knowledge code
environment because many of you emailed
me about the knowledge flow environment
and I'm very glad to see that you really
got you guys really like the knowledge
flow environment of Wicca GUI but stay
tuned I'm going to produce some more
tutorials on knowledge flow but for for
this tutorial this week I'm going to
show you something really new that I
learned last week so the thing is that
many people ask me this question if this
is possible to find out outliers or
extreme values in your data set with the
help of wake up not only that but also
you demanded to know to remove down
flyers and the extreme values from your
data set because you know that outliers
and extreme values sometimes deteriorate
the performance of classifiers you're
going to use on the data set so this
tutorial talks about how to determine
the outliers and extreme values that are
hidden inside your data set and
that are deteriorating your classifier
performance so to work on with that I'll
go back to the wake Explorer and I'm
clicking that I'm going to open a file
of my own research because currently I'm
working on spam detection so I'm going
to talk about spam detection dataset and
it's a very popular data set you know so
I'm going to load some of my work that I
applied on the data set is called CS DMC
2010 spam data set and I'm going to
choose one dot our file for today's
experimentation so this is my 1 dot R
file and you can see that we have over
4,000 instances and the number of
attributes is 21 and the last one is the
class attribute so I'm going to have 20
features in this particular dataset a
plus 1 class feature that has two values
PEM and hem so my test for the tutorial
is to find out doubt liars and the
extreme values first okay
so how am I going to find out the
extreme values and outliers that are
hidden inside my data set in order to
remove them you have to find them first
right ok to find them we are going to
use the filter and I'm going to expand
the filters here I'm moving to
unsupervised filters sorry expand
unsupervised and instance and hello is
that instance because I'm going to
change on the attributes so in the
attributes I have a filter called
interquartile range I'm going to I'm
going to make an abbreviation of that
this is I'm going to call it IQR
okay so interquartile range is a
statistical measure how this works I'm
not going to talk about IQR here you
have to go back to check out on machine
learning or data mining books or even
you can check out with the statistical
books to find out how IQR works so I'm
just selecting that filter this filter
is going to help me to find out the
extreme values or the outliers in my
data set so attributes and IQR so I
selected them here you can see that I've
chosen this filter for the test and you
can you can click it click in here and
it can find lots of options and to know
more about the options you can always go
back to click on the more button here
and you find the synopsis how the
outliers and the extreme values are
calculated and I'm not going to talk
about that I told you before because
this is not a stat class so I'm also
leaving the value 0 Xtreme values factor
and outlie factor here just as the
default one I'm showing you how to use
that I'm not going to explain how this
works and how you can improve your
results so I'm just leaving that to the
default value so I click OK
so now what happens if you apply if you
click on apply apply button here you can
see that something is going on it has
stopped and you produce something new so
previously if you can remember two or
three minutes ago I was telling you that
we have 21 attributes in this data set
20 features plus one feature that's
called cast feature here it is but now
we are having two new features the first
one is called outlier and the outlier
you can see here that it's type is
nominal and it has two distinct values
no and yes so all this is saying is that
at one particular instance in your
dataset is it going to be an outlier or
not if it is an outlier then we have
something called yes in its label for
this particular feature so this is a new
feature 20 second feature in an in my
data set and if it is not an outlier
then we then the IQR will automatically
put a label called no so I have over 3.5
K knows and only 774 yes is here so this
is good for me because I'm not having
too many outliers in my data set ok so
I'm going to explain more about the
outliers and extreme values but here for
this for this moment you can see that we
have another feature included in our
data set it's called extreme value and
extreme values also having the type
nominal and two distinct values called
no NES again if one particular instance
in your data set is is labeled as
extreme values by IQR then IQR is
putting a level label called yes and if
it finds that it's not going to be an
extreme value it's not containing an
extreme value then it puts a label there
as no so what I'm going to show you now
it's always good practice to save your
data set whenever you're modifying this
by using some filter otherwise this is
going to be lost when you are closing
the wake an explorer so I'm going to
save this data set on my desktop and I'm
going to rename this data set as testing
for YouTube ok so it's now been saved on
online desktop and I'm going to read
that open that file and open that our
product you can see that this is a
relation ok and these are the attributes
section when I started this experiment
for this tutorial I was just
having 21 attributes just like this but
when I applied the filter called IQR
interquartile range at that time after
applying that filter I'm having two
extra attributes now in my our file okay
the first one is the nominal attribute
called
outlier it contains either no or yes and
I have I'm having the second attribute
new attribute included in mind our fund
is called extreme value that's also
nominal attribute and it has the value
either low or yes so for instance if you
want to read a particular in start ikkyu
leur instance in your dot in in my
dataset so eighteen point seventy one
seventy eight nineteen point thirty six
thirty six point seven one so these all
are the values I was having previously
up to this point okay so this is my
class attribute that is a spam and then
again I have the twenty twenty second
attribute which is the outlier so it
says yes that means this particular
instance is an outlier but it doesn't
contain any kind of extreme values
because I'm having an O'Hare so TCS is
for being an outlier this particular
instance and this label of no is saying
that this instance in this instance is
not actually containing any extreme
values so in this way you see that by
using IQR this is very easy to identify
if your instances are an outlier or
either an outlier or an extreme fair or
containing an extreme value now it's
time to remove them so in order to
remove them we are again up using a
filter here and this time the filter
will be inside the instance folder here
so I'm going to expand that and you can
see that I have a filter call to remove
with values so I select remove with
values and I'm going to click on that
again to adjust the options
okay so the attribute index here it says
that which attribute are are you going a
widget which actually are you going to
delete or remove from your data set so
I'm saying that okay I'm going to remove
all the outliers is going to be based on
the twenty second attribute in my data
set so the twenty second attribute is
outlier so that's why I choose the
attribute index twenty two and again if
you carefully take a look at here if I
click on here so I'm going to cancel
that I'm choosing outlier and here
instead of class extreme values to color
this with blue and red with extreme
values I'm going to color this week
class outlier the cinoman el feature so
the blue ones are the outliers are not
outliers and the red ones are the
outliers here so what we are going to do
here we are going to remove the outliers
that yesses ok so we are having two the
two distinct values here yes and no so
in this case I'm going to choose that
remove with values feature again and I'm
going to set them up now so as I told
you before I'm going to remove the
outliers from our data set so I choose
22 because that's the number of
attribute attribute number here in my
data set and I'm saying that okay
nominally misses first - last I'm going
to cut that because I'm just going to
I'm just going to keep the up keep the
non outliers in my data set so I'm just
saying that ok remove the last one
because in 20 second attribute the last
one is saying yes labeled as yes because
we are having to add two values here no
and yes so I'm saying that if that is
yes if that is an outlier remove them so
that's why I put last here okay so in
this way I just
okay twenty-second attribute I'm going
to remove them I'm not going to remove
all of the 22nd attribute I'm just
saying that okay you move only those
that are falling inside the last value
which is yes okay so I'm going to click
OK here and I'm again going to apply
this filter and you can see now the
outlier yes is completely gone so if I
color that with outliers there is no
ribs okay so if I save that again
testing for YouTube outlier removed so I
set that on my desktop and I'm going to
view that and you can now see that this
is the outlier feature value you'll see
no ESS there so I removed all the all
the instances that were indicated as
outlier with algorithm called our irq
interval i QR sorry inter quartile range
okay so in the same way you also can
tell okay I'm going to remove the
extreme values as well from my dataset
and input you are in order to do that I
have I have already chosen to remove
with values filter I click on there now
I am going to use the 20 thar attribute
index because that's the number of
attribute attribute index number for the
extreme value feature and I'm also
seeing here no and yes so I'm going to
remove all the yeses okay
those instances that are having extreme
values so nominally in this s is last I
click OK I apply and now I'm going to
save that again with the same name okay
so I'm going to overwrite that if I go
back in here so it is now reloading that
now you can see that the last attribute
here that states either whether or not
it is an extreme value
of this particular instance you can see
that we are not currently having any any
instance with the last attribute that
says yes because we just removed all the
outliers as well as the extreme values
from our dataset so I hope I hope that
you got really helped with this idea of
removing the outliers and extreme values
by applying the interquartile range
first to identify that outliers and
extreme values and then we used the
remove with values filter in order to
remove the outliers and the extreme
values from your data set so the reason
for removing outliers and extreme values
because some of the classifiers they
really cannot perform when you are
having too many outliers or extreme
values in your data set so I hope that
helps I'll be back with a bunch of
tutorial I promise
Weka tutorial in this tutorial I'm not
actually continuing a Wicca tutorial 18
second part because what cultural 18
talked about the knowledge flow
environment and I currently am busy now
to work with knowledge flow environment
but guess what I get some really
interesting stuff in Weka and learn this
very new technique in Weka so that's why
I'm thinking to give you a tutorial on
this because many people also ask me if
this is really possible in wake up to do
this stuff I'm going to teach you in
this tutorial so I think this is a good
idea to work on with the weak Explorer a
game in this 19 tutorial and I can
promise you that when I'm a bit free
when getting away from my research work
at that time I'll make a tutorial on the
second part of the knowledge code
environment because many of you emailed
me about the knowledge flow environment
and I'm very glad to see that you really
got you guys really like the knowledge
flow environment of Wicca GUI but stay
tuned I'm going to produce some more
tutorials on knowledge flow but for for
this tutorial this week I'm going to
show you something really new that I
learned last week so the thing is that
many people ask me this question if this
is possible to find out outliers or
extreme values in your data set with the
help of wake up not only that but also
you demanded to know to remove down
flyers and the extreme values from your
data set because you know that outliers
and extreme values sometimes deteriorate
the performance of classifiers you're
going to use on the data set so this
tutorial talks about how to determine
the outliers and extreme values that are
hidden inside your data set and
that are deteriorating your classifier
performance so to work on with that I'll
go back to the wake Explorer and I'm
clicking that I'm going to open a file
of my own research because currently I'm
working on spam detection so I'm going
to talk about spam detection dataset and
it's a very popular data set you know so
I'm going to load some of my work that I
applied on the data set is called CS DMC
2010 spam data set and I'm going to
choose one dot our file for today's
experimentation so this is my 1 dot R
file and you can see that we have over
4,000 instances and the number of
attributes is 21 and the last one is the
class attribute so I'm going to have 20
features in this particular dataset a
plus 1 class feature that has two values
PEM and hem so my test for the tutorial
is to find out doubt liars and the
extreme values first okay
so how am I going to find out the
extreme values and outliers that are
hidden inside my data set in order to
remove them you have to find them first
right ok to find them we are going to
use the filter and I'm going to expand
the filters here I'm moving to
unsupervised filters sorry expand
unsupervised and instance and hello is
that instance because I'm going to
change on the attributes so in the
attributes I have a filter called
interquartile range I'm going to I'm
going to make an abbreviation of that
this is I'm going to call it IQR
okay so interquartile range is a
statistical measure how this works I'm
not going to talk about IQR here you
have to go back to check out on machine
learning or data mining books or even
you can check out with the statistical
books to find out how IQR works so I'm
just selecting that filter this filter
is going to help me to find out the
extreme values or the outliers in my
data set so attributes and IQR so I
selected them here you can see that I've
chosen this filter for the test and you
can you can click it click in here and
it can find lots of options and to know
more about the options you can always go
back to click on the more button here
and you find the synopsis how the
outliers and the extreme values are
calculated and I'm not going to talk
about that I told you before because
this is not a stat class so I'm also
leaving the value 0 Xtreme values factor
and outlie factor here just as the
default one I'm showing you how to use
that I'm not going to explain how this
works and how you can improve your
results so I'm just leaving that to the
default value so I click OK
so now what happens if you apply if you
click on apply apply button here you can
see that something is going on it has
stopped and you produce something new so
previously if you can remember two or
three minutes ago I was telling you that
we have 21 attributes in this data set
20 features plus one feature that's
called cast feature here it is but now
we are having two new features the first
one is called outlier and the outlier
you can see here that it's type is
nominal and it has two distinct values
no and yes so all this is saying is that
at one particular instance in your
dataset is it going to be an outlier or
not if it is an outlier then we have
something called yes in its label for
this particular feature so this is a new
feature 20 second feature in an in my
data set and if it is not an outlier
then we then the IQR will automatically
put a label called no so I have over 3.5
K knows and only 774 yes is here so this
is good for me because I'm not having
too many outliers in my data set ok so
I'm going to explain more about the
outliers and extreme values but here for
this for this moment you can see that we
have another feature included in our
data set it's called extreme value and
extreme values also having the type
nominal and two distinct values called
no NES again if one particular instance
in your data set is is labeled as
extreme values by IQR then IQR is
putting a level label called yes and if
it finds that it's not going to be an
extreme value it's not containing an
extreme value then it puts a label there
as no so what I'm going to show you now
it's always good practice to save your
data set whenever you're modifying this
by using some filter otherwise this is
going to be lost when you are closing
the wake an explorer so I'm going to
save this data set on my desktop and I'm
going to rename this data set as testing
for YouTube ok so it's now been saved on
online desktop and I'm going to read
that open that file and open that our
product you can see that this is a
relation ok and these are the attributes
section when I started this experiment
for this tutorial I was just
having 21 attributes just like this but
when I applied the filter called IQR
interquartile range at that time after
applying that filter I'm having two
extra attributes now in my our file okay
the first one is the nominal attribute
called
outlier it contains either no or yes and
I have I'm having the second attribute
new attribute included in mind our fund
is called extreme value that's also
nominal attribute and it has the value
either low or yes so for instance if you
want to read a particular in start ikkyu
leur instance in your dot in in my
dataset so eighteen point seventy one
seventy eight nineteen point thirty six
thirty six point seven one so these all
are the values I was having previously
up to this point okay so this is my
class attribute that is a spam and then
again I have the twenty twenty second
attribute which is the outlier so it
says yes that means this particular
instance is an outlier but it doesn't
contain any kind of extreme values
because I'm having an O'Hare so TCS is
for being an outlier this particular
instance and this label of no is saying
that this instance in this instance is
not actually containing any extreme
values so in this way you see that by
using IQR this is very easy to identify
if your instances are an outlier or
either an outlier or an extreme fair or
containing an extreme value now it's
time to remove them so in order to
remove them we are again up using a
filter here and this time the filter
will be inside the instance folder here
so I'm going to expand that and you can
see that I have a filter call to remove
with values so I select remove with
values and I'm going to click on that
again to adjust the options
okay so the attribute index here it says
that which attribute are are you going a
widget which actually are you going to
delete or remove from your data set so
I'm saying that okay I'm going to remove
all the outliers is going to be based on
the twenty second attribute in my data
set so the twenty second attribute is
outlier so that's why I choose the
attribute index twenty two and again if
you carefully take a look at here if I
click on here so I'm going to cancel
that I'm choosing outlier and here
instead of class extreme values to color
this with blue and red with extreme
values I'm going to color this week
class outlier the cinoman el feature so
the blue ones are the outliers are not
outliers and the red ones are the
outliers here so what we are going to do
here we are going to remove the outliers
that yesses ok so we are having two the
two distinct values here yes and no so
in this case I'm going to choose that
remove with values feature again and I'm
going to set them up now so as I told
you before I'm going to remove the
outliers from our data set so I choose
22 because that's the number of
attribute attribute number here in my
data set and I'm saying that okay
nominally misses first - last I'm going
to cut that because I'm just going to
I'm just going to keep the up keep the
non outliers in my data set so I'm just
saying that ok remove the last one
because in 20 second attribute the last
one is saying yes labeled as yes because
we are having to add two values here no
and yes so I'm saying that if that is
yes if that is an outlier remove them so
that's why I put last here okay so in
this way I just
okay twenty-second attribute I'm going
to remove them I'm not going to remove
all of the 22nd attribute I'm just
saying that okay you move only those
that are falling inside the last value
which is yes okay so I'm going to click
OK here and I'm again going to apply
this filter and you can see now the
outlier yes is completely gone so if I
color that with outliers there is no
ribs okay so if I save that again
testing for YouTube outlier removed so I
set that on my desktop and I'm going to
view that and you can now see that this
is the outlier feature value you'll see
no ESS there so I removed all the all
the instances that were indicated as
outlier with algorithm called our irq
interval i QR sorry inter quartile range
okay so in the same way you also can
tell okay I'm going to remove the
extreme values as well from my dataset
and input you are in order to do that I
have I have already chosen to remove
with values filter I click on there now
I am going to use the 20 thar attribute
index because that's the number of
attribute attribute index number for the
extreme value feature and I'm also
seeing here no and yes so I'm going to
remove all the yeses okay
those instances that are having extreme
values so nominally in this s is last I
click OK I apply and now I'm going to
save that again with the same name okay
so I'm going to overwrite that if I go
back in here so it is now reloading that
now you can see that the last attribute
here that states either whether or not
it is an extreme value
of this particular instance you can see
that we are not currently having any any
instance with the last attribute that
says yes because we just removed all the
outliers as well as the extreme values
from our dataset so I hope I hope that
you got really helped with this idea of
removing the outliers and extreme values
by applying the interquartile range
first to identify that outliers and
extreme values and then we used the
remove with values filter in order to
remove the outliers and the extreme
values from your data set so the reason
for removing outliers and extreme values
because some of the classifiers they
really cannot perform when you are
having too many outliers or extreme
values in your data set so I hope that
helps I'll be back with a bunch of
tutorial I promise
Related Exams
About this Video
|
4.74/5 Rating |
|
Dec 23, 2024 Last updated |
Video Description: Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) for Data & Analytics 2024 is part of Weka Tutorial preparation.
The notes and questions for Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) have been prepared according to the Data & Analytics exam syllabus.
Information about Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) covers all important topics for Data & Analytics 2024 Exam.
Find important definitions, questions, notes, meanings, examples, exercises and tests below for Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing).
Introduction of Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) in English is available as part of our Weka Tutorial for Data & Analytics & Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) in
Hindi for Weka Tutorial course. Download more important topics related with notes, lectures and mock test series for
Data & Analytics Exam by signing up for free.
Description
Video Lecture & Questions for Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) Video Lecture | Weka Tutorial - Data & Analytics - Data & Analytics full syllabus preparation | Free video for Data & Analytics exam to prepare for Weka Tutorial.
Information about Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing)
Here you can find the meaning of Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) defined & explained in the simplest way possible.
Besides explaining types of Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) theory, EduRev gives you an ample number of questions to practice Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) tests,
examples and also practice Data & Analytics tests.
|
39 videos
|
|
Explore Courses for Data & Analytics exam
|
|
Signup for Free!
Signup to see your scores go up within 7 days! Learn & Practice with 1000+ FREE Notes, Videos & Tests.
Related Searches
practice quizzes
,Important questions
,Sample Paper
,Semester Notes
,Extra Questions
,Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) Video Lecture | Weka Tutorial - Data & Analytics
,Previous Year Questions with Solutions
,Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) Video Lecture | Weka Tutorial - Data & Analytics
,Summary
,study material
,Exam
,video lectures
,ppt
,Viva Questions
,MCQs
,Objective type Questions
,Free
,Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) Video Lecture | Weka Tutorial - Data & Analytics
,mock tests for examination
,shortcuts and tricks
,past year papers
;
Study Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) on the App
Students of Data & Analytics can study Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) alongwith tests & analysis from the EduRev app,
which will help them while preparing for their exam. Apart from the Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing),
students can also utilize the EduRev App for other study materials such as previous year question papers, syllabus, important questions, etc.
The EduRev App will make your learning easier as you can access it from anywhere you want.
The content of Weka Tutorial 19: Outliers and Extreme Values (Data Preprocessing) is prepared as per the latest Data & Analytics syllabus.
|
© EduRev
|
Education Revolution
|
|
Signup to see your scores
go up within 7 days!
Access 1000+ FREE Docs, Videos and Tests
Takes less than 10 seconds to signup