Working with StatCrunch: Outliers: Exclusions and Transformations Video Lecture | Statcrunch Tutorials for Beginners - Data & Analytics

hi my name is bart poulson and in this
tutorial i'm gonna show how to deal with
outliers or extreme scores in statcrunch
i'm specifically i'm going to cover two
different ways one is simple exclusion
or deletion and the other one is with
the logarithmic transformation now I'm
going to use a data set that's in
statcrunch is called responses to the
social networking survey if you go to
explore data answers for Facebook this
one will come up it'll be the top one
and you know in this one we have data
from a whole bunch of people with C many
we've got data from yep you know a
little over 200 people and we have their
gender their age what a social
networking site they access most
frequently their primary site Facebook
MySpace Twitter and so on and how many
times in one week that they access that
site so anyhow one of the variables in
here has some serious issues with
outliers and it's this one right here
it's x and i'll show you that by making
first a histogram of x and let's see
here normally I would do a title I'm
just gonna hit create graph and you can
see that right here we have this huge
bar that has nearly everybody it goes
from 0 times per week to 100 times broke
and then we got just a tiny bit of
people over a hundred and then we've got
somebody way up here in the 700s and
this of course is not a distribution
that you wouldn't want to be working
with I'll show you another way to deal
with this distribution to look at it and
that is for the box plot because that
helps identify outliers so I'm gonna do
that one but when you use box plots in
statcrunch ii absolutely positively must
go to the next page to say use fences to
identify outliers I also like to draw
them horizontally but that's optional
I'll press create graph and there you
can see I got this tiny little box was
squished way down here
with all these outliers including
one-way appear to 700 needless to say
this is a problem because most of the
statistics that we would use things like
the mean and the standard deviation and
correlation and regression assume that
you're dealing with a nice smooth bell
curve and outliers are one of the worst
things that can happen in a distribution
now I'm gonna show you two ways of
dealing with this number one is we can
simply exclude all of the outlying
scores and I'm gonna do this in a
slightly iterative process by the way
I'm gonna bring up the other shirt also
so I have both the histogram and the box
plot right here but I'm just gonna
modify the box plot for right now by
clicking on options and edit I come back
to the dialog box and if I back up I can
use this statement right here that says
where and I type in x and let's start by
getting it down to 100 so x has to be
less than 100 that means look they have
to access their site less than 100 times
per week hey Chris create graph okay
it's better but I still have a lot of
outliers in fact let me come down let me
cut these ones off this is this is
getting a little better right here so
I'm gonna do the edit again and I think
I'll do it at 25 and look I still got
some outliers up here 20 by the way the
whiskers are moving because I'm
eliminating a number of scores and
that's changing the quartiles a little
bit I'm gonna do one more exclusion
modify it one more time down to 20 so
only look at people who've done it less
than 20 times finally I have a very
reasonable box block if I do a similar
thing over here and I'd back up and I do
where it's climbs by the way
capitalization matters with variables
you gotta capitalize it exactly the same
way and then I have a much better box
plot here it's you know is skewed but
it's nothing like it was before
now if you do this you can go ahead and
you can do your analyses however there
is an issue and that is I've deleted a
lot of scores because I've done this
before I we started with a hundred and
seventy one I've deleted twenty-eight
scores or sixteen percent of the sample
now sometimes you can deal with that we
still have a reasonably large sample
size but you don't really want to lose
that much data especially because there
are other ways to deal with it that
maintain the data so what I'm gonna do
now so I'll just move these up here and
I'm gonna show another way and this was
called a transformation well let me just
show how it works I come over here to
data to transform data and what I'm
gonna do is I'm going to take this one
variable x and because I have outliers
on the high end I'm gonna use something
called the logarithmic transformation
now it's logarithms and what it does is
it come to calculate the logarithm for
each number it takes a base number you
can be 10 or it can be what's called Yi
or the base for the natural logarithm
which is about 2.71828 bla bla bla and
then it finds out what exponent you have
to use to get the the number in question
and then that number the exponent is the
logarithm and what it does is it brings
it gives high numbers much smaller
numbers and can bring things in let me
show you how it works this gets a little
complicated here but I'm gonna say the
variable that I am dealing with is this
one x and I want the logarithm of it so
I just click that one right there I like
the natural logarithm better but the two
function identically they accomplish the
same thing this is if I had a second
variable you know if you come down here
I can have more complicated expressions
and this is where I could put in
constants if I wanted not dealing with
them I just want the logarithm of Y
which is time to press set expression
and see there it is right there
logarithm natural log that's Ln and then
in parenthesis of the variable times
that it puts in quotation marks I press
compute and I get a little dialog box
that says it's added in a new column to
the data table
and here it is Ln for natural log of
times and you see how the numbers they
go from zero zero gives you is for the
score of one if they put zero originally
you can't do a logarithm
that's so we're gonna lose a little bit
of data from people who don't access the
sites at all per week then again if we
want to limit ourselves to people who
actually do access this ice this is a
reasonable thing but you take some of
the large numbers like this 15 you see
becomes a much smaller 2.7 whereas the
score of 4 becomes 1.3 so they're a lot
closer anyhow watch what happens if I do
the histogram of log x this is with all
of the scores in it including the person
who accessed sites over seven hundred
times a week which by the way
corresponds to every ten minutes during
the 16 waking hours every day for seven
days a week so anyhow
somebody's got to find something better
to do but please note it's not a
horrible distribution it's skewed but
it's not you know pathological and I'll
show you another one or I just get a box
plot and I'm gonna get for long times
I'm gonna make sure that I have the
fences and I've drawn it horizontally
and I press create graph and check it
out this part right here in the middle
is almost perfectly symmetrical up to
here I do have two outliers but there's
only two out of 171 scores it's not
horrible if I want to I could exclude
those ones I'm gonna bring up all four
charts now two three four here we go
shaking through the parent here's the
histogram with an exclusion here's the
box bought with an exclusion here's the
histogram with everybody using the
logarithm scores is the box bought with
everybody using logarithm scores let me
just emphasize the difference here by
removing this exclusion statement
there's the original box plot and I'm
gonna remove that statement too and look
this is with everybody in the data and
this is with everybody in the data it
became a much more usable distribution
over here and over here now the
transformation I use the logarithms is
good for when the outliers are on the
high end
outliers on the low end then something
like squaring the scores which would
push the high the higher numbers out
much further would be a good choice
anyhow what this does is it gives you a
way of helping meet some of the
assumptions or prerequisites for the
statistical procedures that you use and
make better use of your data I hope that
helps