07 Turbo Prep Cleansing Video Lecture | Rapid Miner: Learn and Master - Data & Analytics

[Music]
inst real-world scenarios the data for
your analysis does not come in perfect
shape often it needs to be joined
together from multiple tables and when
that is done you find out that entries
are missing have inconsistent formats
duplicates or simple typos so the task
of data cleansing is one of the
important steps during data preparation
therefore we have created a whole
section dedicated to this pre-processing
step in rapid miner turbo prep we have
preloaded the Titanic data set from
rapid miner Studios samples folder the
example set has 1309 rows and 12 columns
or attributes some of them like name or
ticket number are similar to a unique
identifier as indicated others are
unstructured or have missing entries
when you open a cleanse session then
most actions are deactivated because
we've not selected a column yet as I
select the lifeboat column you can see
that replace missing dummy encoding and
remove duplicates are being activated
the many missing entries are not because
they've not been populated but because
most of the passengers did unfortunately
not reach a lifeboat therefore I can
replace the missing entries with
relevant information and use for example
the specific value none the other active
options are dummy encoding and remove
duplicates for remove duplicates it is
important to know that it only checks
for duplicates in the data that you have
selected so if I apply it here you see
that only a few rows are remaining
because each distinct value in this
column is now shown only once if you
want to remove entries which are similar
in some or all attributes and select the
relevant columns or press ctrl-a or
command a to mark all and then use the
remove duplicates operation ok I'll go
back and highlight only my lifeboat
column it is currently formatted as a
categorical column because it contains
numbers and text some models can't
handle text so if we would want to turn
this into a numeric column we could use
dummy encoding dummy encoding creates a
column for each distinct value and then
it assigns ones to each row where that
value occurs so for a large number of
distinct values in a column it will
increase the width of your
table substantially the column details
can help to assess this for our lifeboat
attribute it is twenty eight distinct
valve imbues make sure to weigh cost and
benefit when using dummy encoding as it
may have a negative impact on the speed
with which your models are being
calculated but it can also help you to
yield useful information speaking of the
number of columns the generic cleansing
operations remove correlated and remove
low-quality are available to tackle that
exact problem when you're creating a
model and you have two highly correlated
columns you won't gain any extra boost
for your model if you take both into
consideration but you will have a longer
calculation time if you do so getting
rid of highly correlated columns as well
as those with many missing is a good
best practice the remove correlated
option is very simple to execute as you
only define their correlation threshold
before you run it note that the input is
interpreted as an absolute value so it
applies to positive and negative
correlations alike remove low-quality is
configured through a combination of
quality measure thresholds you can use
the default or in case you want to
fine-tune this procedure then you can
find more details here okay so far we've
looked at cleansing categorical
variables individually and at reducing
the number of columns with a more
general approach the only thing left is
to look at numeric attributes I'll
select age here and you can see that the
applicable operations are becoming
active of course we can replace missing
which in case of age could be achieved
by simply filling in the average age in
cases where you want to change a numeric
variable into a category you can use the
disk read Isaiah to do a binning with
bins of equal size or range if you have
multiple numeric attributes with values
of different scale and variability
like for example age and ticket price
then you can normalize each or all of
them the normalization will be useful in
such cases to balance your numeric
values in cases where you have many
numeric attributes a principal component
analysis might be helpful to apply the
PCA result will be several numerical
columns which are linear combinations of
the original data while keeping the
variance okay this was a brief review of
all the end
procedures for numerical values we won't
use any on our data so we can commit our
cleansing adjustments now let's see what
Auto cleanse does if we open the help
then we can see the steps which are
applied we've just discussed and applied
some of the steps shown and so instead
of applying all those manually you could
use auto cleansing instead it will apply
all of them for you with a structured
walkthrough which leads to the desired
results very quickly we will go through
it now but first please take note of the
number of rows and columns so we can see
the impact in the first step we're asked
to select a target column selecting one
means that this column is excluded from
the cleansing this is important because
you would want to preserve it even in
cases of high correlation with any of
the other variables rows with missing
values in the target column should also
be excluded from cleansing since you
might want to apply your model to those
later in this next step we get a preview
of which columns will be removed name
and ticket ID are excluded because with
all their unique data they're not
helpful to generate a generalized model
and neither is cabin due to too many
missing entries as you can see our dummy
coded lifeboat column which represents
the value none is being kept as it
contains valuable information in
contrast to the others which are mostly
zero and therefore very stable the type
conversion step is up next keep in mind
here that this is always applied to all
columns if you want to do it for a
subset then going through multiple
cleansing cycles like we did here is the
best approach in doubt just leave things
as they are
now this last configuration screen deals
with numeric columns and so only appears
if you have any in your data set you can
choose to apply a PCA or normalize or
even do both I've explained the general
principles of each operation before so
I'm just finishing this dialog
walkthrough before I commit the
cleansing we have now fewer than ten
columns left and prepared them to be
used for further processing or as input
for auto model I can jump into it right
here and as I go through the dialog
quickly you can see that we're warned
about this lifeboat variable this type
of warning is in fact important and
should not be ignored when building
models
unlike we're doing here
you look at the results you see
extremely high accuracies which are of
course unrealistic these are due to the
highly correlated lifeboat column the
information who did or didn't reach a
lifeboat is an almost perfect proxy to
predict survival and would not have been
suitable for use in our scenario as
indicated by auto model before to learn
more on this and other details on how to
use auto model please check out our
other tutorials thank you very much for
watching
[Music]
you