25 Introducing RapidMiner Radoop Video Lecture | Rapid Miner: Learn and Master - Data & Analytics

Hello.
Welcome to this RapidMiner Radoop Tutorial.
In this tutorial we will briefly describe
how to connect to Hadoop using the Radoop
extension and then we will show you how to
create a prediction using the Radoop operators.
First thing to do before you get started with
Hadoop is to install the 'Radoop' extension
from the 'RapidMiner Marketplace'.
Please check out the video on extensions for
detailed instructions.
After the installation you will see an additional
folder called 'Radoop' in your 'Extensions'
folder of the 'operators' panel and also the
additional 'Hadoop Data'-view.
For this demonstration we are using the Hortonworks
sandbox.
The 72 operators coming from the Radoop extension
allow you in essence to seamlessly use a variety
of functions for a number of technologies
such as Hive, Apache Spark and Mahout to prepare
and to model your data.
In order to access the data we of course need
to establish a connection to our Hadoop cluster
or in our case to our small sandbox.
The details on connecting to a cluster are
beyond the scope of this tutorial but we still
want to briefly touch on the different elements
of the setup dialog.
To see the details of our existing connection
click on the 'Connections'-menu and select
'Manage Radoop Connections'.
You can see that we have a connection setup
here called 'Hortonworks Sandbox'.
I'll select 'configure' to give you a quick
walk-through.
You can see below the name of the connection
and the option to fill in information about
your specific installation.
If you expend the 'Hadoop version' drop-down
you can see the large number of platforms
which we are supporting.
In the next block you have the 'Master settings'
where you can also add Multiple Masters for
larger installations.
I have filled in the local IP-address as you
can see and left everything else at the default
to match the sandbox settings.
In the SQL-settings I have adjusted the default
to be set to the latest Hive-Server version.
Of course you can also add your security settings
as for example your Kerberos setup or others.
For this demo we have further adjusted the
Spark version and the location of the Spark
library and set one additional default Spark
parameter.
After you have adjusted your settings we recommend
you execute the 'Quick test' and the 'full
test' to ensure everything is working properly.
Before we start creating a process let's check
out quickly the 'Hadoop Data'- view.
This view helps us to assess the data which
is available for us in our cluster.
We can retrieve the metadata by double-clicking
on our connection.
Once loaded we have a nice overview of our
tables.
If we click for example on our 'titanic_training'
table we can see what attributes it contains.
You could even retrieve a preview here if
you need to take a quick look at a sample
of your data.
Now let's go back to our design View and start
building a process with Radoop.
We will drag and drop in the most important
operator which is called 'Radoop Nest'.
You will find it all by itself on the top
level of the ‘Radoop extension’ folder,
emphasizing its importance.
The nest operator causes everything to happen
on the cluster, instead of on your local machine,
which is what you want in order to use the
power of your cluster.
Let's connect the 'Nest'-operator to the results
port.
By doing so, we are actually allowing information
to be brought back to our local machine for
display in the 'Results' view.
However, it will always only be a sample to
not clog your system.
You can tick the "change sample size" check-box
in the 'parameters' panel in order to see
the sample size and adjust it to your needs.
This is, by the way, one of the options to
pull data from your cluster into your local
environment and work with it there, using
the standard RapidMiner operators or even
in case you need to do a data-dump.
Keep in mind that the sample size will always
work like a hard-coded filter.
Now let's double-click the 'Nest' operator
to enter the sub-process level, where we will
configure a simple prediction on the titanic
incident.
For that, we are going to use the same data
which is available in your RapidMiner Studio
'samples' folder - just in case you want to
follow along in a non-Radoop fashion.
We will start by using the 'Retrieve' operator
from our Radoop extension.
The name is in this case a bit misleading
as we are not really retrieving the data but
it remains in Hadoop, as mentioned before.
When we click on the table drop-down in the
parameters Panel, we can see that we are now
connecting our sandbox with the available
tables so the meta-data has been gathered
here for us here as well.
The table we want to use now is the "Titanic
Training" data.
To show you what is in this data we can simply
execute the process.
You are seeing an indication that it is running
and finally we get the data displayed because
we had previously connected the 'Nest' operator
to our results port of RapidMiner Studio.
In our case it is really all the data as our
table is with just a bit more than 900 rows
- or examples as we call them in RapidMiner
– a very small dataset.
Looking at the data you can see some of the
known information about the passengers of
that famous cruise.
We do have most importantly a label - indicated
here in green - which tells us who survived
and who didn't.
Also we have some further information about
the passengers like: Age, Passenger class,
gender, some information about family members
on board and the price they paid for the voyage
which we want to use to build a predictive
model from.
Now let's go back to our 'design' view and
into our 'nest' operator.
The second thing we are going to show you
is a simple validation.
We will use a 'split validation' operator
here - again make sure to use the one from
the Radoop extension folder.
We will do a 70 to 30 split on our data.
Now within the 'split' operator we have the
two sides, one for 'training' and one for
'testing' the model.
We will populate the two with a 'decision
tree' model, an 'apply model' and a 'performance'
operator.
Which we can drag and drop conveniently from
our 'wisdom of the crowds' section or from
our ‘Operators’ panel.
For the decision tree you can see the 'Spark'
icon and for the other two you see the 'Hive'
icon displayed on the operator.
So for the Nesting we use standard Hadoop
technology and for the other features we are
using as mentioned other technologies such
as Spark and Hive - all that just by pulling
in the RapidMiner Radoop operators.
We will leave all settings as default, except
for the decision tree where we will select
'entropy' as criterion as that is better suited
for attributes that occur in classes.
If you would like to learn some more on, models,
validation and performance interpretation,
please refer to the respective video in our
'Getting Stared' series and also to the '5-minutes
with Ingo' videos, which you can find on our
YouTube channel.
In this demo we will just continue to connect
all the relevant ports in the 'training' and
in the 'testing' section, so we can create
and assess the quality of our model.
In the last step we will apply the model to
some unlabeled data which will conclude our
prediction process.
So let's move up to the relevant level to
retrieve the unlabeled data.
The unlabeled data has no information about
the survival of the passengers which it contains,
so this is what we want to model.
To retrieve it, again we need to use the 'Retrieve
from Hive' operator and this time we will
have it pull up the table called 'titanic
unlabeled'.
We are going to apply our model to the new
data.
For that we have to use again our 'apply model'
operator.
Remember to use the one with the Hive-icon!
So now we can connect the 'out' port from
our second retrieve operator to our 'apply
model' operator.
We retrieve the model from the 'mod' port
of our validation operator and we send the
information to our results.
We will output the, now 'labeled' data, the
model and the performance from our validation.
This means we have three outputs which we
need to account for also in our 'Nest' operator
if we want to see everything displayed in
the 'results' view of RapidMiner Studio.
So let's go one level up and provide three
output connections.
And as we execute this by hitting the 'Run
process' button, you can see the progress
on the execution indicated here by the icon.
In the bottom we get an information displayed
what is currently executed.
We are now in the validation.
If we access the Hadoop cluster via our web-browser
we can see that a job has been submitted and
is status undefined.
If we give it a bit of time it will finish,
so let's go back to RapidMiner Studio in the
meantime and have a look at the results.
The model we generated is a decision tree
which is showing us that gender was the most
important attribute to determine survival.
So the "women and children first" etiquette
was actually followed.
The next important attributes were - not surprisingly
- of a less noble background.
It was 'passenger class' and 'price of the
ticket' which determined the faith of the
passengers.
If we look at the output of our performance
we can see the overall accuracy and finally
we can then see the results of our prediction
on the unlabeled data.
It has been amended with a predicted label
together with the confidence values.
So this concludes our introductory tutorial
on RapidMiner Radoop.
The most important items to keep in mind is
that, everything needs to be nested in order
to have it executed on the Hadoop cluster
instead of on your local machine or RapidMiner
Server, and the Radoop operators need to be
used to execute the relevant steps.
Of course there are options to use the RapidMiner
Radoop operators in conjunction with the usual
RapidMiner operators but this is beyond the
scope of this brief video.
Thank you very much for watching!