Big Data Tutorial For Beginners - 9 |Hadoop Pig Tutorial For Beginners| Pig Programming |Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

hello and welcome to lesson I'm of the
big data and lube developer course
offered by sin to learn this lesson
we'll focus on TIG which is the
analytics component of the Hadoop
ecosystem what you will learn in this
class after completing the lesson you
should be able to first explain the
concepts of pig understand the types of
data models and data types supported by
pic differentiate pig and sequel we will
go over the similarities and the
differences between the two understand
the functionality required to operate
with the scripting language and finally
you should be familiar with basic
vignettes prior to 2006 there was only
one option to to develop MapReduce
programs on Hadoop and that was using
Java programming language while
MapReduce with Java is quite powerful
it's not very easy to work with
developers had to be mindful of
MapReduce specifics like labs or truffle
and reduce simple even simple operation
like joining and filtering required
quite a lot of code finally developers
had to be mindful of the complex data
flow where the output of one task can be
used as the input to another task to
overcome all these complexities kid was
developed
in late two thousand six by the people
at Yahoo it later became an Apache open
source project so basically pig is
another language besides Java to to
write MapReduce programs so what is pic
let's define it first peg is a scripting
platform the truant on Hadoop clusters
designed to process and analyze large
data sets the features that make take
very popular with developers are its
extensibility self-optimizing feature
and the ease of programming you first
you obviously you don't need to know
about in order to write pig scripts it
can work with both structured and
unstructured data and it uses HDFS both
as a bigger source and as the
destination to store the results of the
analysis and finally tag includes a
number of self optimization features
which we will shortly call in greater
detail so let's look an example to
understand in what circumstances pig
would be most useful so in our example
data scientists fuse grid tools to scan
through terabytes of data they write
scripts to test areas or to gain deeper
insights but the challenges that in the
data factory the data may not be uniform
and may not be standardized it can be
structured though semi structured
the schemas may only be partially known
or unknown and with this set of
circumstances these are all the
challenges that are addressed by pig
letting language and and this is what
makes pick a good solution for this sort
of challenges let's look at the
components of static the two major
components are first the pig leading
scripting language and second the
runtime engine the scripting language is
basically a procedural dataflow engine
and it has a simple and intuitive syntax
and commands that are applied to
implement business logic for example
command as simple as load em store the
runtime engine on the other hand it's
basically a complete compiler that
converts tick scripts into MapReduce
jobs that advance submitted to Hadoop
and the results can distort the HDFS so
the tip interacts with the dupe with
both of its components the HDFS and the
MapReduce Pig operations can be broken
down into three major stages first stage
consists of reading the script and
submitting it in this example it's a
simple script that filters out data and
n returns summary group by field
the first line of the command is the
load command which defines our data
source the second stage of processing
happens in the execution engine where
the script script is parsed validated
checked for errors and optimizations are
performed at this stage and as a result
of those optimizations execution terms
are generated now then according to
those execution plans the MapReduce jobs
are created and submitted for execution
while the execution is in progress the
ancient continues to monitor those jobs
and the final final stage of the
execution would be the actual execution
of the plan and returning the results
and those results can be either returned
to the screen or they can be written to
HD SS next we'll look at a few salient
features of fig that make it so popular
with developers first one is
step-by-step procedural control many
programmers find this step-by-step
procedural mode more intuitive and
easier to work with them equal second
one is how it approaches the schemas
keema's our first of all optional and
second they can be assigned dynamically
so developers are not required to have a
complete understanding of their entire
schemas and finally take offer support
for many data types some of them they
plaques and very flexible and it's offer
support for user-defined functions next
we'll look at the data type supported by
pit there are four basic data types
first is Adam and that can be any atomic
value can be of any data type in the
long double string integer as long as
it's an atomic data type second one is a
couple still tuples can also be off any
data type the elements within the topple
third one is a bag and that's basically
a collection of towels and nice thing
about it is that those tuples within the
back they don't have to be uniform they
can have elements of different data
types and the number of elements can
also vary for each sample and the lost
data type is map which is basically an
associative array the array itself must
be of character data types but the
elements within very can be of any type
so those are the four data types
supported by pic now how how does the
engine recognize and interpret data
types there are several ways first they
can be explicitly defined either in the
schema or in the f clause of the script
so you can explicitly tell the engine
what the data facts are but it can also
infer the data types
and it can baste it can base it on on
several things one would be they did the
return type of a user-defined function
and the other one could be based on the
iteration that you are trying to perform
for example if you are performing in
America operation ticket will assume
that the data types are numeric another
important point is that type conversion
is lazy which means that the evaluation
does not happen until the execution
stage now if we do not declare data type
and if it is no way to infer the data
tech it will by default assume binary
another important point about tech data
model is that it is fully nestable which
means that one type can be nested within
the other as the scene with complex
types like bags or lists and this offers
a lot of flexibility and it's natural
and more intuitive to programmers and
another important benefit of this
flexibility is that in a single data
source you can even combine records of
many different types and that means you
can avoid expensive joints because all
of your data lets in a single file
single source there are two execution
modes available with pick the local mode
the script
runs on your local machine and in that
forgives mode the script will be
interpreted and and it will be
transformed into a set of MapReduce jobs
which will be then submitted for
execution and that's the mode you will
be most often operating with the
MapReduce mode interactivity in the
interactive mode you can you can submit
pic commands one line at a time and they
will be executed as they are submitted
in the batch mode you create a script
and then submit the entire script for
execution again this is the bedroom
business most common modes for executing
pic next let's compare pic and sequel
there are some similarities but there
are some important differences as well
well see fool is a query language that
is intended to interact with the
relational databases it is really
scripting language that is designed to
interact with HDFS simple queries are
executed every single block well with
pig is is it uses a procedural
step-by-step style evaluation with
sequel the evaluation is immediate you
need to understand your entire structure
on before optimizer can come up with
their execution plan while 10 users lazy
execution
it does not happen until the execution
step pipelines love supported in pick in
sequel it's quite difficult to create
intermediate results while in pig this
is natural and it's something used quite
often next let's look at examples of
simple prey and equivalent X crypt so in
our example we're looking at two data
sources sales and customers in in the
case of sequel those are database tables
and in case of Ted those files on HDFS
so our task is to join the two tables
and then to return the summary of
amounts by customer we apply filters by
city and the final results are also
filtered as well based on the aggregated
summary and finally the output is sorted
so when we look at tags the first ones
execute the load command that is where
the we define our two data sources one
is the customer file and the other one
is the sales file in in the f clause we
define the data elements in each in each
source we apply a filter by setting we
execute the join between the two files
we execute a group by command
and finally we generate the summary of
amounts for each customer grouping now
note that while in the sequel query we
use the having clause to filter the
output in pic we using another filter so
we using the filter first to filter by
city and then after generating our
summaries we using another filter to
filter out those summaries and we using
the order command to sort o output and
the dump command to return the results
to the screen in this case we are
outputting to the screen instead of
writing data to HDFS which would be
another option one important stylistic
difference to keep in mind here as we
mentioned before sequel is a crazy
language while pig is a procedural
step-by-step style and equal with equal
you basically define the result set that
you want returned and it is the job of
the optimizer to figure out how best to
to return the data in what operations to
execute with pig on the other hand we
are explicitly stating the operations
that we want performed
now where can you give some sample data
for big development well fuel what's the
places few examples here and you can use
a book library you can use Wikipedia
database public data available in the
Amazon Cloud another good source would
be the national climate data at the CDC
so next we'll look at a specific command
moderations first we'll look eps loading
and the storing data loading refers to
loading relations from the files to the
pig buffer and this is done with the
load keyword if you recall in the
example of the script that we just
looked at we executed to load command 11
for the customer file and the other one
for the sales file store command is used
to store the output of our analysis to
HDFS another alternative would be to use
the dump command that's returned data to
the screen so with the it's store
command it it typically will be followed
by the directory where you want to write
the data to on the HDFS
next let's look at how pic script is
being interpreted by the engine so the
four steps are first syntax is validated
semantics of validated error check is
executed second step type check with
schema is performed third references
were validated and finally the statement
is executed the diagram on the right
shows a few points where optimization is
performed note that there are several
steps along the way where optimization
is performed and as the result the
execution plan is submitted before
MapReduce jobs are generated and after
our next below cat relations performed
by developers working the script with
elliptic scripts first less them and
next we'll examine each one of them and
be in detail so those decorations are
filtering transforming drooping sorting
and combining in the next few slides
we'll look closely at each of them
transforming that refers to operation
similar to the transformation that you
perform is equal would be appending the
strings together or splitting them or
any other common data transformation
filtering that criticizes
self-explanatory that refers to
disregarding the certain records based
on the condition grouping them similar
to equals group by clause basically we
are compressing the data based on it
specified set of columns sorting similar
two sequels order by Clause performed
with the order command in Pig combining
now combining is executed with the Union
keyword and it is similar two sequels
Union all command note that unlike a
union
and in sequel combining does not filter
out the duplicates it will just append
the two data sets so combining is
similar to union all in sequel and
splitting which is similar if we should
basically opposite to combining you
perform splitting when you want to
petition your data next we'll go over
several pit collapse load which we
already looked at refers to loading data
from the file system store write data to
HDFS another option would be dumped
where you will drive the output to
scream for each applies the expression
to each record now now that the output
of for each can be one or more records
but filter a command to disregard
certain rows based on their condition
group or cool group they command you use
to compress the data based on the
specified set of columns join joins two
more steps based on the shared key order
performs a sort based on specified heat
distinct removes duplicates records that
union the parents two data sets without
removing duplicates split the opposite
of the union its petition till they
accept stream now
that allows you to direct the output of
a big script to to any other program
that you have defined and limit limit
the number of records to return so
takeaways let's go over the key points
that we have learned in this class peg
is a high level data flow scripting
language and has two major component
runtime engine and pig latin language so
the pig latin language that your
interface with the runtime engine which
execute the script head runs into
execution mode local and MapReduce local
mode runs on your local machine while in
MapReduce mode pig script is converted
to MapReduce jobs that have been
submitted to reduce this as they mowed
your wealth most often been working with
peg engine can be installed by
downloading the mirror web link from the
apology that work website so to figure
out aperture that work that's like a
normal leg engine three parameters need
to be followed before setting the
environment for piglet now first ensure
that also dupe services are running
properly so your cluster has to be up
and running second make sure it pig is
completely installed and configured and
third all required data sets should be
uploaded to HDFS to make sure that the
load command will find your new source