Big Data Hadoop Tutorial For Beginners | What Is Hadoop? | What is Big Data? | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
data is critical to organizations for
the immense value that it offers in the
past organizations have considered data
an economical and thus have used
solutions which limited their
capabilities to derive the value from
data in modern times technologies have
advanced businesses have become more
dynamic and organizations now want to
derive every bit of value from the data
they always had it is important for them
to identify the data that needs to be
captured and stored with the increase in
data volume variety and velocity there
is much needed change in the approach to
capture manage process and visualize
data the need to aggregate and analyze
this data to derive hidden insights has
grown with exponential data growth at an
exponential rate the capability to do
this has become the basis of competition
and the success that follows Google's
director of research Peter Norvig
has well said we don't have better
algorithms we just have more data the
graph here depicts the worldwide
installed raw data storage capability
that has taken a leap from 2012 to 2017
the usefulness of this unprecedented but
available massive data depends on the
ability of organizations to prioritize
capture store and retrieve data easily
as they intend to leverage the value it
provides the desire to do this also
brings with it the associated risks in
earlier times organizations use
traditional solutions such as our DBMS
inexpensive tapes and storage solutions
for long term data archiving today these
solutions are replaced with solutions
that offer capabilities to store more
data online and they allow streaming and
support agility and dynamism such as Big
Data technologies and no sequel data
stores
mobile devices online sources electronic
communication and instrumented
machineries can be considered data
generators data was generated earlier as
well
however organizations now seek to glean
intelligence from the available data and
translate that into business advantage
the digitization of business activities
newer sources of information and
probable use of cheaper equipment bring
us into a new era replace or be replaced
the recent explosion of digital data and
organizations interest in data has
positioned us at the beginning of a new
revolution that will touch every
business and change the way businesses
work initially data was being
categorized as big data when it had
characteristics of high volume velocity
and variety later other V's were added
to the list of characteristics veracity
viscosity morality volatility and
validity big data is a term coined to
describe large or complex datasets
traditional data processing solutions
are inadequate to analyze this data and
fail to capture store analyse cure
search share transfer visualise query
process and handle data benefits of big
data are it offers competitive edge and
advantage data has emerged as the
world's new measure of success and
provides competitive advantage when
leveraged to the core of its
capabilities it improves decision-making
effective and precise decision-making is
possible when more data is considered
decision-making is moving from the elite
few to the empowered many it identifies
the value of data as data gets generated
with high velocity and variety existing
solutions start showing weaknesses to
catch up and deliver the real value of
data for such instances big data
solutions offer promising capabilities
let's now look at a few interesting
facts and statistics about Big Data
according to the International Data
Corporation or IDC worldwide revenues
for big data and business analytics will
grow nearly from one hundred twenty two
billion dollars in 2015 to more than 187
billion dollars in 2019 organizations
able to take advantage of the new
generation of business analytic solution
can leverage digital transformation to
adapt disruptive changes and to create
competitive differentiation in their
markets said dan Bassett IDC group
vice-president analytics and information
management he added these organizations
don't just automate existing processes
they treat data and information as they
would any valued asset by using a
focused approach to extracting and
developing the value and utility of
information there is little question
that big data and analytics can have a
considerable impact on just about every
industry said Jessica Gope firt program
director for ID C's customer insights
and analysis group its promise speaks to
the pressure to improve margins and
performance while simultaneously
enhancing responsiveness and delighting
customers and prospects forward-thinking
organizations turn to this technology
for better and faster data-driven
decisions she added now let's look at
some statistics and understand why big
data is a challenge and how it also
brings bigger opportunities fifty-seven
point six percent of organizations
surveyed say that big data is a
challenge seventy two point seven
percent consider driving operational
efficiencies to be the biggest benefit
of a big data strategy fifty percent say
that big data helps in meeting customer
demands and facilitating growth so far
we have discussed the emerging
importance of big data and the change
that it brings to data utilization now
let's look at a few infographics that
showcases the four V's of big data and
the statistics of big data volume refers
to the scale of data it is estimated
that by 2020 the digital universe will
equal 4 D Zeta bytes the amount of data
created every day is
summated to be 2.5 quintillion bytes
velocity refers to the analysis of
streaming data or data that needs to be
analyzed quickly digest these numbers
the new york stock exchange generates
about one terabyte of information in a
single trading session variety refers to
different forms of data consider this 30
billion pieces of content are shared on
facebook in a month 4 billion hours of
video are watched on YouTube in a month
400 million tweets are sent per day by
about 200 million active users veracity
refers to the uncertainty of data
consider this in a year the US economy
loses three point one trillion dollars
because data quality is poor in a survey
27 percent of the respondents said that
they were unsure about the accuracy of
the data they had in the upcoming
screens you will observe some
statistical records that emphasize about
the use of big data if success in
various industries and its future growth
the key questions that enterprises ask
about Big Data how to store and protect
Big Data how to backup and restore Big
Data how to organize and catalog the
data that you have backed up how to keep
costs down while ensuring that all the
critical data is available when you need
it
there are several volumes of data that
generate from different social media
websites like Facebook
Zynga YouTube lol and Twitter 30 billion
pieces of content were added to Facebook
this past month by 600 million plus
users Zynga processes 1 petabyte of
content for players every day a volume
of data that is unmatched in the social
game industry big data and analytics is
the core of next-gen manufacturing with
more asia-pacific manufacturers looking
at making better and faster business
decisions based on evidence Big Data
technologies can help the manufacturers
to analyze larger volumes of data from a
variety of sources and deliver the
analysis at a greater velocity IDC
observes that regional manufacturers are
taking a cautious approach and the use
of analytic tools today is basic here is
the 2013 landscape in asia-pacific big
data is proving a great boon to the
healthcare industry as well
62% organizations say that big data will
significantly improve patient care in
military healthcare systems 59% believe
in five years
fulfilling their agency's mission and
objectives will depend on successful
leveraging of big data our agency is
taking the steps to prepare so what is
the big picture
since 2014 there has been a 125 percent
increase in the number of organizations
that have implemented or deployed
data-driven projects there is a
significant adoption seen across all
organizations take a look at the
infographic to know the timeline for big
data plans the statistics here gives the
insight into how big data budgets will
change the future business goals driving
investments managing
current types of data let's take a look
at the statistics highlighting the top
big data challenges for 2015 and
overcoming challenges with IT staff and
vendors the infographic here explains
the success story of EMC the primary
sponsor of the human face of big data
project EMC is powering dramatic new
insights into our lives organizations
such as LinkedIn eBay Silver Spring and
many others use EMC storage EMC stores
protect and they analyze data with the
help of big data let's look at another
success story of big data users
companies are using Hadoop for analytics
to uncover new meaning in satellite
imagery Hadoop allows industries to gain
new insights from geospatial data
satellite imaging data contains huge
amounts of information images to be
processed and indexed make it difficult
for users to search through databases
for useful information skybox imaging
works to apply analytics to satellite
images to leverage the data within the
company builds and operates high
resolution imaging satellites and makes
data with images available for customers
this image data can tell users the speed
of a ship carrying containers as it
moves or available parking space in a
parking lot or percentage of completion
in a construction site this could be a
challenge that the data output is raw
sensory data and not easily recognizable
skybox partnered with cloud era to build
its own distribution of hadoop with this
skybox customers can embed their
algorithms in the company's platform and
use analytics engine to crunch data for
their own use cases by studying and
looking for changes in imaging data
agriculture clients can monitor crop
yields shipping and supply chain
companies can monitor vehicles and oil
or gas companies can evaluate land areas
implementation of Hadoop into image
databases enables users to assemble data
normalize it index it and make
meaningful connections here is a list of
some more customers that have adopted
Big Data and Hadoop
technologies to empower big data
applications to transform their
businesses Caesars Entertainment casino
entertainment company Caesars
Entertainment uses a new Hadoop
environment built on CDH cloud era
distribution of Hadoop that runs on
Intel Xeon processor ePHI family to
identify customer segments and create
specific marketing campaigns aligned to
these segments the new environment has
reduced processing times for jobs from 6
hours
to 45 minutes this allows the Caesars to
carry on faster and precise data
analysis to enhance customer experiences
and improve security
Cerner healthcare technology company
Cerner has built an enterprise hub
powered by CDH on Intel Xeon processors
to create a more comprehensive view of
any patient conditions or trends this
helps the company and its clients to
monitor 1 million patients daily it also
helps them to determine the probability
of patients who have a potential fatal
bloodstream infection sepsis etc with
higher accuracy MasterCard MasterCard
was the first company to implement CDH
farm logs farm management software
company farm logs uses real-time
analytics that run on Intel Xeon
processor ePHI family to provide
actionable data on harvest growing
conditions and vegetable health this
helps farmers increase the productivity
now that you have learned about the
importance of big data and also
understood from the statistics that big
data is not just an emerging fad let's
now look at the solution and the
framework which allows us to work on Big
Data Apache Hadoop Apache Hadoop is an
open source software framework from the
open source community
Apache org various vendor specific
distributions also exist in market such
as cloud era distribution of Hadoop
Hortonworks data platform IBM big
insight and map are etc Apache Hadoop
popularly known as Hadoop allows
distributed storage horizontal
scalability to strip
and parallel processing of large
datasets on commodity machines which
form a cluster Apache Hadoop consists of
a storage layer a dupe distributed file
system processing layer and MapReduce we
will learn more about HDFS MapReduce in
the subsequent lessons Apache Hadoop
comprises of an ecosystem that is a
collection of open source components for
software packages such as Apache hive
HBase flume Pig spark zookeeper scoop
Kafka storm and so on the components or
packages can be installed on top of
Hadoop which can leverage the benefits
of distributed file system and
distributed processing these components
or packages change the way Enterprise
store process and analyze data you can
see here a Hadoop ecosystem and how
components are layered to work together
in an ecosystem the list of software
components continues to grow as the
business needs grow leading to
innovative solutions let's now learn
about the modules that comprise that a
dupe framework the base Apache Hadoop
framework consists of the following
modules a dupe common consists of
libraries and utilities needed by other
Hadoop modules Hadoop distributed file
system a distributed file system that
stores data on commodity machines Hadoop
yarn a resource management platform for
managing computing resources in cluster
and using them for scheduling and
processing user applications hadoop
mapreduce a programming model for
large-scale distributed and parallel
data processing other open source
components packages as already discussed
apache hadoop consists of two logical
layers storage layer processing layer
both the logical layers contain master
and slave daemons which take care of
storage and processing Hadoop
distributed file system or HDFS is a
distributed file system which holds the
data for entire cluster although it
actually stores the data on underlying
nodes that
the data nodes we will learn more about
HDFS and working of HDFS in the next
lesson the storage layer consists of
named nodes data nodes and secondary
name node the illustration here shows a
complete picture of a Hadoop cluster
although here we won't mention any other
components or packages that can be part
of a cluster
ññ name node VN data node s n n
secondary name node RM /jt resource
manager or job tracker n n / TT node
manager or tasktracker m map our reduce
m and r map and reduce HDFS a dupe
distributed file system Hey want to
become an expert in Big Data then
subscribe to the simply learned channel
and click here to watch more such videos
centered up and get certified in Big
Data click here