Hadoop HDFS Tutorial For Beginners | What Is HDFS In Hadoop | Hadoop Training | Simplilearn Video Lecture | Taming the Big Data with HAdoop and MapReduce - Software Development

[Music]
HDFS is a distributed file system that
provides access to data across Hadoop
clusters a cluster is a group of
computers that work together like other
Hadoop related technologies HDFS is a
key tool that manages and supports
analysis of very large volumes petabytes
and zettabytes of data before 2011
storing and retrieving petabytes or Zeta
bytes of data had the following three
major challenges cost speed reliability
traditional file systems approximately
cost ten to fourteen thousand dollars
per terabyte searching and analyzing
data was also time-consuming and
expensive
also if search components were saved in
different servers fetching data was
difficult let's discuss how HDFS
resolves all of the three major issues
of traditional file systems costs HDFS
is an open source software so it can be
used with zero licensing and support
costs it is designed to run on a regular
computer speed large Hadoop clusters can
read or write more than a terabyte of
data per second a cluster comprises
multiple systems logically
interconnected in the same network HDFS
can easily deliver more than two
gigabits of data per second per computer
to MapReduce which is a data processing
framework of Hadoop reliability HDFS
copies the data multiple times and
distributes the copies to individual
nodes a node is a commodity server which
is interconnected through a network
device HDFS then places at least one
copy of data on a different server in
case any of the data is deleted from the
nodes it can be found within the cluster
a regular file system like a Linux file
system is different from HDFS with
respect to the size of the data in a
regular file system each block of data
is small usually about 51
however in hdfs each block is of 128
megabytes by default a regular file
system provides access to large data but
may suffer from disk input/output
problems mainly due to multiple seek
operations on the other hand HDFS can
read large quantities of data
sequentially after a single seek
operation this makes HDFS unique since
all of these operations are performed in
a distributed mode let's list the
characteristics of HDFS HDFS has a high
fault tolerance and HDFS may consist of
thousands of server machines each
machine stores a part of the file
systems data HDFS directs faults that
can occur on any of the machines and
recovers it quickly and automatically
HDFS has a high throughput HDFS is
designed to store and scan millions of
rows of data and to counter add some
subsets of the data the time required in
this process is dependent on the
complexities involved it has been
designed to support large datasets in
batch style jobs however the emphasis is
on high throughput of data access rather
than low latency HDFS is economical HDFS
is designed in such a way that it can be
built on commodity hardware and
heterogeneous platforms which is low
priced and easily available similar to
the example explained on the previous
screen HDFS stores files in a number of
blocks each block is replicated to a few
separate computers the count of
replication can be modified by the
administrator data is divided into 128
megabytes per block and replicated
across local disks of cluster nodes
metadata controls the physical location
of a block and its replication within
the cluster it is stored in name node
HDFS is the storage system for both
input output of MapReduce jobs let's
understand how HDFS stores files with an
example a patron gifted a collection of
popular books to a college library
the librarian decided to arrange the
books on a small rack and then
distribute multiple copies of each book
on other racks this way the students
could easily pick a book from any of the
racks
similarly HDFS creates multiple copies
of a data block and keeps them in
separate systems for easy access hey
want to become an expert in Big Data
then subscribe to the simply learned
Channel and click here to watch more
such videos to nerd up and get certified
in Big Data click here