Apache HADOOP

Open source software framework designed

for storage and processing of large scale
data on clusters of commodity hardware
Created by Doug Cutting and Mike
Carafella in 2005.
Cutting named the program after his son’s
toy elephant.
Uses for Hadoop
Data-intensive text processing
Assembly of large genomes
Graph mining
Machine learning and data mining
Large scale social network analysis
Who Uses Hadoop?
How much data?
Facebook
◦ 500 TB per day
Yahoo
◦ Over 170 PB
eBay
◦ Over 6 PB
Getting the data to the processors becomes
the bottleneck
Requirements for Hadoop
Must support partial failure
 Must be scalable
Partial Failures
 Failure of a single component must not
cause the failure of the entire system only a
degradation of the application performance
 Failure should not result in the loss of
any data
Component Recovery
If a component fails, it should be able to
recover without restarting the entire system
Component failure or recovery during a job
must not affect the final output
Scalability
Increasing resources should increase load
capacity
 Increasing the load on the system should
result in a graceful decline in performance
for all jobs
 Not system failure
The four key characteristics of Hadoop are:
 Economical: Its systems are highly
economical as ordinary computers can be
used for data processing.
 Reliable: It is reliable as it stores copies of
the data on different machines and is
resistant to hardware failure.
 Scalable: It is easily scalable both,
horizontally and vertically. A few extra
nodes help in scaling up the framework.
◦ Flexible: It is flexible and can store as much
structured and unstructured data
Difference between Traditional Database
System and Hadoop
The table given below will help you distinguish between Traditional Database System
and Hadoop.
Traditional Database
Hadoop
System
In Hadoop, the program
goes to the data. It initially
Data is stored in a distributes the data to
central location and sent multiple systems and later
to the processor at runs the computation
runtime. wherever the data is
located.

Hadoop works better when

Traditional Database the data size is big. It can
Systems cannot be used process and store a large
to process and store a amount of data efficiently
significant amount of and effectively.
data(big data).

Traditional RDBMS is Hadoop can process and

used to manage only store a variety of data,
structured and semi-
structured data. It cannot whether it is structured or
be used to control unstructured.
unstructured data.

The Hadoop Ecosystem

Hadoop
Contains Libraries and other modules
Common

HDFS Hadoop Distributed File System

Hadoop YARN Yet Another Resource Negotiator

Hadoop A programming model for large scale
MapReduce data processing

Hadoop ecosystem is continuously growing to meet

the needs of Big Data. It comprises the following
twelve components:
 HDFS(Hadoop Distributed file system)
 HBase
 Sqoop
 Flume
 Spark
 Hadoop MapReduce
 Pig
 Impala
 Hive
 Cloudera Search
 Oozie
 Hue.
 HDFS: Hadoop Distributed File System
 YARN: Yet Another Resource Negotiator
 MapReduce: Programming based Data
Processing
 Spark: In-Memory data processing
 PIG, HIVE: Query based processing of data
services
 HBase: NoSQL Database
 Mahout, Spark MLLib: Machine
Learning algorithm libraries
 Solar, Lucene: Searching and Indexing
 Zookeeper: Managing cluster
 Oozie: Job Scheduling

HDFS (Hadoop Distributed File System):

HDFS is the primary or major component of

Hadoop ecosystem and is responsible for
storing large data sets of structured or
unstructured data across various nodes and
 thereby maintaining the metadata in the form
of log files.

 Hadoop distributed file system (HDFS) is a

Java based file system that provides scalable,
fault tolerance, reliable and cost efficient data
storage for Big data.

HDFS is a distributed file system that runs on

Commodity hardware.

Hadoop interact directly with HDFS by shell-

like commands.

HDFS consists of two core components i.e.

1. Name node
2. Data Node

HDFS uses a master/slave architecture in which

one device (master) termed as NameNode controls
one or more other devices (slaves) termed as
DataNode.
◦ It breaks Data/Files into small blocks
(Typically 64MB or 128MB each block) and
stores on DataNode and each block
replicates on other nodes to accomplish fault
tolerance.

 NameNode keeps the track of blocks written

to the DataNode.

i. NameNode
It is also known as Master node. NameNode
does not store actual data or dataset. NameNode
stores Metadata i.e. number of blocks, their location,
on which Rack, which Datanode the data is stored
and other details. It consists of files and directories.

Tasks of HDFS NameNode

 Manage file system namespace.

 Regulates client’s access to files.

 Executes file system execution such as naming,

closing, opening files and directories.

ii. DataNode

It is also known as Slave.

HDFS Datanode is responsible for storing actual
data in HDFS.
Datanode performs read and write
operation as per the request of the clients.
Replica block of Datanode consists of 2 files on
the file system.
 The first file is for data and second file is
for recording the block’s metadata.
HDFS Metadata includes checksums for data.

At startup, each Datanode connects to its

corresponding Namenode and does
handshaking. Verification of namespace ID and
software version of DataNode take place by
handshaking. At the time of mismatch found,
DataNode goes down automatically.

Tasks of HDFS DataNode

 DataNode performs operations like block replica

creation, deletion, and replication according to the

instruction of NameNode.
 DataNode manages data storage of the system.

YARN(Yet Another Resource Negotiator):

 As the name implies, YARN is the one who helps

to manage the resources across the clusters. In

short, it performs scheduling and resource
allocation for the Hadoop System.

 Consists of three major components i.e.

1.Resource Manager
2.Nodes Manager
3.Application Manager

 Resource manager has the privilege of allocating

resources for the applications in a system whereas
 Node managers work on the allocation of
resources such as CPU,s memory, bandwidth per
machine and later on acknowledges the resource
manager.
 Application manager works as an interface
between the resource manager and node manager
and performs negotiations as per the requirement
of the two.
MapReduce
Hadoop MapReduce is the core Hadoop ecosystem
component which provides data processing.

MapReduce is a software framework for easily

writing applications that process the vast amount of
structured and unstructured data stored in the
Hadoop Distributed File system.

MapReduce programs are parallel in nature, thus are

very useful for performing large-scale data analysis
using multiple machines in the cluster. Thus, it
improves the speed and reliability of cluster this
parallel processing.
Working of MapReduce
Hadoop Ecosystem component ‘MapReduce’ works
by breaking the processing into two phases:
 Map phase
 Reduce phase

Each phase has key-value pairs as input and output.

In addition, programmer also specifies two
functions: map function and reduce function
1.Map() performs sorting and filtering of data and
thereby organizing them in the form of group.
Map generates a key-value pair based result
which is later on processed by the Reduce()
method.
2.Reduce(), as the name suggests does the
summarization by aggregating the mapped data.
In simple, Reduce() takes the output generated by
Map() as input and combines those tuples into
smaller set of tuples.

The Mapper
 Reads data as key/value pairs
◦ The key is often discarded
 Outputs zero or more key/value pairs
Shuffle and Sort
 Output from the mapper is sorted by key
 All values with the same key are guaranteed to go to
the same machine
The Reducer
 Called once for each unique key
  Gets a list of all values associated with a key as input
 The reducer outputs zero or more final key/value
pairs
◦ Usually just one output per input key
MapReduce: Word Count

Features of MapReduce
 Simplicity – MapReduce jobs are easy to run.

Applications can be written in any language such

as java, C++, and python.
 Scalability – MapReduce can process petabytes
of data.
 Speed – By means of parallel processing
problems that take days to solve, it is solved in
hours and minutes by MapReduce.
 Fault Tolerance – MapReduce takes care of
failures. If one copy of data is unavailable,
another machine has a copy of the same key pair
which can be used for solving the same subtask.

Other Tools
 Hive
◦ Hadoop processing with SQL
 Pig
◦ Hadoop processing with scripting
 Cascading
◦ Pipe and Filter processing model
 HBase
◦ Database model built on top of Hadoop
 Flume
 ◦ Designed for large scale data movement
Matrix Multiplication
https://www.youtube.com/watch?v=RIMA4rvNpI8