Hadoop cluster - lite

Build a lightweight hadoop cluster with below mentioned hadoop services. These versions match what Hortonworks HDP 2.3.4 distribution has.

Versions

• Alpine Linux: latest
• Debian: 8.3
• Java: Open JDK 7
• hadoop : 2.7.1
• Zookeeper: 3.4.6
• Hbase: 1.1.2
• Kafka: 0.9.0.1
• Kafka manager: latest
• Storm: 0.10.0
• Phoenix: 4.7.0

Note: Hadoop and storm are built on Debian Linux and all others on Alpine Linux

Prerequsities

• Install Docker
• Install Docker compose

The Docker project is tested on Ubuntu host. There is no guarantee that this will work on Windows.

Building all images

• ./build.sh

This will individually build the above mentioned images. This will take sometime and is required to be done only once.

• Get zookeeper IP address using ```docker inspect --format='{{.NetworkSettings.IPAddress}}' zookeeper````
• Get hbase IP address using ```docker inspect --format='{{.NetworkSettings.IPAddress}}' hbase````
• Add zookeeper and hbase entries in the host /etc/hosts file

Verification

• docker images to see all images created with proper tag version
• docker logs -f __CONTAINER_NAME__ to follow logs for a specific container

If there is inconsistency in the image creation process, run the affected build script only. The build script is in each component directory.

Starting

• ./start.sh (background process) OR
• docker-compose up (foreground process)

Stopping

• docker-compose stop

Web UI

• HBASE
• Storm
• Kafka manager
• Hadoop cluster

READMEs

Please go through below READMEs to get detailed information about each image

• Alpine base README
• Debian base README
• Hadoop README
• Zookeeper README
• Hbase README
• Kafka README
• Storm README

Reference

• https://hub.docker.com/r/wurstmeister/kafka/
• https://hub.docker.com/r/sheepkiller/kafka-manager/
• https://hub.docker.com/r/wurstmeister/storm/
• https://hub.docker.com/r/sequenceiq/hadoop-ubuntu/
• http://sookocheff.com/post/docker/containerizing-zookeeper-a-guided-tour/
• http://www.slideshare.net/fabiofumarola1/8a-setup-hbase
• https://docs.docker.com/engine/userguide/intro/
• https://docs.docker.com/compose/
• https://phoenix.apache.org/