Spark on Yarn initial commit

akanto · akanto · commit a1f672378040 · 2014-07-11T18:38:50.000+02:00
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1 @@
+.project
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,11 @@
+FROM sequenceiq/hadoop-docker:2.4.1
+MAINTAINER SequenceIQ
+
+RUN curl -s https://s3-eu-west-1.amazonaws.com/seq-spark/spark-v1.0.1-rc2.tar.gz | tar -xz -C /usr/local/
+RUN cd /usr/local && ln -s spark-v1.0.1-rc2 spark
+RUN $BOOTSTRAP && $HADOOP_PREFIX/bin/hadoop dfsadmin -safemode leave && $HADOOP_PREFIX/bin/hdfs dfs -put /usr/local/spark/assembly/target/scala-2.10 /spark
+
+ENV YARN_CONF_DIR $HADOOP_PREFIX/etc/hadoop
+ENV SPARK_JAR hdfs:///spark/spark-assembly-1.0.1-hadoop2.4.1.jar
+
+CMD ["/etc/bootstrap.sh", "-d"]
diff --git a/README.md b/README.md
@@ -0,0 +1,49 @@
+Apache Spark on Docker
+==========
+
+This repository contains a docker file to build a docker image with Apache Spark. This docker image depends on our previous Hadoop docker image, available at the SequenceIQ GitHub page.
+The base Hadoop docker image is also available as an official Docker image (sequenceiq/hadoop-docker).
+
+## Building the image
+```
+docker build --rm -t sequenceiq/spark .
+```
+
+## Running the image
+```
+docker run -i -t -h sandbox sequenceiq/spark /etc/bootstrap.sh -bash
+```
+
+## Versions
+Hadoop 2.4.1 and Apache Spark v1.0.1-rc2
+
+## Testing
+
+You can run one of the stock examples:
+
+```
+cd /usr/local/spark
+# run the spark shell
+./bin/spark-shell --master yarn-client --driver-memory 1g --executor-memory 1g --executor-cores 1
+
+# execute the the following command which should return 1000
+scala> sc.parallelize(1 to 1000).count()
+```
+
+There are two deploy modes that can be used to launch Spark applications on YARN. In yarn-cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
+
+Estimating Pi (yarn-cluster mode): :
+```
+cd /usr/local/spark
+
+# execute the the following command which should write the "Pi is roughly 3.1418" into the logs
+./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 examples/target/scala-2.10/spark-examples_2.10-1.0.1.jar
+```
+
+Estimating Pi (yarn-client mode):
+```
+cd /usr/local/spark
+
+# execute the the following command which should print the "Pi is roughly 3.1418" to the screen
+./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 1g --executor-memory 1g --executor-cores 1 examples/target/scala-2.10/spark-examples_2.10-1.0.1.jar
+```
