Skip to content

Commit 2c324d3

Browse files
tnachenAndrew Or
tnachen
authored and
Andrew Or
committed
[SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode.
Adding more documentation about submitting jobs with mesos cluster mode. Author: Timothy Chen <tnachen@gmail.com> Closes apache#10086 from tnachen/mesos_supervise_docs. (cherry picked from commit c2de99a) Signed-off-by: Andrew Or <andrew@databricks.com>
1 parent 9e4ac56 commit 2c324d3

File tree

2 files changed

+35
-6
lines changed

2 files changed

+35
-6
lines changed

‎docs/running-on-mesos.md

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -150,14 +150,30 @@ it does not need to be redundantly passed in as a system property.
150150
Spark on Mesos also supports cluster mode, where the driver is launched in the cluster and the client
151151
can find the results of the driver from the Mesos Web UI.
152152

153-
To use cluster mode, you must start the MesosClusterDispatcher in your cluster via the `sbin/start-mesos-dispatcher.sh` script,
154-
passing in the Mesos master url (e.g: mesos://host:5050).
153+
To use cluster mode, you must start the `MesosClusterDispatcher` in your cluster via the `sbin/start-mesos-dispatcher.sh` script,
154+
passing in the Mesos master URL (e.g: mesos://host:5050). This starts the `MesosClusterDispatcher` as a daemon running on the host.
155155

156-
From the client, you can submit a job to Mesos cluster by running `spark-submit` and specifying the master url
157-
to the url of the MesosClusterDispatcher (e.g: mesos://dispatcher:7077). You can view driver statuses on the
156+
If you like to run the `MesosClusterDispatcher` with Marathon, you need to run the `MesosClusterDispatcher` in the foreground (i.e: `bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher`).
157+
158+
From the client, you can submit a job to Mesos cluster by running `spark-submit` and specifying the master URL
159+
to the URL of the `MesosClusterDispatcher` (e.g: mesos://dispatcher:7077). You can view driver statuses on the
158160
Spark cluster Web UI.
159161

160-
Note that jars or python files that are passed to spark-submit should be URIs reachable by Mesos slaves.
162+
For example:
163+
{% highlight bash %}
164+
./bin/spark-submit \
165+
--class org.apache.spark.examples.SparkPi \
166+
--master mesos://207.184.161.138:7077 \
167+
--deploy-mode cluster
168+
--supervise
169+
--executor-memory 20G \
170+
--total-executor-cores 100 \
171+
http://path/to/examples.jar \
172+
1000
173+
{% endhighlight %}
174+
175+
176+
Note that jars or python files that are passed to spark-submit should be URIs reachable by Mesos slaves, as the Spark driver doesn't automatically upload local jars.
161177

162178
# Mesos Run Modes
163179

‎docs/submitting-applications.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,18 @@ export HADOOP_CONF_DIR=XXX
115115
--master spark://207.184.161.138:7077 \
116116
examples/src/main/python/pi.py \
117117
1000
118+
119+
# Run on a Mesos cluster in cluster deploy mode with supervise
120+
./bin/spark-submit \
121+
--class org.apache.spark.examples.SparkPi \
122+
--master mesos://207.184.161.138:7077 \
123+
--deploy-mode cluster
124+
--supervise
125+
--executor-memory 20G \
126+
--total-executor-cores 100 \
127+
http://path/to/examples.jar \
128+
1000
129+
118130
{% endhighlight %}
119131

120132
# Master URLs
@@ -132,9 +144,10 @@ The master URL passed to Spark can be in one of the following formats:
132144
<tr><td> <code>mesos://HOST:PORT</code> </td><td> Connect to the given <a href="running-on-mesos.html">Mesos</a> cluster.
133145
The port must be whichever one your is configured to use, which is 5050 by default.
134146
Or, for a Mesos cluster using ZooKeeper, use <code>mesos://zk://...</code>.
147+
To submit with <code>--deploy-mode cluster</code>, the HOST:PORT should be configured to connect to the <a href="running-on-mesos.html#cluster-mode">MesosClusterDispatcher</a>.
135148
</td></tr>
136149
<tr><td> <code>yarn</code> </td><td> Connect to a <a href="running-on-yarn.html"> YARN </a> cluster in
137-
<code>client</code> or <code>cluster</code> mode depending on the value of <code>--deploy-mode</code>.
150+
<code>client</code> or <code>cluster</code> mode depending on the value of <code>--deploy-mode</code>.
138151
The cluster location will be found based on the <code>HADOOP_CONF_DIR</code> or <code>YARN_CONF_DIR</code> variable.
139152
</td></tr>
140153
<tr><td> <code>yarn-client</code> </td><td> Equivalent to <code>yarn</code> with <code>--deploy-mode client</code>,

0 commit comments

Comments
 (0)