Skip to content

Commit 8935c38

Browse files
nathanmarznathanmarz
committed
move master to 0.9.0
2 parents 446037a + 3e29f13 commit 8935c38

File tree

72 files changed

+1284
-272
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

72 files changed

+1284
-272
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ _release
2121
*.zip
2222
.lein-deps-sum
2323
*.iml
24+
/target
25+
/.project/
26+
/.lein-plugins/
2427
*.ipr
2528
*.iws
2629
.idea

‎CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
## Unreleased (0.9.0)
2+
3+
* All logging now done with slf4j
4+
* Replaced log4j logging system with logback
5+
* Logs are now limited to 1GB per worker (configurable via logging configuration file)
6+
* Build upgraded to leiningen 2.0
7+
* Revamped Trident spout interfaces to support more dynamic spouts, such as a spout who reads from a changing set of brokers
8+
19
## 0.8.2
210

311
* Added backtype.storm.scheduler.IsolationScheduler. This lets you run topologies that are completely isolated at the machine level. Configure Nimbus to isolate certain topologies, and how many machines to give to each of those topologies, with the isolation.scheduler.machines config in Nimbus's storm.yaml. Topologies run on the cluster that are not listed there will share whatever remaining machines there are on the cluster after machines are allocated to the listed topologies.

‎bin/build_release.sh

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,35 @@
11
#!/bin/bash
2+
function quit {
3+
exit 1
4+
}
5+
trap quit 1 2 3 15 #Ctrl+C exits.
26

3-
RELEASE=`cat project.clj | sed '6q;d' | awk '{print $3}' | sed -e 's/\"//' | sed -e 's/\"//'`
7+
RELEASE=`head -1 project.clj | awk '{print $3}' | sed -e 's/\"//' | sed -e 's/\"//'`
8+
LEIN=`which lein2 || which lein`
9+
export LEIN_ROOT=1
410

511
echo Making release $RELEASE
612

713
DIR=_release/storm-$RELEASE
814

915
rm -rf _release
10-
export LEIN_ROOT=1
11-
rm *.zip
12-
rm *jar
13-
rm -rf lib
14-
rm -rf classes
15-
lein deps
16-
lein jar
17-
mkdir -p $DIR
18-
mkdir $DIR/lib
19-
cp storm*jar $DIR/
20-
cp lib/*.jar $DIR/lib
16+
rm -f *.zip
17+
$LEIN with-profile release clean
18+
$LEIN with-profile release deps
19+
$LEIN with-profile release jar
20+
$LEIN with-profile release pom
21+
mvn dependency:copy-dependencies
22+
23+
mkdir -p $DIR/lib
24+
cp target/storm-*.jar $DIR/storm-${RELEASE}.jar
25+
cp target/dependency/*.jar $DIR/lib
2126
cp CHANGELOG.md $DIR/
2227

2328
echo $RELEASE > $DIR/RELEASE
2429

25-
cp -R log4j $DIR/
26-
mkdir $DIR/logs
30+
mkdir -p $DIR/logback
31+
mkdir -p $DIR/logs
32+
cp -R logback/cluster.xml $DIR/logback/cluster.xml
2733

2834
mkdir $DIR/conf
2935
cp conf/storm.yaml.example $DIR/conf/storm.yaml

‎bin/storm

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -243,10 +243,10 @@ def nimbus(klass="backtype.storm.daemon.nimbus"):
243243
See Setting up a Storm cluster for more information.
244244
(https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
245245
"""
246-
cppaths = [STORM_DIR + "/log4j", STORM_DIR + "/conf"]
246+
cppaths = [STORM_DIR + "/conf"]
247247
jvmopts = parse_args(confvalue("nimbus.childopts", cppaths)) + [
248248
"-Dlogfile.name=nimbus.log",
249-
"-Dlog4j.configuration=storm.log.properties",
249+
"-Dlogback.configurationFile=" + STORM_DIR + "/logback/cluster.xml",
250250
]
251251
exec_storm_class(
252252
klass,
@@ -263,10 +263,10 @@ def supervisor(klass="backtype.storm.daemon.supervisor"):
263263
See Setting up a Storm cluster for more information.
264264
(https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
265265
"""
266-
cppaths = [STORM_DIR + "/log4j", STORM_DIR + "/conf"]
266+
cppaths = [STORM_DIR + "/conf"]
267267
jvmopts = parse_args(confvalue("supervisor.childopts", cppaths)) + [
268268
"-Dlogfile.name=supervisor.log",
269-
"-Dlog4j.configuration=storm.log.properties",
269+
"-Dlogback.configurationFile=" + STORM_DIR + "/logback/cluster.xml",
270270
]
271271
exec_storm_class(
272272
klass,
@@ -284,16 +284,16 @@ def ui():
284284
See Setting up a Storm cluster for more information.
285285
(https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
286286
"""
287-
cppaths = [STORM_DIR + "/log4j", STORM_DIR + "/conf"]
287+
cppaths = [STORM_DIR + "/conf"]
288288
jvmopts = parse_args(confvalue("ui.childopts", cppaths)) + [
289289
"-Dlogfile.name=ui.log",
290-
"-Dlog4j.configuration=storm.log.properties",
290+
"-Dlogback.configurationFile=" + STORM_DIR + "/logback/cluster.xml",
291291
]
292292
exec_storm_class(
293293
"backtype.storm.ui.core",
294294
jvmtype="-server",
295295
jvmopts=jvmopts,
296-
extrajars=[STORM_DIR + "/log4j", STORM_DIR, STORM_DIR + "/conf"])
296+
extrajars=[STORM_DIR, STORM_DIR + "/conf"])
297297

298298
def drpc():
299299
"""Syntax: [storm drpc]
@@ -304,12 +304,15 @@ def drpc():
304304
See Distributed RPC for more information.
305305
(https://github.com/nathanmarz/storm/wiki/Distributed-RPC)
306306
"""
307-
jvmopts = ["-Xmx768m", "-Dlogfile.name=drpc.log", "-Dlog4j.configuration=storm.log.properties"]
307+
jvmopts = ["-Xmx768m",
308+
"-Dlogfile.name=drpc.log",
309+
"-Dlogback.configurationFile=" + STORM_DIR + "/logback/cluster.xml"
310+
]
308311
exec_storm_class(
309312
"backtype.storm.daemon.drpc",
310313
jvmtype="-server",
311314
jvmopts=jvmopts,
312-
extrajars=[STORM_DIR + "/log4j", STORM_DIR + "/conf"])
315+
extrajars=[STORM_DIR + "/conf"])
313316

314317
def dev_zookeeper():
315318
"""Syntax: [storm dev-zookeeper]
@@ -318,11 +321,11 @@ def dev_zookeeper():
318321
"storm.zookeeper.port" as its port. This is only intended for development/testing, the
319322
Zookeeper instance launched is not configured to be used in production.
320323
"""
321-
cppaths = [STORM_DIR + "/log4j", STORM_DIR + "/conf"]
324+
cppaths = [STORM_DIR + "/conf"]
322325
exec_storm_class(
323326
"backtype.storm.command.dev_zookeeper",
324327
jvmtype="-server",
325-
extrajars=[STORM_DIR + "/log4j", STORM_DIR + "/conf"])
328+
extrajars=[STORM_DIR + "/conf"])
326329

327330
def version():
328331
"""Syntax: [storm version]

‎bin/to_maven.sh

Lines changed: 24 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,27 @@
1-
#!/bin/bash
1+
#!/bin/bash
2+
function quit {
3+
exit 1
4+
}
5+
trap quit 1 2 3 15 #Ctrl+C exits.
26

3-
RELEASE=`cat project.clj | sed '6q;d' | awk '{print $3}' | sed -e 's/\"//' | sed -e 's/\"//'`
7+
RELEASE=`head -1 project.clj | awk '{print $3}' | sed -e 's/\"//' | sed -e 's/\"//'`
8+
LEIN=`which lein2 || which lein`
49

5-
rm -rf classes
6-
rm *jar
7-
rm *xml
8-
lein jar
9-
lein pom
10-
scp storm*jar pom.xml clojars@clojars.org:
10+
echo ==== Storm Jar ====
11+
$LEIN with-profile release clean
12+
$LEIN with-profile release jar
13+
$LEIN with-profile release pom
14+
scp target/storm*jar pom.xml clojars@clojars.org:
15+
rm -Rf target *.xml
1116

12-
rm *jar
13-
rm -rf classes
14-
rm conf/log4j.properties
15-
lein jar
16-
mv pom.xml old-pom.xml
17-
sed 's/artifactId\>storm/artifactId\>storm-lib/g' old-pom.xml > pom.xml
18-
mv storm-$RELEASE.jar storm-lib-$RELEASE.jar
19-
scp storm*jar pom.xml clojars@clojars.org:
20-
rm *xml
21-
rm *jar
22-
git checkout conf/log4j.properties
17+
echo ==== Storm-Lib Jar ====
18+
rm conf/logback.xml
19+
$LEIN with-profile lib clean
20+
$LEIN with-profile lib jar
21+
$LEIN with-profile lib pom
22+
sed -i '' -e 's/artifactId\>storm/artifactId\>storm-lib/g' pom.xml
23+
mv target/storm-$RELEASE.jar target/storm-lib-$RELEASE.jar
24+
scp target/storm*jar pom.xml clojars@clojars.org:
25+
rm -Rf target *.xml
26+
27+
git checkout conf/logback.xml

‎conf/defaults.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ topology.max.task.parallelism: null
8989
topology.max.spout.pending: null
9090
topology.state.synchronization.timeout.secs: 60
9191
topology.stats.sample.rate: 0.05
92+
topology.builtin.metrics.bucket.size.secs: 60
9293
topology.fall.back.on.java.serialization: true
9394
topology.worker.childopts: null
9495
topology.executor.receive.buffer.size: 1024 #batched
@@ -105,5 +106,4 @@ topology.max.error.report.per.interval: 5
105106
topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
106107
topology.trident.batch.emit.interval.millis: 500
107108

108-
109109
dev.zookeeper.path: "/tmp/dev-storm-zookeeper"

‎conf/log4j.properties

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎conf/logback.xml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
3+
<configuration scan="true" scanPeriod="30 seconds">
4+
<appender name="A1" class="ch.qos.logback.core.ConsoleAppender">
5+
<encoder>
6+
<pattern>%-4r [%t] %-5p %c - %m%n</pattern>
7+
</encoder>
8+
</appender>
9+
<logger name="org.apache.zookeeper" level="WARN"/>
10+
<root level="INFO">
11+
<appender-ref ref="A1"/>
12+
</root>
13+
</configuration>

‎conf/storm.yaml.example

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,10 @@
2121
# drpc.servers:
2222
# - "server1"
2323
# - "server2"
24+
25+
## Metrics Consumers
26+
# topology.metrics.consumers.register:
27+
# - class: "org.mycompany.MyMetricsConsumer"
28+
# argument:
29+
# - endpoint: "metrics-collector.mycompany.org"
30+
# parallelism.hint: 1

‎log4j/storm.log.properties

Lines changed: 0 additions & 9 deletions
This file was deleted.

0 commit comments

Comments
 (0)