Skip to content

Commit 7b54922

Browse files
lrytzlrytz
committed
Documentation and some updates for the release build script
1 parent 5c75f48 commit 7b54922

File tree

2 files changed

+153
-107
lines changed

2 files changed

+153
-107
lines changed

‎build.number

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,13 @@
1-
#Tue Sep 11 19:21:09 CEST 2007
1+
# The version number in this file should be the next un-released minor version,
2+
# e.g., 2.11.7, 2.12.0, 2.12.1. It's used to determine version numbers for
3+
# SNAPSHOT / nightly builds and local builds of source checkouts.
4+
25
version.major=2
36
version.minor=12
47
version.patch=0
5-
# This is the -N part of a version. if it's 0, it's dropped from maven versions.
8+
9+
# This is the -N part of a version (2.9.1-1). If it's 0, it's dropped from maven versions. It should not be used again.
610
version.bnum=0
711

812
# To build a release, see scripts/jobs/scala-release-2.11.x-build
9-
# (normally run by the eponymous job on scala-ci.typesafe.com).
13+
# (normally run by the eponymous job on scala-ci.typesafe.com).

‎scripts/jobs/integrate/bootstrap

Lines changed: 146 additions & 104 deletions
Original file line numberDiff line numberDiff line change
@@ -1,74 +1,98 @@
11
#!/bin/bash -e
2-
# TODO: different scripts for the different phases -- usually we don't need to bootstrap the modules,
3-
# since we can use the previous version of scala for STARR as well as for compiling the modules (assuming it's binary compatible)
4-
# We should move away from the complicated bootstrap and set up our release schedule so we always have a previous build that satisfies these criteria.
5-
# (Potentially trivially, by splitting up this script, and publishing locker as if it were a real release.)
6-
7-
# requirements:
8-
# sbtCmd must point to sbt from sbt-extras (this is the standard on the Scala jenkins, so we only support that one)
9-
# - ~/.sonatype-curl that consists of user = USER:PASS
10-
# - ~/.m2/settings.xml with credentials for sonatype
11-
# <server>
12-
# <id>private-repo</id>
13-
# <username>jenkinside</username>
14-
# <password></password>
15-
# </server>
16-
# - ~/.credentials (for sonatype)
17-
# realm=Sonatype Nexus Repository Manager
18-
# host=oss.sonatype.org
19-
# user=lamp
20-
# password=
21-
# - ~/.credentials-private-repo for private-repo.typesafe.com, as follows:
22-
# realm=Artifactory Realm
23-
# host=private-repo.typesafe.com
24-
# user=jenkinside
25-
# password=
26-
# - ~/.sbt/0.13/plugins/gpg.sbt with:
27-
# addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "0.8.1")
282

29-
# Modus operandi:
30-
#
31-
# Determine Scala version as:
32-
#
33-
# $SCALA_VER_BASE$SCALA_VER_SUFFIX (if former variable is set)
34-
# By parsing the tag (if HEAD is tagged as v$base$suffix)
35-
# By parsing build.number for the base version, suffixing with -$sha-nightly
36-
# Serialize these versions to jenkins.properties, which are passed downstream to scala-release-2.12.x-dist.
37-
# This also removes the need to tag scala/scala-dist (not possible for nightlies, still encouraged for releases, but not a hard requirement).
38-
#
39-
# Determine Module Versions
40-
#
41-
# When running in "versions.properties" mode (the default), derive tags from these versions and build, publishing only those modules that are not available yet.
42-
# Otherwise, build HEAD for all modules, derive a -nightly version for them.
43-
# Bootstrap:
44-
#
45-
# Build minimal core of Scala as this version (aka locker), publish to private-repo
46-
# Build modules required to bootstrap, publish to private-repo
47-
# Build Scala using the previously built core and bootstrap modules, publish to private-repo This overwrites the minimal core on private-repo
48-
# Stage to sonatype (unless building a -nightly release):
3+
# Script Overview
4+
# - determine scala version
5+
# - determine module versions
6+
# - build minimal core (aka locker) of Scala, use the determined version number, publish to private-repo
7+
# - build those modules where a binary compatible version doesn't exist, publish to private-repo
8+
# - build Scala using the previously built core and bootstrap modules, publish to private-repo (overwrites the minimal core version on private-repo)
9+
# - for releases (not nightlies)
10+
# - stage Scala on sonatype
11+
# - rebuild modules that needed a rebuild with this Scala build, and stage them on sonatype
12+
# - for nightlies
13+
# - force rebuild all modules and publish them locally (for testing purposes)
14+
# - the Scala version is serialized to jenkins.properties, which is passed downstream to scala-release jobs
15+
# - this removes the need to tag scala/scala-dist (it's still encouraged for releases, but not a hard requirement)
16+
17+
18+
# Specifying the Scala version:
19+
# - To build a release (this enables publishing to sonatype):
20+
# - Either specify SCALA_VER_BASE. You may also specify SCALA_VER_SUFFIX, the Scala version is SCALA_VER=$SCALA_VER_BASE$SCALA_VER_SUFFIX.
21+
# - Or have the current HEAD tagged as v$base$suffix
22+
# - To prevent staging on sonatype (for testing), set publishToSonatype to anything but "yes"
23+
# - Note: After building a release, the jenkins job provides an updated versions.properties file as artifact.
24+
# Put this file in the Scala repo and create a pull request, and also update the file build.number.
25+
#
26+
# - Otherwise, a nightly release is built:
27+
# - version number is read from the build.number file, extended with -$sha-nightly
28+
29+
30+
# Specifying module versions: there are two modes
31+
# - If moduleVersioning="versions.properties" (default): in this mode we use release versions for the modules.
32+
# - Module versions are read from the versions.properties file.
33+
# - Set <MODULE>_VER to override the default, e.g. XML_VER="1.0.4".
34+
# - The git revision is set to <MODULE>_REF="v$<MODULE>_VER". Make sure the tag exists (you can't override <MODULE>_REF).
4935
#
50-
# Stage this Scala build on sonatype
51-
# Rebuild modules with this Scala build, and stage them on sonatype as well
52-
# This script can be run in multiple modes. It is design to work without any input,
53-
# so that it could be run in Travis CI. In that mode, it'll build a release when
54-
# the current HEAD of the checkout in $WORKSPACE is tagged, and stage to sonatype. Otherwise,
55-
# it'll build a nightly.
36+
# - Otherwise (moduleVersioning has some other value): in this mode we use nightly version nubmers for modules.
37+
# - By default the script sets all <MODULE>_REF to "HEAD", override to build a specific revision.
38+
# - The <MODULE>_VER is set to a nightly version, for example "1.0.3-7-g14888a2-nightly" (you can't override <MODULE>_VER)
39+
40+
41+
# Modules are automatically built if necessary.
42+
# - A module is built if it doesn't exist in the maven repository. Note that the lookup uses two versions:
43+
# - The version of the module (see below how it's determined)
44+
# - The binary version of of the SCALA_VER release that is being built
45+
# - sbt computes the binary version when looking up / building modules (*). Examples:
46+
# - 2.12.0-M1, 2.12.0-RC3: the full version is used
47+
# - 2.12.0, 2.12.1-M1, 2.12.1-RC3, 2.12.1: the binary version 2.12 is used
5648
#
57-
# Since the nightlies are intended to be a drop in replacement, all modules are built with the
58-
# full Scala version as their binary version, so that you can just set scalaVersion to the
59-
# nightly's sha-derived version and be good to go.
49+
# - Example: assume that `scala-xml_2.11 % 1.0.3` and `scala-xml_2.12.0-M1 % 1.0.3` both exists
50+
# - XML_VER=1.0.3 and SCALA_VER=2.11.7 => no rebuild (binary version remains 2.11)
51+
# - XML_VER=1.0.3 and SCALA_VER=2.12.0-M2 => rebuild (new binary version 2.12.0-M2)
52+
# - XML_VER=1.0.4 and SCALA_VER=2.11.7 => rebuild (new version for the module, not yet on maven)
53+
# NOTE: this is not the recommended way of publishing a module. Instead, prefer to release `scala-xml_2.11 % 1.0.4`
54+
# using the existing scala 2.11.6 compiler before releasing 2.11.7. Sometimes it's necessary though. One
55+
# example was 2.11.1, which contained a fix in the backend (SerialVersionUID was ignored). All modules needed
56+
# to be re-built using the 2.11.1 release, we could not use 2.11.0. We could also not release the modules
57+
# after 2.11.1 was out, because that way the scala-library-all pom of 2.11.1 would depend on the old modules.
6058
#
61-
# The other way to trigger a release is by setting the SCALA_VER_BASE env var.
62-
#
63-
# By default, we build the versions of the modules as specified by versions.properties
64-
# (as specified in the HEAD commit). Set moduleVersioning to something random
65-
# to trigger building HEAD of each module, generating a fresh -$(git describe)-nightly version for each.
66-
#
67-
# PS: set publishToSonatype to anything but "yes" to avoid publishing to sonatype
68-
# (publishing only done when $WORKSPACE checkout's HEAD is tagged / SCALA_VER_BASE is set.)
59+
# (*) https://github.com/sbt/sbt/blob/0.13.8/util/cross/src/main/input_sources/CrossVersionUtil.scala#L39
60+
61+
62+
# Binary incompatible changes in Modules: example with Scala 2.11 / 2.12 and scala-xml
63+
# - Simple case: Scala 2.11 and 2.12 use the same scala-xml version, for example v1.0.3
64+
# - Only one branch in the module repostiory. This branch is cross-built against 2.11 and 2.12.
65+
# - This set-up does not allow incompatible changes in scala-xml for Scala 2.12.
66+
# - General case: The scala-xml authors want to introduce incompatible changes for 2.12
67+
# - The scala-xml minor version needs to be increased, so 2.12 would depend on scala-xml v1.1.x (or maybe v2.0.x)
68+
# - The repostiory for scala-xml would have separate branches for each minor version (1.0.x, 1.1.x)
69+
# - This allows building new (binary compatible) versions that target Scala 2.11
70+
# - The Scala release script would require some work, it is currently not prepared to handle branches in module repositories
71+
72+
73+
# Requirements
74+
# - sbtCmd must point to sbt from sbt-extras
75+
# - ~/.sonatype-curl that consists of user = USER:PASS
76+
# - ~/.m2/settings.xml with credentials for sonatype
77+
# <server>
78+
# <id>private-repo</id>
79+
# <username>jenkinside</username>
80+
# <password></password>
81+
# </server>
82+
# - ~/.credentials (for sonatype)
83+
# realm=Sonatype Nexus Repository Manager
84+
# host=oss.sonatype.org
85+
# user=lamp
86+
# password=
87+
# - ~/.credentials-private-repo for private-repo.typesafe.com, as follows:
88+
# realm=Artifactory Realm
89+
# host=private-repo.typesafe.com
90+
# user=jenkinside
91+
# password=
92+
# - ~/.sbt/0.13/plugins/gpg.sbt with:
93+
# addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "0.8.1")
6994

7095

71-
# set to something besides the default to build nightly snapshots of the modules instead of some tagged version
7296
moduleVersioning=${moduleVersioning-"versions.properties"}
7397

7498
publishPrivateTask=${publishPrivateTask-"publish"}
@@ -78,22 +102,13 @@ publishLockerPrivateTask=${publishLockerPrivateTask-$publishPrivateTask} # set t
78102

79103
sbtCmd=${sbtCmd-sbt} # TESTING (this is a marker for defaults to change when testing locally: should be sbtx on my mac)
80104

81-
# 0.13.5 does not respect "set every scalaVersion", see
82-
# https://github.com/scala/scala-parser-combinators/pull/27
83-
sbtCmd="$sbtCmd -sbt-version 0.13.2"
105+
sbtCmd="$sbtCmd -sbt-version 0.13.8"
84106

85107
forceRebuild=${forceRebuild-no}
86108

87-
# publishToSonatype
88-
# set to anything but "yes" to avoid publishing to sonatype
89-
# overridden to "no" when no SCALA_VER_BASE is passed and HEAD is not tagged with a valid version tag
90-
#
91-
92109
antBuildTask="${antBuildTask-nightly}" # TESTING leave empty to avoid the sanity check (don't set it to "init" because ant will croak)
93110
clean="clean" # TESTING leave empty to speed up testing
94111

95-
96-
97112
baseDir=${WORKSPACE-`pwd`}
98113
scriptsDir="$baseDir/scripts"
99114
. $scriptsDir/common
@@ -182,8 +197,8 @@ sbtBuild() {
182197
sbtResolve() {
183198
cd $baseDir/resolutionScratch_
184199
touch build.sbt
185-
# TODO lry: not needed after removing continuations
186-
cross=${4-binary} # Disabled / binary / full
200+
# Can be set to `full` if a module requires cross-versioning against the full Scala version, like the continuations plugin used to.
201+
cross=${4-binary}
187202
echo "### sbtResolve: $sbtCmd $sbtArgs " "${scalaVersionTasks[@]}" "\"$1\" % \"$2\" % \"$3\" cross CrossVersion.$cross"
188203
$sbtCmd $sbtArgs "${scalaVersionTasks[@]}" \
189204
"set libraryDependencies := Seq(\"$1\" % \"$2\" % \"$3\" cross CrossVersion.$cross)" \
@@ -279,16 +294,47 @@ buildModules() {
279294

280295
## BUILD STEPS:
281296

297+
scalaVerToBinary() {
298+
# $1 = SCALA_VER
299+
# $2 = SCALA_VER_BASE
300+
# $3 = SCALA_VER_SUFFIX
301+
302+
local RE='\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)'
303+
local majMin="$(echo $2 | sed -e "s#$RE#\1.\2#")"
304+
local patch="$(echo $2 | sed -e "s#$RE#\3#")"
305+
306+
# The binary version is majMin (e.g. "2.12") if
307+
# - there's no suffix : 2.12.0, 2.12.1
308+
# - the suffix starts with "-bin": 2.12.0-bin-M1
309+
# - the patch version is > 0 : 2.12.1-M1, 1.12.3-RC2, 2.12.1-sha-nightly, 2.12.2-SNAPSHOT
310+
#
311+
# Othwersise, the binary version is the full version: 2.12.0-M1, 2.12.0-RC2, 2.12.0-sha-nightly, 2.12.0-SNAPSHOT
312+
#
313+
# Adapted from sbt: https://github.com/sbt/sbt/blob/0.13.8/util/cross/src/main/input_sources/CrossVersionUtil.scala#L39
314+
#
315+
# Note: during the pre-release cycle of a major release (e.g. before 2.12.0), the SCALA_BINARY_VER of nightly / SNAPSHOT
316+
# versions is the full version, e.g. 2.12.0-sha-nightly, so modules are always re-built. This is in line with what sbt
317+
# does: for example, with scalaVersion := "2.12.0-SNAPSHOT", sbt will resolve scala-xml as scala-xml_2.12.0-SNAPSHOT.
318+
# Once the 2.12.0 release is out, the binary version is 2.12 for all versions (e.g. for 2.12.1-sha-nightly).
319+
320+
if [[ "$3" == "" || "${3:0:4}" == "-bin" || "$patch" != "0" ]]; then
321+
echo "$majMin"
322+
else
323+
echo "$1"
324+
fi
325+
}
326+
282327
determineScalaVersion() {
283328
cd $WORKSPACE
284329
parseScalaProperties "versions.properties"
330+
echo "repo_ref=2.12.x" >> $baseDir/jenkins.properties # for the -dist downstream jobs that build the actual archives
331+
285332

333+
# each of the branches below defines the following vars: SCALA_VER_BASE, SCALA_VER_SUFFIX, SCALADOC_SOURCE_LINKS_VER, publishToSonatype
286334
if [ -z "$SCALA_VER_BASE" ]; then
287335
echo "No SCALA_VER_BASE specified."
288336

289-
scalaTag=$(git describe --exact-match ||:)
290-
291-
SCALA_BINARY_VER=${SCALA_BINARY_VER-"$scala_binary_version"}
337+
scalaTag=$(git describe --tag --exact-match ||:)
292338

293339
if [ -z "$scalaTag" ]
294340
then
@@ -298,9 +344,8 @@ determineScalaVersion() {
298344
SCALA_VER_SUFFIX="-$(git rev-parse --short HEAD)-nightly"
299345
SCALADOC_SOURCE_LINKS_VER=$(git rev-parse HEAD)
300346

301-
# TODO: publish nightly snapshot using this script
347+
# TODO: publish nightly snapshot using this script - currently it's a separate jenkins job still running at EPFL.
302348
publishToSonatype="no"
303-
echo "repo_ref=2.12.x" >> $baseDir/jenkins.properties # for the -dist downstream jobs that build the actual archives
304349
else
305350
echo "HEAD is tagged as $scalaTag."
306351
# borrowed from https://github.com/cloudflare/semver_bash/blob/master/semver.sh
@@ -322,59 +367,56 @@ determineScalaVersion() {
322367
fi
323368

324369
SCALA_VER="$SCALA_VER_BASE$SCALA_VER_SUFFIX"
370+
SCALA_BINARY_VER=$(scalaVerToBinary $SCALA_VER $SCALA_VER_BASE $SCALA_VER_SUFFIX)
371+
325372
echo "version=$SCALA_VER" >> $baseDir/jenkins.properties
326373
echo "sbtDistVersionOverride=-Dproject.version=$SCALA_VER" >> $baseDir/jenkins.properties
327374

328-
# We don't override the scala binary version: when running in -nightly + versions.properties versioning mode,
329-
# we intend to be a drop-in replacement -- all you need to do is change the Scala version
330-
# In order to override this, add 'set every scalaBinaryVersion := "'$SCALA_BINARY_VER'"',
331-
# which, when used with pre-release Scala version numbers, will require tweaking at the sbt usage site as well.
332375
scalaVersionTasks=('set every scalaVersion := "'$SCALA_VER'"')
333376

334377
echo "Building Scala $SCALA_VER."
335378
}
336379

337380
deriveVersion() {
338381
update $1 $2 $3 &> /dev/null
339-
echo "$(git describe --match=v* | cut -dv -f2)-nightly"
382+
echo "$(git describe --tag --match=v* | cut -dv -f2)-nightly"
340383
}
341384

342385
deriveVersionAnyTag() {
343386
update $1 $2 $3 &> /dev/null
344-
echo "$(git describe | cut -dv -f2)-nightly"
387+
echo "$(git describe --tag | cut -dv -f2)-nightly"
345388
}
346389

347390
# determineScalaVersion must have been called
348391
deriveModuleVersions() {
349-
if [ "$moduleVersioning" == "versions.properties" ]
350-
then
351-
# use versions.properties as defaults when no version specified on command line
392+
if [ "$moduleVersioning" == "versions.properties" ]; then
393+
# use versions.properties as defaults when no version specified on the command line
352394
XML_VER=${XML_VER-$scala_xml_version_number}
353395
PARSERS_VER=${PARSERS_VER-$scala_parser_combinators_version_number}
354396
SWING_VER=${SWING_VER-$scala_swing_version_number}
355397
PARTEST_VER=${PARTEST_VER-$partest_version_number}
356398
SCALACHECK_VER=${SCALACHECK_VER-$scalacheck_version_number}
357399

358-
# If a _VER was not specified, the corresponding _REF will be non-empty by now (as specified, or HEAD)
359-
XML_REF=${XML_REF-"v$XML_VER"}
360-
PARSERS_REF=${PARSERS_REF-"v$PARSERS_VER"}
361-
SWING_REF=${SWING_REF-"v$SWING_VER"}
362-
PARTEST_REF=${PARTEST_REF-"v$PARTEST_VER"}
363-
# PARTEST_IFACE_REF=${PARTEST_IFACE_REF-"v$PARTEST_IFACE_VER"}
364-
SCALACHECK_REF=${SCALACHECK_REF-"$SCALACHECK_VER"}
400+
XML_REF="v$XML_VER"
401+
PARSERS_REF="v$PARSERS_VER"
402+
SWING_REF="v$SWING_VER"
403+
PARTEST_REF="v$PARTEST_VER"
404+
# PARTEST_IFACE_REF="v$PARTEST_IFACE_VER"
405+
SCALACHECK_REF="$SCALACHECK_VER" # no `v` in their tags
365406
else
366-
XML_VER=${XML_VER-$(deriveVersion scala scala-xml "$XML_REF")}
367-
PARSERS_VER=${PARSERS_VER-$(deriveVersion scala scala-parser-combinators "$PARSERS_REF")}
368-
SWING_VER=${SWING_VER-$(deriveVersion scala scala-swing "$SWING_REF")}
369-
PARTEST_VER=${PARTEST_VER-$(deriveVersion scala scala-partest "$PARTEST_REF")}
370-
SCALACHECK_VER=${SCALACHECK_VER-$(deriveVersionAnyTag rickynils scalacheck "$SCALACHECK_REF")}
371-
407+
# use HEAD as default when no revision is specified on the command line
372408
XML_REF=${XML_REF-"HEAD"}
373409
PARSERS_REF=${PARSERS_REF-"HEAD"}
374410
SWING_REF=${SWING_REF-"HEAD"}
375411
PARTEST_REF=${PARTEST_REF-"HEAD"}
376-
# PARTEST_IFACE_REF=${PARTEST_IFACE_REF-"HEAD"}
412+
# PARTEST_IFACE_REF=${PARTEST_IFACE_REF-"HEAD"}
377413
SCALACHECK_REF=${SCALACHECK_REF-"HEAD"}
414+
415+
XML_VER=$(deriveVersion scala scala-xml "$XML_REF")
416+
PARSERS_VER=$(deriveVersion scala scala-parser-combinators "$PARSERS_REF")
417+
SWING_VER=$(deriveVersion scala scala-swing "$SWING_REF")
418+
PARTEST_VER=$(deriveVersion scala scala-partest "$PARTEST_REF")
419+
SCALACHECK_VER=$(deriveVersionAnyTag rickynils scalacheck "$SCALACHECK_REF")
378420
fi
379421

380422
echo "Module versions (versioning strategy: $moduleVersioning):"

0 commit comments

Comments
 (0)