Updates for 0.9.0 releasev0.9.0

4 files changed, 733 insertions, 2 deletions

diff --git a/doc/.gitignore b/doc/.gitignore
index a5873dc..36afaee 100644
--- a/doc/.gitignore
+++ b/doc/.gitignore
@@ -1 +1,3 @@
 build2-bash-style.xhtml
+build2-release.xhtml
+
diff --git a/doc/bash-style.cli b/doc/bash-style.cli
index 9e90e8a..d22cc23 100644
--- a/doc/bash-style.cli
+++ b/doc/bash-style.cli
@@ -9,7 +9,7 @@
 // - Maximum <pre> line is 70 characters.
 //
 
-"\h1|Contents|"
+"\h1|Table of Contents|"
 "\$TOC$"
 
 "
@@ -27,7 +27,7 @@ You just need to be clear on why you are doing it.
 See also \l{https://google.github.io/styleguide/shell.xml Google's Bash Style
 Guide} as well as \l{https://github.com/progrium/bashstyle Let's do Bash
 right!}; we agree with quite a few (but not all) items in there. In particular,
-it provides a lot more rationale compared to this guide.
+the former provides a lot more rationale compared to this guide.
 
 \h1#style|Style|
 
diff --git a/doc/cli.sh b/doc/cli.sh
index 44d311a..e737ee7 100755
--- a/doc/cli.sh
+++ b/doc/cli.sh
@@ -22,3 +22,8 @@ cli --generate-html --html-suffix .xhtml \
 --html-prologue-file doc-prologue.xhtml \
 --html-epilogue-file doc-epilogue.xhtml \
 --output-prefix build2- bash-style.cli
+
+cli --generate-html --html-suffix .xhtml \
+--html-prologue-file doc-prologue.xhtml \
+--html-epilogue-file doc-epilogue.xhtml \
+--output-prefix build2- release.cli
diff --git a/doc/release.cli b/doc/release.cli
new file mode 100644
index 0000000..6639371
--- /dev/null
+++ b/doc/release.cli
@@ -0,0 +1,724 @@
+// file      : doc/release.cli
+// copyright : Copyright (c) 2014-2018 Code Synthesis Ltd
+// license   : MIT; see accompanying LICENSE file
+
+"\title=Release Process"
+
+// NOTES
+//
+// - Maximum <pre> line is 70 characters.
+//
+
+"\h1|Table of Contents|"
+"\$TOC$"
+
+"
+\h0#process|Process|
+
+Review the state and services list (currently on paper) for any new additions.
+Consider how/when they are updated/tested during the release process.
+
+
+\h1#stage|Stage|
+
+The staging repository is completely independent which means it must contain
+all the \c{build2} dependencies (plus a few extra packages for testing; see
+the \c{etc/stage} script for the complete list). This in turn means that if
+any of these dependencies are in the unreleased state, then they should go
+through the applicable steps in this section (e.g., updating of \c{NEWS}, etc)
+and then be queued and published (effectively released) as part of the
+\c{build2} release. Generally, however, we should strive to not unnecessarily
+bundle the release of dependencies with the release of \c{build2} to keep
+the process as streamlined as possible.
+
+\N|When unbundling the release of a dependency we need to remove its
+distribution from \c{etc/stage} and add the pre-distributed packages
+(for example, from \c{public}) to \c{staging/repository/1/}.|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|Current \c{stage} build is clean.||
+
+\h#copy|Update copyright if new year|
+
+  Use the \c{git/copyright} script. See the script header for instructions.
+
+\h#etc|Update \c{etc/git/modules}|
+
+  Review for any new modules. Remove \c{etc/} and \c{private/} from
+  \c{modules} to reduce noise during stat.
+
+\h#review|Review \c{@@}|
+
+  Review \c{@@} notes:
+
+  \
+  ./review.sh | less -R
+  \
+
+  At least look for \c{@@\ TMP}
+
+  \
+  ./review.sh | grep TMP
+  \
+
+\h#review-db|Review database schema changes|
+
+  Review database schema changelog differences in \c{bpkg}, \c{bdep}, and
+  \c{brep} compared to the previous release (tag) for any schema/data
+  migration that may be required (if something is missing, then it would also
+  have to be tested).
+
+\h#news|Update \c{NEWS} files|
+
+  Update in all projects (including dependencies) that will have a release
+  version.
+
+  See \c{etc/stage} for the complete list.
+
+
+\h#packaging|Release \c{packaging/} dependencies|
+
+  Release dependencies that have the snapshot version as described in
+  \c{private/build2-packaging.txt}.
+
+  \N|Maybe this should be done during queuing? Why do we release (but not
+     publish) these now and other dependencies later?|
+
+
+\h#dependencies|Finalize all other dependencies|
+
+  Make sure all other unreleased dependencies listed in \c{etc/stage} are
+  ready to be released. Effectively, the only remaining step should be to
+  change the version.
+
+  Do this in the dependency order and finish each off with:
+
+  \
+  git pull && bdep sync -fura && bdep test -ar
+  \
+
+
+\h#upgrade|Upgrade dependencies|
+
+  Upgrade all dependencies in the \c{build2} toolchain build:
+
+  \
+  etc/upgrade
+  \
+
+
+\h#hello|Update \c{hello/} projects|
+
+  These projects should closely track the output of \c{bdep-new(1)}. The
+  recommended procedure is to generate a new project with the same name and
+  then review/resolve the differences with \c{diff\ -ru}.
+
+  Every change in these projects must be accompanied by a revision increment
+  and a re-tag. This should be managed with \c{bdep-release(1)}:
+
+  \
+  # make changes
+  git add .
+  bdep release --revision --show-push
+  # review commit
+  git push ...
+  \
+
+  Once done, run the \c{intro} scripts and review any changes in the output
+  (this information will be helpful on the next step):
+
+  \
+  cd etc
+
+  ./intro2-tldr 2>&1 | tee intro2-tldr.out
+  diff -u intro2-tldr.orig intro2-tldr.out  # Or use gitk.
+  mv intro2-tldr.out intro2-tldr.orig
+
+  ./intro2-tour 2>&1 | tee intro2-tour.out
+  diff -u intro2-tour.orig intro2-tour.out  # Or use gitk.
+  mv intro2-tour.out intro2-tour.orig
+  \
+
+\h#doc|Review documentation|
+
+  Review the following documentation for (1) sample output changes and (2)
+  still being relevant/making sense.
+
+  \N|Ideally this should be done during development but it's easy to forget.
+     Also, check if there is any new documentation that has been added that is
+     not on the below list.|
+
+  \ul|
+
+    \li|Install guide: 1 & 2.|
+
+    \li|Toolchain introduction: 1 & 2 (use \c{intro} script output).|
+
+    \li|Introduction in the build system manual: 1 (uses \c{bdep-new(1)}
+        output).|
+
+    \li|Testscript manual: 1.||
+
+
+\h#submod|Update all submodules|
+
+  Make sure all working trees are clean. Review \c{sub_modules} in
+  \c{etc/git/modules} for any missing new modules, then:
+
+  \N|\c{etc/} and \c{private/} need to be committed manually.|
+
+  \
+  ./modup.sh
+  ./commit.sh # If changes.
+  ./push.sh
+
+  cd build2-toolchain
+
+  git submodule update --remote --checkout
+  git submodule foreach git submodule update --init --recursive
+
+  git status  # There should only be 'new commits'; see README-GIT
+
+  git commit -a -m \"Update submodules\"
+  git push
+  \
+
+\h#stage-machines|Update \c{stage} \c{buildtab}s and build machines|
+
+  Review \c{stage} \c{buildtab} for any configurations to drop (for example,
+  an intermediate version of a compiler).
+
+  Based on these changes update \c{stage} CI \c{buildtab}, which is a subset
+  of the \c{stage} configurations (and is a base for the \c{queue}/\c{public}
+  configurations).
+
+  Review deployed machines against the updated \c{stage} \c{buildtab} and
+  remove those that are no longer used:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -c stage -c devel
+
+  ~/work/buildos/remove-machine <host> <machine>
+  \
+
+  Also review deployed machines against the latest available versions and
+  upgrade those that are not the latest:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\"
+  ./ls-machines -c stage -c devel
+
+  ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver
+  \
+
+
+\h#restage|Restage|
+
+  Confirm \c{stage} and \c{devel} \c{buildos} images, hardware-specific
+  configurations are the same.
+
+  Review \c{staging/0/} and \c{staging/repository/1/} for anything stray.
+
+  Update all submodules in \c{build2-toolchain}:
+
+  \
+  git submodule update --remote --checkout
+  \
+
+  Restage with \c{baseutils}/\c{mingw} regeneration:
+
+  \
+  etc/stage -b
+  \
+
+  Upgrade \c{brep} on \c{stage} and sync latest \c{buildtab}s.
+
+  Verify \c{stage} build is clean, nothing is unbuilt.
+
+
+\h#install-stage|Test install scripts|
+
+  Test \l{https://stage.build2.org/0/ \c{stage} install scripts}, including
+  upgrading, as described in \c{private/install/testing.txt}.
+
+
+\h1#queue|Queue|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|Final \c{stage} build is clean.|
+
+\li|Build with the \c{queue} toolchain is inactive.||
+
+
+\h#queue-repo|Review \c{queue} repository|
+
+  Pull, review, and clean the \c{queue} \c{git} repository for any stale
+  packages or anything (ownership, etc) that may conflict with packages being
+  released.
+
+  Also clean toolchain distribution:
+
+  \
+  rm -rf cppget.org/queue/0/*
+  \
+
+\h#version-release|Change to release version|
+
+  Change to the release version in all the packages being released and tag
+  (see \c{etc/stage} for the list). Use \c{bdep-release(1)} unless a custom
+  versioning/tagging scripts are used:
+
+  \
+  bdep release --no-open --show-push [--alpha|--beta]
+  # review commit
+  git push ...
+  \
+
+  Do this in the dependency order and finish each off with:
+
+  \
+  bdep sync -fura && bdep update
+  \
+
+  For the \c{build2} packages (commit but don't push after each step if
+  required):
+
+  \ul|
+
+    \li|Close schema versions in \c{bpkg}, \c{bdep}, and \c{brep}.|
+
+    \li|Change \c{BUILD2_STAGE} in \c{build2/build2/config.hxx.in} to \c{false}.|
+
+    \li|Change version by updating (including with new modules) and then
+        executing:
+
+       \
+       etc/version
+       ./commit.sh
+       git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\"
+       \
+
+    |
+
+    \li|Tag by executing \c{tag.sh\ <version>}.|
+
+    \li|Regenerate documentation in each package.|
+
+    \li|Upgrade all dependencies in configure-only mode by executing
+    \c{etc/upgrade\ -c}.|
+
+    \li|Trigger regeneration of version files (might require several runs
+        to \"close off\"):
+
+        \
+        BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7-asan/
+        \
+
+        If using GCC prior to 8 then might also need explicit version target
+        upgrades:
+
+        \
+        BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/.../hxx{version}
+        \
+
+    |
+
+    \li|Regenerate ODB in relevant packages passing upgraded configuration
+        path explicitly (\c{bdep} is not runnable):
+
+       \
+       ./odb.sh ~/work/build2/builds/gcc7-asan/
+       \
+
+    |
+
+    \li|Finish upgrading all dependencies by executing in the upgraded
+    configuration:
+
+       \
+       BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/
+       \
+
+    ||
+
+  Verify key tests pass (in particular, the \c{bdep} tests will now be running
+  against \c{public} services):
+
+  \
+  b test: build2/ bpkg/ bdep/
+  \
+
+  \N|We could have queued after this step before preparing
+     \c{build2-toolchain}. However, splitting this seems to only increase
+     complexity without any major benefits (it's not hard to update submodules
+     and regenerated \c{build2-toolchain} if there are any issues with
+     packages). Plus we can start testing install scripts, etc.|
+
+  Next push the above changes and update all submodules in
+  \c{build2-toolchain}:
+
+  \
+  ./push.sh
+
+  cd build2-toolchain
+
+  git submodule update --remote --checkout
+
+  git status  # There should only be 'new commits'; see README-GIT
+
+  git commit -a -m \"Update submodules\"
+  \
+
+  As well as (commit after each step if required):
+
+  \ul|
+
+    \li|Regenerate documentation in each package inside as well as in
+        \c{build2-toolchain} itself.|
+
+    \li|Update ODB by copying relevant files from the previous step (trust
+        me, this is the easy way).|
+
+    \li|Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to
+    \c{queue}.||
+
+  Finally, push the changes:
+
+  \
+  git push
+  \
+
+\h#queuing|Queue|
+
+   Prepare packages and the toolchain distribution:
+
+   \
+   etc/stage -q -b
+   \
+
+   Sort non-alpha packages from \c{cppget.org/queue/1/alpha/} into appropriate
+   sections (we could probably automate this similar to \c{bdep-release(1)}).
+
+   Note also that we assume all the packages already have the corresponding
+   ownership information either in \c{queue} or \c{public}. However, if any
+   new packages were added, then that will have to be added as well.
+
+   Commit and push queue repository:
+
+   \
+   cd cppget.org/queue/
+   git add .
+   git ci -m \"Queue build2 toolchain X.Y.Z\"
+   git push
+   \
+
+\h#build-public|Verify queued packages build with \c{public}|
+
+   This makes sure that the new version can be built with the old toolchain.
+
+   While all the packages should build (except perhaps \c{libhello} which,
+   being based on the latest \c{bdep-new(1)} output, may use new features),
+   some tests may fail. In particular, we will be running \c{bpkg} tests using
+   a previous version of the build system and \c{bdep} tests also using a
+   previous version of \c{bpkg}. None of that is or will be supported. So
+   for now we manually review all the failed builds for anything suspicious
+   and in the future we may skip tests if the tool versions mismatch.
+
+
+\h#install-queue|Test install scripts|
+
+  Test \l{https://download.build2.org/queue/ \c{queue} install scripts}. Here
+  we just smoke-test each script on its \"primary\" platform and make sure
+  \c{queue} URLs/repositories are used.
+
+
+\h#upgrade-brep-bpub|Upgrade \c{brep} and \c{bpub} on \c{cppget.org}|
+
+  Upgrade \c{brep} and \c{bpub} using the queued toolchain according
+  to \c{private/brep-bpub-upgrade.txt}.
+
+  After upgrade, drop all package builds for \c{public} and \c{queue} and
+  verify there are no regressions. This makes sure that the \c{brep} services
+  in the new version are usable with the old toolchain. If there are issues,
+  they have to be fixed by publishing/queuing revisions.
+
+  \N|The \c{bdep} tests that exercise CI and publishing services do it in the
+  simulation mode. To properly test that these services are compatible with
+  the old version of the toolchain we would need to CI/publish a real test
+  package manually.|
+
+
+\h#start-queue|Start \c{queue} builds|
+
+  Update \c{queue} \c{buildtab} based on the \c{stage} CI \c{buildtab}
+  (normally just a copy sans the sanitized toolchain configurations).
+
+  Adjust \c{stage} and \c{devel} build host configurations to enable the
+  \c{queue} toolchain. Shift most instances from \c{stage} to \c{queue}
+  in the hardware class-specific configurations. Regenerate affected
+  configurations and reboot build hosts:
+
+  \
+  cd private/buildos/
+
+  ./gen-config stage <class>
+  ./gen-config devel <class>
+
+  ./po-hosts -r -c stage -c devel
+  \
+
+  Verify both \c{queue} and \c{public} builds with the queued toolchain,
+  fixing any breakages with revisions, moving to legacy, and/or queuing
+  replacements. We can only proceed further once we have a \"resolution\"
+  for every (newly) broken package.
+
+
+\h#stop-queue|Stop \c{queue} builds|
+
+  Adjust \c{stage} and \c{devel} build host configurations to disable the
+  \c{queue} toolchain (comment out). Regenerate affected configurations and
+  reboot build hosts as on the previous step.
+
+\h1#public|Public|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|A \"resolution\" is queued or published for every (newly) broken package.||
+
+\h#public-machines|Update \c{public} \c{buildtab}s and build machines|
+
+  Poweroff the old set of \c{public} build hosts:
+
+  \
+  ./po-hosts -c public
+  \
+
+  Update \c{public} \c{buildtab}s based on the \c{queue} \c{buildtab}
+  (normally just a copy).
+
+  Adjust build host configurations and add/remove new/old machines.
+
+  Replace the \c{public} \c{buildos} image on \c{build-cache} with the
+  one for \c{stage}.
+
+  Comment out the \c{public} toolchain in the build host configuration
+  (effectively making it a no-toolchain configuration) and power on the new
+  set of \c{public} build hosts.
+
+  Review deployed machines against the updated \c{public} \c{buildtab} and
+  remove those that are no longer used:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -c public
+
+  ~/work/buildos/remove-machine <host> <machine>
+  \
+
+  Also review deployed machines against the latest available versions and
+  upgrade those that are not the latest:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\"
+  ./ls-machines -c public
+
+  ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver
+  \
+
+  Uncomment the \c{public} toolchain in the build host configuration. The
+  only remaining step is to reboot (not yet):
+
+  \
+  ./po-hosts -r -c public
+  \
+
+\h#pub-dist|Publish distribution|
+
+  Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to \c{public}
+  and publish the distribution (this also cleans/disables the \c{queue}
+  toolchain):
+
+  \
+  etc/stage -p
+  \
+
+\h#pub-pkg|Publish packages|
+
+  Move packages (manually for now) from the \c{queue} to \c{public} \c{git}
+  repository, including ownership information. Move old/replaced/FTB
+  packages either to legacy or delete.
+
+  Commit everything (see the commit log for procedure @@ this needs
+  automation) and push (which should trigger auto-publish). Note: commit and
+  push both \c{queue} and \c{public} \c{git} repositories.
+
+  Note that once published, the existing install instructions/download
+  links are no longer usable, so do not linger.
+
+\h#start-public|Start \c{public} builds|
+
+  Reboot \c{public} build hosts. They should next bootstrap and proceed
+  with building all packages.
+
+\h#install-public|Test install scripts|
+
+  Test \l{https://download.build2.org/ \c{public} install scripts}. Here
+  we just smoke-test each script on its \"primary\" platform and make sure
+  \c{public} URLs/repositories are used.
+
+\h#web|Update web|
+
+  \ul|
+
+  \li|Write release notes, use placeholder for announcement URL (or guess).
+      Add link from the \c{doc.cli}. Add blog entry.|
+
+  \li|Add any new FAQ entries or any other updates (main page, etc). Any new
+      documentation to link from the Doc page?|
+
+  \li|Update the Download page.
+
+      \
+      cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256`
+      \
+
+  |
+
+  \li|Update the Install page.
+
+      \
+      cat cppget.org/public/0/toolchain.sha256
+      \
+
+  |
+
+  \li|Regenerate documentation in \c{private/} for all hosts:
+
+      \
+      cd private/
+
+      cd <host>/www/
+      ./cli.sh
+      \
+
+  |
+
+  \li|Test locally, publish, and test remote:
+
+      \
+      cd private/
+      ./publish --dry-run
+      ./publish
+      \
+
+  ||
+
+
+\h#ann|Announce|
+
+  \ul|
+
+  \li|Write and send the announcement to the mailing lists.
+
+      \
+      cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256`
+      \
+
+      Add \c{reply-to:} header for \c{users@} announcement.|
+
+  \li|Patch (or verify) the announcement URL in release notes, re-publish.|
+
+  \li|Announce on reddit, slack, and other relevant places.||
+
+\h#tag|Commit, tag, and push the rest.|
+
+   Tag \c{build2-toolchain}:
+
+   \
+   cd build2-toolchain
+   git tag -a vX.Y.Z -m \"Tag version X.Y.Z\"
+   git push --follow-tags
+   \
+
+   Add \c{etc/} and \c{private/} back to \c{modules} in \c{etc/git/modules}
+   (essentially reverse \l{#etc Update \c{etc/git/modules}}).
+
+   Finalize changes, commit, tag, and push \c{style/}, \c{etc/}, and
+   \c{private/}.
+
+   Snapshot \c{buildos} subvolumes:
+
+   \
+   btrfs subvolume snapshot buildos-3 buildos-3-X.Y.Z
+   btrfs subvolume snapshot buildos-6 buildos-6-X.Y.Z
+   \
+
+\h1#reopen|Reopen|
+
+\h#version-snapshot|Change to snapshot version|
+
+  Change to the snapshot version in all the released packages. Use
+  \c{bdep-release(1)} unless a custom versioning script is used:
+
+  \
+  bdep release --open --show-push [--open-*]
+  # review commit
+  git push ...
+  \
+
+  Essentially, the same steps as in \l{#version-release Change to
+  release version} (but no tagging).
+
+\h#stage-machines-reopen|Update \c{stage} \c{buildtab}s and build machines|
+
+  Essentially, the same steps as in \l{#public-machines Update \c{public}
+  \c{buildtab}s and build machines} but for stage. Some differences:
+
+  Clean \c{buildtab}s (both \c{stage} and CI) by removing no longer relevant
+  configurations, moving some to \c{legacy}, etc.
+
+  More generally, this is the time to do sweeping changes such as renaming
+  machines/configurations, adjusting classes, etc. This is also the time to
+  rebalance machines across available hosts.
+
+\h#restage-reopen|Restage|
+
+  Make symlinks for the new version in \c{private/baseutils} (for both
+  \c{baseutils} and \c{mingw}; the idea is that we will start with those and
+  maybe upgrade later).
+
+  Then cleanup and restage:
+
+  \
+  rm -r staging/0/*
+  rm -r staging/repository/1/*/
+
+  etc/stage -b
+  \
+
+\h#start-stage|Start \c{stage} builds|
+
+  Reboot \c{stage} build hosts. They should next bootstrap and proceed with
+  building all the staged packages. Make sure all the builds of the new
+  development snapshot are successful and there is nothing unbuilt.
+
+\h#commit-reopen|Commit and push \c{etc/} and \c{private/}.|
+
+  Commit and push changes to \c{etc/} and \c{private/}.
+
+"