Updates for 0.17.0 releasev0.17.0

1 files changed, 56 insertions, 93 deletions

diff --git a/doc/release.cli b/doc/release.cli
index 78d505f..c387714 100644
--- a/doc/release.cli
+++ b/doc/release.cli
@@ -23,7 +23,8 @@ Consider how/when they are updated/tested during the release process.
 @@ We currently have an issue in that \c{queue} builds \c{public} using
 \c{public} \c{buildtabs} (since it's querying \c{public} brep) which means
 existing packages are not tested with new build configurations. But maybe
-that's correct, conceptually.
+that's correct, conceptually. We also have the inverse problem: stage bots
+may no longer have the machines to build public.
 
 \h1#stage|Stage|
 
@@ -37,7 +38,7 @@ and then be queued and published (effectively released) as part of the
 bundle the release of dependencies with the release of \c{build2} to keep the
 process as streamlined as possible. In fact, we now have \c{queue.stage} (see
 \c{etc/stage-queue}) which is the place for such \"extra packages for
-testing\".
+testing (or that require the staged toolchain)\".
 
 \N|When unbundling the release of a dependency we need to remove its
 distribution from \c{etc/stage} and add the pre-distributed packages
@@ -125,7 +126,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages
 
 \h#packaging|Release \c{packaging/} dependencies|
 
-  Release dependencies that have the snapshot version as described in
+  Release dependencies mentioned in \c{etc/stage} that have the snapshot
+  version as described in
   \l{https://github.com/build2-packaging/README#version-and-release-management
   Version and Release Management}.
 
@@ -135,6 +137,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages
      publish) these now and other dependencies later? Maybe so that we can
      stage them one more time?|
 
+  Note: may want to update dependencies bundled as \c{git} submodules after
+  that.
 
 \h#dependencies|Finalize all other dependencies|
 
@@ -161,38 +165,24 @@ distribution from \c{etc/stage} and add the pre-distributed packages
   etc/upgrade
   \
 
-  Or, if released ODB on the previous step, update \c{libodb*} version
-  constraint using \c{etc/version}, commit, and then:
+  If released ODB on the previous step:
 
   \
   # Step 0: make sure b, bpkg, bdep are runnable.
 
-  # Step 1a: pull CLI compiler and build it, make sure runnable.
-  # Step 1b: pull ODB compiler and build it, make sure runnable.
-
-  # Step 2: pull everything in build2 (including brep).
-
-  # Step 3: upgrade (in configure-only mode).
-  #
-  # If this step goes sideways, use the saved .bak configurations to
-  # revert, fix the issue, and try again. If unable to fix the issue
-  # (for example, the toolchain is not runnable), then do the from-
-  # scratch bootstrap using the etc/bootstrap script.
-  #
-  etc/upgrade -c
-
-  # Step 4: trigger header regeneration (ignore ODB version errors).
-  #
-  BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7/
-
-  # Step 5: regenerated ODB code in the relevant projects (bpkg, bdep, brep):
-  #
-  ./odb.sh ~/work/build2/builds/gcc7/
-
-  # Step 6: finish toolchain update:
-  #
-  BDEP_SYNC=0 b build2/ bpkg/ bdep/
-  b build2/ bpkg/ bdep/ # Should be noop.
+  # Step 1a: update odb submodule in libbutl (see README-DEV)
+  # Step 1b: run odb.sh in bpkg, bdep.
+  # Step 1c: update b, bpkg, bdep.
+  # Step 1d: copy *-odb.?xx for bpkg and bdep to build2-toolchain.
+
+  # Step 2a: update ODB version in etc/version
+  # Step 2b: run etc/version (should only change brep)
+  # Step 2c: commit the changes to brep
+  # Step 2d: run bdep sync -f (should upgrade libodb*)
+  # Step 2e: manually update version.hxx in libodb, libodb-pgsql,
+  #          and libbrep by running b .../hxx{version}
+  # Step 2f: run odb.sh ib brep/libbrep
+  # Step 2g: update brep
   \
 
   Then push \c{build2} repositories.
@@ -226,10 +216,13 @@ distribution from \c{etc/stage} and add the pre-distributed packages
   diff -u intro2-tldr.orig intro2-tldr.out  # Or use gitk.
   mv intro2-tldr.out intro2-tldr.orig
 
+  # NOTE: comment out !config.import.libbuild2_hello in ~/.build2/b.options
 
   script -qc ./intro2-tour intro2-tour.out && sed -i -e 's/\r//g' intro2-tour.out
   diff -u intro2-tour.orig intro2-tour.out  # Or use gitk.
   mv intro2-tour.out intro2-tour.orig
+
+  # NOTE: uncomment !config.import.libbuild2_hello in ~/.build2/b.options
   \
 
 \h#doc|Review documentation|
@@ -370,16 +363,16 @@ distribution from \c{etc/stage} and add the pre-distributed packages
 
 \h#test-extra|Perform extra testing|
 
-  CI: (check for any new repositories in github.com/build2/)\n
+  CI: (check for any new repositories in \c{github.com/build2/})\n
   \n
   \c{libauto-symexport}\n
   \c{hello-thrift}\n
   \c{assembler-with-cpp}\n
 
-  Test \c{cxx20-modules-examples} (see \c{test} script).
+  Test \c{cxx20-modules-examples} (see \c{test} script) and \c{async_simple}.
 
   Test any third-party/demos (\c{build2-dynamic-module-demo},
-  \c{cherimcu}, \c{boost-dependency}).
+  \c{cherimcu}, \c{cherio-rtos}, \c{boost-dependency}).
 
   Test on ARM Mac (run tests for \c{libbutl/build2/bpkg/bdep}).
 
@@ -472,7 +465,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages
 
         \
         etc/version
-        ./commit.sh
+        ./commit.sh                            # \"Release version X.Y.Z\"
         git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\"
         \
 
@@ -499,7 +492,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages
         pre-installed build system modules, use \c{BDEP_SYNC=0}).|
 
     \li|Upgrade all dependencies in configure-only mode by executing
-        \c{etc/upgrade\ -c}.
+        \c{etc/upgrade\ -c} (don't skip this step even if all the dependencies
+        were released during staging).
 
         Avoid this (see above): If the \c{build2}/\c{bpkg} requirements in the
         manifests have been bumped to the version being released, then first
@@ -519,23 +513,24 @@ distribution from \c{etc/stage} and add the pre-distributed packages
         to \"close off\"):
 
         \
-        BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7-asan/
+        BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc/
         \
 
         If using GCC prior to 8 then might also need explicit version target
         upgrades:
 
         \
-        BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/.../hxx{version}
+        BDEP_SYNC=0 b ~/work/build2/builds/gcc/.../hxx{version}
         \
 
     |
 
     \li|Regenerate ODB in relevant packages passing upgraded configuration
-        path explicitly (\c{bdep} is not runnable):
+        path explicitly (\c{bdep} is not runnable; can skip if ODB was
+        released during staging):
 
        \
-       ./odb.sh ~/work/build2/builds/gcc7-asan/
+       ./odb.sh ~/work/build2/builds/gcc/
        \
 
     |
@@ -544,7 +539,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages
     configuration:
 
        \
-       BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/
+       BDEP_SYNC=0 b ~/work/build2/builds/gcc/
 
        b build2/ bpkg/ bdep/
 
@@ -595,8 +590,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages
   \ul|
 
     \li|Regenerate documentation in each package inside as well as in
-        \c{build2-toolchain} itself. @@ \c{libbuild-kconfig} not configured
-        out of tree? Or maybe it gets updated automatically during dist?|
+        \c{build2-toolchain} itself. Note: for \c{libbuild-kconfig}
+        it gets updated automatically during dist.|
 
     \li|Update ODB by copying relevant files from the previous step (trust me,
         this is the easy way for now). Make sure all \c{*-odb.*} are copied!|
@@ -621,7 +616,9 @@ distribution from \c{etc/stage} and add the pre-distributed packages
 
    Sort non-alpha packages from \c{cppget.org/queue/1/alpha/} into appropriate
    sections (we could probably automate this similar to \c{bdep-release(1)}).
-   Also check if any of them are already in \c{public}.
+   Also check if any of them are already in \c{public}. Note that some
+   build system modules may already in \c{public}. If standard pre-installed,
+   then need to copy archives from \c{public} (so that checksums match).
 
    Note also that we assume all the packages already have the corresponding
    ownership information either in \c{queue} or \c{public}. However, if any
@@ -672,7 +669,10 @@ distribution from \c{etc/stage} and add the pre-distributed packages
   After upgrade, drop all package builds for \c{public} and \c{queue} and
   verify there are no regressions. This makes sure that the \c{brep} services
   in the new version are usable with the old toolchain. If there are issues,
-  they have to be fixed by publishing/queuing revisions.
+  they have to be fixed by publishing/queuing revisions. Note that if
+  \c{public} was tested with staged toolchain (and thus \c{brep} was already
+  upgraded), perhaps this can be skipped (since it will take days to
+  completely rebuild \c{public}).
 
   \N|The \c{bdep} tests that exercise CI and publishing services do it in the
   simulation mode. To properly test that these services are compatible with
@@ -731,68 +731,31 @@ distribution from \c{etc/stage} and add the pre-distributed packages
 
 \h#public-machines|Update \c{public} \c{buildtab}s and build machines|
 
-  Poweroff the old set of \c{public} build hosts:
+  Stop \c{public} build hosts:
 
   \
-  ./po-hosts -c public
+  ./po-hosts -s public -c public
   \
 
   Update \c{public} \c{buildtab}s based on the \c{queue} \c{buildtab}
   (normally just a copy).
 
-  Adjust build host configurations (hardware classes, etc) and add/remove
-  new/old build hosts.
-
-  Replace the \c{public} \c{buildos} image on \c{build-cache} with the
-  one for \c{stage}.
-
-  Comment out the \c{public} toolchain in the build host configuration
-  (effectively making it a no-toolchain configuration), regenerate, and power
-  on the new set of \c{public} build hosts.
+  Adjust \c{public} build host configurations (hardware classes, etc) and
+  add/remove new/old build hosts.
 
+  Use new sync scripts to update \c{public} build hosts with new VMs.
 
-  @@ See new sync scripts for this step: Review deployed machines against the
-  updated \c{public} \c{buildtab} and remove those that are no longer used:
+  Power off \c{public} build hosts:
 
   \
-  cd private/buildos/
-
-  grep '.*' .../brep-buildtab | cut -d' ' -f1 | sort -u
-  ./ls-machines -c public
-
-  ~/work/build2/buildos/remove-machine <host> <machine>
-  \
-
-  Then move now legacy machines to the \"legacy\" build host:
-
-  \
-  grep 'legacy' .../brep-buildtab | cut -d' ' -f1 | sort -u
-  \
-
-  Also review deployed machines against the latest available versions and
-  upgrade those that are not the latest:
-
-  \
-  cd private/buildos/
-
-  ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" | sort
-  ./ls-machines -c public
-
-  ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver
-  \
-
-  Finally, add any new machines:
-
-  \
-  grep -v 'legacy' .../brep-buildtab | cut -d' ' -f1 | sort -u
+  ./po-hosts -c public
   \
 
-  Uncomment the \c{public} toolchain in the build host configuration and
-  regenerate. The only remaining step is to reboot (not yet):
+  Replace the \c{public} \c{buildos} image on \c{build-cache} with the
+  one for \c{stage} (note: this will cause build hosts to reboot, thus
+  power off above).
 
-  \
-  ./po-hosts -r -c public
-  \
+  The only remaining step is to boot \c{public} build hosts (not yet).
 
 \h#pub-dist|Publish distribution|