diff options
Diffstat (limited to 'doc/release.cli')
| -rw-r--r-- | doc/release.cli | 274 |
1 files changed, 233 insertions, 41 deletions
diff --git a/doc/release.cli b/doc/release.cli index 6def002..8aba681 100644 --- a/doc/release.cli +++ b/doc/release.cli @@ -17,6 +17,13 @@ Review the state and services list (currently on paper) for any new additions. Consider how/when they are updated/tested during the release process. +@@ We have switched to a single configuration for the entire toolchain + (plus -libs). + +@@ We currently have an issue in that \c{queue} builds \c{public} using +\c{public} \c{buildtabs} (since it's querying \c{public} brep) which means +existing packages are not tested with new build configurations. But maybe +that's correct, conceptually. \h1#stage|Stage| @@ -27,8 +34,10 @@ any of these dependencies are in the unreleased state, then they should go through the applicable steps in this section (e.g., updating of \c{NEWS}, etc) and then be queued and published (effectively released) as part of the \c{build2} release. Generally, however, we should strive to not unnecessarily -bundle the release of dependencies with the release of \c{build2} to keep -the process as streamlined as possible. +bundle the release of dependencies with the release of \c{build2} to keep the +process as streamlined as possible. In fact, we now have \c{queue.stage} (see +\c{etc/stage-queue}) which is the place for such \"extra packages for +testing\". \N|When unbundling the release of a dependency we need to remove its distribution from \c{etc/stage} and add the pre-distributed packages @@ -44,6 +53,14 @@ distribution from \c{etc/stage} and add the pre-distributed packages Use the \c{git/copyright} script. See the script header for instructions. + Note that most of (all?) our project have been converted to the automatic + copyright update notification via a pre-commit hook. So perhaps this step + can be removed. + +\h#install-times|Update install script times.| + + See \c{private/install/build2-times.txt} for instructions. + \h#etc|Update \c{etc/git/modules}| Review for any new modules. Remove \c{etc/} and \c{private/} from @@ -51,7 +68,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages \h#review|Review \c{@@}| - Review \c{@@} notes: + Review \c{etc/review} for new modules. Then review \c{@@} notes: \ etc/review | less -R @@ -81,7 +98,10 @@ distribution from \c{etc/stage} and add the pre-distributed packages \h#packaging|Release \c{packaging/} dependencies| Release dependencies that have the snapshot version as described in - \c{private/build2-packaging.txt}. + \l{https://github.com/build2-packaging/README#version-and-release-management + Version and Release Management}. + + Also, consider upgrading the \c{curl/ca-certificates-curl}. \N|Maybe this should be done during queuing? Why do we release (but not publish) these now and other dependencies later? Maybe so that we can @@ -113,6 +133,41 @@ distribution from \c{etc/stage} and add the pre-distributed packages etc/upgrade \ + Or, if released ODB on the previous step, update \c{libodb*} version + constraint using \c{etc/version}, commit, and then: + + \ + # Step 0: make sure b, bpkg, bdep are runnable. + + # Step 1: pull ODB compiler and build it, make sure runnable. + + # Step 2: pull everything in build2 (including brep). + + # Step 3: upgrade (in configure-only mode). + # + # If this step goes sideways, use the saved .bak configurations to + # revert, fix the issue, and try again. If unable to fix the issue + # (for example, the toolchain is not runnable), then do the from- + # scratch bootstrap using the etc/bootstrap script. + # + etc/upgrade -c + + # Step 4: trigger header regeneration (ignore ODB version errors). + # + BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7/ + + # Step 5: regenerated ODB code in the relevant projects (bpkg, bdep, brep): + # + ./odb.sh ~/work/build2/builds/gcc7/ + + # Step 6: finish toolchain update: + # + BDEP_SYNC=0 b build2/ bpkg/ bdep/ + b build2/ bpkg/ bdep/ # Should be noop. + \ + + Then push \c{build2} repositories. + \h#hello|Update \c{hello/} projects| @@ -131,17 +186,19 @@ distribution from \c{etc/stage} and add the pre-distributed packages git push ... \ - Once done, run the \c{intro} scripts and review any changes in the output - (this information will be helpful on the next step): + Once done, make sure the latest \c{libhello} revision is on stage, run the + \c{intro} scripts, and review any changes in the output (this information + will be helpful on the next step): \ cd etc - ./intro2-tldr 2>&1 | tee intro2-tldr.out + script -qc ./intro2-tldr intro2-tldr.out && sed -i -e 's/\r//g' intro2-tldr.out diff -u intro2-tldr.orig intro2-tldr.out # Or use gitk. mv intro2-tldr.out intro2-tldr.orig - ./intro2-tour 2>&1 | tee intro2-tour.out + + script -qc ./intro2-tour intro2-tour.out && sed -i -e 's/\r//g' intro2-tour.out diff -u intro2-tour.orig intro2-tour.out # Or use gitk. mv intro2-tour.out intro2-tour.orig \ @@ -158,7 +215,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ul| - \li|Install guide: 1 & 2.| + \li|Install guide: 1 & 2 (also review \c{build2-toolchain} commit log).| \li|Toolchain introduction: 1, 2 & 3 (use \c{intro} script output for 2).| @@ -193,6 +250,12 @@ distribution from \c{etc/stage} and add the pre-distributed packages \h#stage-machines|Update \c{stage} \c{buildtab}s and build machines| + NOTE: may want to keep old machines around for public testing (since we use + existing public buildtabs, none of the new machines will be used). + + Note: normally, we try to do all of this as part of normal development + (e.g., when adding new machines, etc). + Review \c{stage} \c{buildtab} for any configurations to drop (for example, an intermediate version of a compiler), classes to adjust (\c{legacy}, \c{default}, \c{latest}, etc). @@ -207,6 +270,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ cd private/buildos/ + grep '.*' .../brep-buildtab | cut -d' ' -f1 | sort -u ./ls-machines -c stage -c devel ~/work/buildos/remove-machine <host> <machine> @@ -218,7 +282,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ cd private/buildos/ - ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" + ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" | sort ./ls-machines -c stage -c devel ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver @@ -235,6 +299,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages If no upgrade is possible from the previous version, uncomment errors in install scripts (and add a note to restore after the release). + Enable extra packages for testing in \c{etc/stage} script. + Restage and upgrade \c{brep} by performing the following steps: \ol| @@ -248,6 +314,12 @@ distribution from \c{etc/stage} and add the pre-distributed packages etc/stage -b \ + Consider restaging queue for good measure. + + \ + etc/stage-queue + \ + | \li|While build machines are bootstrapping, upgrade \c{brep} on \c{stage}, @@ -265,6 +337,27 @@ distribution from \c{etc/stage} and add the pre-distributed packages Verify \c{stage} build is clean, nothing is unbuilt. +\h#test-extra|Perform extra testing| + + CI: (check for any new repositories in github.com/build2/)\n + \n + \c{libauto-symexport}\n + \c{hello-thrift}\n + \c{assembler-with-cpp}\n + + Test \c{cxx20-modules-examples} (see \c{test} script). + + Test any third-party/demos (\c{build2-dynamic-module-demo}, + \c{cherimcu}, \c{boost-dependency}). + + Test on ARM Mac (run tests for \c{libbutl/build2/bpkg/bdep}). + + Test build system modules (especially standard pre-installed). + + Test old toolchain version (can fetch and build old packages from + \c{queue.stage}; add dummy package if all require to be released toolchain). + + \h#install-stage|Test install scripts| Test \l{https://stage.build2.org/0/ \c{stage} install scripts}, including @@ -309,7 +402,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ git pull bdep release --no-open --show-push [--alpha|--beta] - # review commit + # review commit, run update (in case anything comitted is pre-generated) git push ... \ @@ -333,29 +426,63 @@ distribution from \c{etc/stage} and add the pre-distributed packages done if created projects use new features.} \N|Why shouldn't we always do this for simplicity? Maybe because then - we cannot run tests using \c{public} services? Also the below upgrade - steps will break since there is no continuity.|| + we cannot run tests using \c{public} services? - \li|Change version by updating (including with new modules and/or new - dependencies) and then executing: + Also if we change this in toolchain packages, the below upgrade steps + will break since there is no continuity. Perhaps do it in two stages: + first change version to final, upgrade, then change toolchain + dependencies to final, upgrade again? But then nobody involed in + development will be able to pull and upgrade. Maybe KISS and keep + it pre-release.|| - \ - etc/version - ./commit.sh - git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\" - \ + \li|Change version by updating \c{etc/version} (including with new modules + and/or new dependencies, but keep pre-release in minimum toolchain + version requirements) and then executing: - Note that \c{libbuild2-hello} is independently versioned but may still - need to update minimum \c{build2} version requirements (see below). + \ + etc/version + ./commit.sh + git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\" + \ - | + Note that \c{libbuild2-*} modules (e.g., \c{libbuild2-hello}) are + independently versioned but may still need to update minimum toolchain + version requirements (see below).| \li|Tag by executing \c{tag.sh\ <version>}.| - \li|Regenerate documentation in each package.| + + \li|Release all standard pre-installed build system modules. Update + minimum toolchain version requirements. + + \ + bdep release --no-open --show-push + \ + + Also release \c{libbuild2-hello} (it's not standard pre-installed + but it gets published). + + | + + \li|Regenerate documentation in each package (including standard + pre-installed build system modules, use \c{BDEP_SYNC=0}).| \li|Upgrade all dependencies in configure-only mode by executing - \c{etc/upgrade\ -c}.| + \c{etc/upgrade\ -c}. + + Avoid this (see above): If the \c{build2}/\c{bpkg} requirements in the + manifests have been bumped to the version being released, then first + bootstrap the build system and update \c{bpkg}/\c{bdep} (might have to + hack their generated \c{version.hxx} to disable constraint checking; + also if you forget \c{BDEP_SYNC=0} it will most likely hose the build + configuration). + + \ + BDEP_SYNC=0 b-boot build2/build2/ + BDEP_SYNC=0 b bpkg/ bdep/ + \ + + | \li|Trigger regeneration of version files (might require several runs to \"close off\"): @@ -387,12 +514,16 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/ + + b build2/ bpkg/ bdep/ + + # Update standard pre-installed build system modules. \ | - \li|Update \c{libbuild2-hello} if required.|| + \li|Update other \c{libbuild2-*} modules if required.|| Verify key tests pass (in particular, the \c{bdep} tests will now be running against \c{public} services): @@ -401,6 +532,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages b test: build2/ bpkg/ bdep/ b test: bpkg/ config.bpkg.test.remote=true b test: libbuild2-hello/libbuild2-hello-tests/ + + # Test standard pre-installed build system modules. \ \N|We could have queued after this step before preparing @@ -415,6 +548,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ ./push.sh + # Push (with tags) standard pre-installed build system modules. + cd build2-toolchain git submodule update --remote --checkout @@ -429,12 +564,13 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ul| \li|Regenerate documentation in each package inside as well as in - \c{build2-toolchain} itself.| + \c{build2-toolchain} itself. @@ \c{libbuild-kconfig} not configured + out of tree? Or maybe it gets updated automatically during dist?| \li|Update ODB by copying relevant files from the previous step (trust me, this is the easy way for now). Make sure all \c{*-odb.*} are copied!| - \li|Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to + \li|Change \c{build2_repo} in \c{build2-toolchain} \c{buildfile} to \c{queue}.|| Finally, push the changes: @@ -445,7 +581,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages \h#queuing|Queue| - Prepare packages and the toolchain distribution: + Prepare packages and the toolchain distribution (disable extra packages + first if any were enabled in the \c{etc/stage} script): \ etc/stage -q -b @@ -468,10 +605,13 @@ distribution from \c{etc/stage} and add the pre-distributed packages git push \ - If queued package manifests contain new values, then the bpkg-rep-publish - script will fail to create repository due to unknown manifest values. To - resolve this we temporarily add (to \c{crontab}) \c{--ignore-unknown} and - make a note to restore. + If queued package manifests contain new values, then the + \c{bpkg-rep-publish} script will fail to create repository due to unknown + manifest values. To resolve this we temporarily add (to \c{crontab}) + \c{--ignore-unknown} and make a note to restore. + + Also change \c{--min-bpkg-version} from previous to current release + (not the one being released). \h#build-public|Verify queued packages build with \c{public}| @@ -516,7 +656,12 @@ distribution from \c{etc/stage} and add the pre-distributed packages \N|Note that the \c{queue} \c{buildtab} is shared between \c{public} and \c{queue} builds. As a result, after this update, \c{public} build hosts - may not have some of the new (or renamed) build machines.| + may not have some of the new (or renamed) build machines. + + This also means that if the \c{buildtab} contains anything new (options, + etc) that are incompatible with \c{public}, then they should only be + enabled later, when upgrading \c{public} \c{buildtab} (make a note if + that's the case).| Adjust \c{stage} and \c{devel} build host configurations (both \c{*-config} and hardware classes) to enable the \c{queue} toolchain. Shift most @@ -535,6 +680,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages replacements. We can only proceed further once we have a \"resolution\" for every (newly) broken package. + \N|If public has been built with the staged toolchain, rebuilding of the + public repository (which takes days) can be omitted.| \h#stop-queue|Stop \c{queue} builds| @@ -572,18 +719,24 @@ distribution from \c{etc/stage} and add the pre-distributed packages (effectively making it a no-toolchain configuration), regenerate, and power on the new set of \c{public} build hosts. - Review deployed machines against the updated \c{public} \c{buildtab} and - remove those that are no longer used: + + @@ See new sync scripts for this step: Review deployed machines against the + updated \c{public} \c{buildtab} and remove those that are no longer used: \ cd private/buildos/ + grep '.*' .../brep-buildtab | cut -d' ' -f1 | sort -u ./ls-machines -c public ~/work/build2/buildos/remove-machine <host> <machine> \ - Then move now legacy machines to the \"legacy\" build host. + Then move now legacy machines to the \"legacy\" build host: + + \ + grep 'legacy' .../brep-buildtab | cut -d' ' -f1 | sort -u + \ Also review deployed machines against the latest available versions and upgrade those that are not the latest: @@ -591,12 +744,18 @@ distribution from \c{etc/stage} and add the pre-distributed packages \ cd private/buildos/ - ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" + ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" | sort ./ls-machines -c public ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver \ + Finally, add any new machines: + + \ + grep -v 'legacy' .../brep-buildtab | cut -d' ' -f1 | sort -u + \ + Uncomment the \c{public} toolchain in the build host configuration and regenerate. The only remaining step is to reboot (not yet): @@ -606,7 +765,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages \h#pub-dist|Publish distribution| - Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to \c{public}, + Change \c{build2_repo} in \c{build2-toolchain} \c{buildfile} to \c{public}, commit, and publish the distribution (this also cleans/disables the \c{queue} toolchain): @@ -616,6 +775,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages \h#pub-pkg|Publish packages| + @@ Need to add --min-bpkg-version 0.13.0 + Move packages (manually for now) from the \c{queue} to \c{public} \c{git} repository, including ownership information. Move old/replaced/FTB packages either to legacy or delete. @@ -627,7 +788,7 @@ distribution from \c{etc/stage} and add the pre-distributed packages Note that once published, the existing install instructions/download links are no longer usable, so do not linger (in fact, may make sense to update Download and Install pages before publishing packages and - sync only them immediately after). + only sync them immediately after). \h#start-public|Start \c{public} builds| @@ -642,6 +803,9 @@ distribution from \c{etc/stage} and add the pre-distributed packages we just smoke-test each script on its \"primary\" platform and make sure \c{public} URLs/repositories are used. + Also test building an old package with the previous version of the + toolchain. + \h#web|Update web| \ul| @@ -747,13 +911,17 @@ distribution from \c{etc/stage} and add the pre-distributed packages Essentially, the same steps as in \l{#version-release Change to release version} (but no tagging). + Revert changes to install scripts if upgrade was disabled. + \h#stage-machines-reopen|Update \c{stage} \c{buildtab}s and build machines| Essentially, the same steps as in \l{#public-machines Update \c{public} \c{buildtab}s and build machines} but for stage. Some differences: Clean \c{buildtab}s (both \c{stage} and CI) by removing no longer relevant - configurations, moving some to \c{legacy}, etc. + configurations, moving some to \c{legacy}, etc. NOTE: we now keep the + same set of machines as public until next release so that we can build + public with stage for testing. More generally, this is the time to do sweeping changes such as renaming machines/configurations, adjusting classes, etc. This is also the time to @@ -765,6 +933,8 @@ distribution from \c{etc/stage} and add the pre-distributed packages \c{baseutils} and \c{mingw}; the idea is that we will start with those and maybe upgrade later). + Review \c{etc/stage} for any packages to enabled/disable. + Then cleanup and restage: \ @@ -782,6 +952,14 @@ distribution from \c{etc/stage} and add the pre-distributed packages Upgrade \c{brep} on \c{stage}. + Review \c{etc/stage-queue} and restage queue.stage: + + \ + rm -r staging/repository/1/*/ + + etc/stage-queue + \ + \h#commit-reopen|Commit and push \c{etc/} and \c{private/}.| Commit and push changes to \c{etc/} and \c{private/}. @@ -830,6 +1008,20 @@ distribution from \c{etc/stage} and add the pre-distributed packages Adjust \c{bdep-new}-generated projects to use any newly available features. +\h1#re-review|Re-review \c{@@}| + + Review \c{@@} notes (in case some should be fixed after the release): + + \ + etc/review | less -R + \ + + At least look for \c{@@\ TMP} + + \ + etc/review | grep TMP + \ + \h1#plan|Plan| Plan the next release and update the project roadmap. |