diff options
Diffstat (limited to 'doc/release.cli')
| -rw-r--r-- | doc/release.cli | 724 |
1 files changed, 724 insertions, 0 deletions
diff --git a/doc/release.cli b/doc/release.cli new file mode 100644 index 0000000..6639371 --- /dev/null +++ b/doc/release.cli @@ -0,0 +1,724 @@ +// file : doc/release.cli +// copyright : Copyright (c) 2014-2018 Code Synthesis Ltd +// license : MIT; see accompanying LICENSE file + +"\title=Release Process" + +// NOTES +// +// - Maximum <pre> line is 70 characters. +// + +"\h1|Table of Contents|" +"\$TOC$" + +" +\h0#process|Process| + +Review the state and services list (currently on paper) for any new additions. +Consider how/when they are updated/tested during the release process. + + +\h1#stage|Stage| + +The staging repository is completely independent which means it must contain +all the \c{build2} dependencies (plus a few extra packages for testing; see +the \c{etc/stage} script for the complete list). This in turn means that if +any of these dependencies are in the unreleased state, then they should go +through the applicable steps in this section (e.g., updating of \c{NEWS}, etc) +and then be queued and published (effectively released) as part of the +\c{build2} release. Generally, however, we should strive to not unnecessarily +bundle the release of dependencies with the release of \c{build2} to keep +the process as streamlined as possible. + +\N|When unbundling the release of a dependency we need to remove its +distribution from \c{etc/stage} and add the pre-distributed packages +(for example, from \c{public}) to \c{staging/repository/1/}.| + +\b{Pre-Conditions:} + +\ul| + +\li|Current \c{stage} build is clean.|| + +\h#copy|Update copyright if new year| + + Use the \c{git/copyright} script. See the script header for instructions. + +\h#etc|Update \c{etc/git/modules}| + + Review for any new modules. Remove \c{etc/} and \c{private/} from + \c{modules} to reduce noise during stat. + +\h#review|Review \c{@@}| + + Review \c{@@} notes: + + \ + ./review.sh | less -R + \ + + At least look for \c{@@\ TMP} + + \ + ./review.sh | grep TMP + \ + +\h#review-db|Review database schema changes| + + Review database schema changelog differences in \c{bpkg}, \c{bdep}, and + \c{brep} compared to the previous release (tag) for any schema/data + migration that may be required (if something is missing, then it would also + have to be tested). + +\h#news|Update \c{NEWS} files| + + Update in all projects (including dependencies) that will have a release + version. + + See \c{etc/stage} for the complete list. + + +\h#packaging|Release \c{packaging/} dependencies| + + Release dependencies that have the snapshot version as described in + \c{private/build2-packaging.txt}. + + \N|Maybe this should be done during queuing? Why do we release (but not + publish) these now and other dependencies later?| + + +\h#dependencies|Finalize all other dependencies| + + Make sure all other unreleased dependencies listed in \c{etc/stage} are + ready to be released. Effectively, the only remaining step should be to + change the version. + + Do this in the dependency order and finish each off with: + + \ + git pull && bdep sync -fura && bdep test -ar + \ + + +\h#upgrade|Upgrade dependencies| + + Upgrade all dependencies in the \c{build2} toolchain build: + + \ + etc/upgrade + \ + + +\h#hello|Update \c{hello/} projects| + + These projects should closely track the output of \c{bdep-new(1)}. The + recommended procedure is to generate a new project with the same name and + then review/resolve the differences with \c{diff\ -ru}. + + Every change in these projects must be accompanied by a revision increment + and a re-tag. This should be managed with \c{bdep-release(1)}: + + \ + # make changes + git add . + bdep release --revision --show-push + # review commit + git push ... + \ + + Once done, run the \c{intro} scripts and review any changes in the output + (this information will be helpful on the next step): + + \ + cd etc + + ./intro2-tldr 2>&1 | tee intro2-tldr.out + diff -u intro2-tldr.orig intro2-tldr.out # Or use gitk. + mv intro2-tldr.out intro2-tldr.orig + + ./intro2-tour 2>&1 | tee intro2-tour.out + diff -u intro2-tour.orig intro2-tour.out # Or use gitk. + mv intro2-tour.out intro2-tour.orig + \ + +\h#doc|Review documentation| + + Review the following documentation for (1) sample output changes and (2) + still being relevant/making sense. + + \N|Ideally this should be done during development but it's easy to forget. + Also, check if there is any new documentation that has been added that is + not on the below list.| + + \ul| + + \li|Install guide: 1 & 2.| + + \li|Toolchain introduction: 1 & 2 (use \c{intro} script output).| + + \li|Introduction in the build system manual: 1 (uses \c{bdep-new(1)} + output).| + + \li|Testscript manual: 1.|| + + +\h#submod|Update all submodules| + + Make sure all working trees are clean. Review \c{sub_modules} in + \c{etc/git/modules} for any missing new modules, then: + + \N|\c{etc/} and \c{private/} need to be committed manually.| + + \ + ./modup.sh + ./commit.sh # If changes. + ./push.sh + + cd build2-toolchain + + git submodule update --remote --checkout + git submodule foreach git submodule update --init --recursive + + git status # There should only be 'new commits'; see README-GIT + + git commit -a -m \"Update submodules\" + git push + \ + +\h#stage-machines|Update \c{stage} \c{buildtab}s and build machines| + + Review \c{stage} \c{buildtab} for any configurations to drop (for example, + an intermediate version of a compiler). + + Based on these changes update \c{stage} CI \c{buildtab}, which is a subset + of the \c{stage} configurations (and is a base for the \c{queue}/\c{public} + configurations). + + Review deployed machines against the updated \c{stage} \c{buildtab} and + remove those that are no longer used: + + \ + cd private/buildos/ + + ./ls-machines -c stage -c devel + + ~/work/buildos/remove-machine <host> <machine> + \ + + Also review deployed machines against the latest available versions and + upgrade those that are not the latest: + + \ + cd private/buildos/ + + ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" + ./ls-machines -c stage -c devel + + ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver + \ + + +\h#restage|Restage| + + Confirm \c{stage} and \c{devel} \c{buildos} images, hardware-specific + configurations are the same. + + Review \c{staging/0/} and \c{staging/repository/1/} for anything stray. + + Update all submodules in \c{build2-toolchain}: + + \ + git submodule update --remote --checkout + \ + + Restage with \c{baseutils}/\c{mingw} regeneration: + + \ + etc/stage -b + \ + + Upgrade \c{brep} on \c{stage} and sync latest \c{buildtab}s. + + Verify \c{stage} build is clean, nothing is unbuilt. + + +\h#install-stage|Test install scripts| + + Test \l{https://stage.build2.org/0/ \c{stage} install scripts}, including + upgrading, as described in \c{private/install/testing.txt}. + + +\h1#queue|Queue| + +\b{Pre-Conditions:} + +\ul| + +\li|Final \c{stage} build is clean.| + +\li|Build with the \c{queue} toolchain is inactive.|| + + +\h#queue-repo|Review \c{queue} repository| + + Pull, review, and clean the \c{queue} \c{git} repository for any stale + packages or anything (ownership, etc) that may conflict with packages being + released. + + Also clean toolchain distribution: + + \ + rm -rf cppget.org/queue/0/* + \ + +\h#version-release|Change to release version| + + Change to the release version in all the packages being released and tag + (see \c{etc/stage} for the list). Use \c{bdep-release(1)} unless a custom + versioning/tagging scripts are used: + + \ + bdep release --no-open --show-push [--alpha|--beta] + # review commit + git push ... + \ + + Do this in the dependency order and finish each off with: + + \ + bdep sync -fura && bdep update + \ + + For the \c{build2} packages (commit but don't push after each step if + required): + + \ul| + + \li|Close schema versions in \c{bpkg}, \c{bdep}, and \c{brep}.| + + \li|Change \c{BUILD2_STAGE} in \c{build2/build2/config.hxx.in} to \c{false}.| + + \li|Change version by updating (including with new modules) and then + executing: + + \ + etc/version + ./commit.sh + git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\" + \ + + | + + \li|Tag by executing \c{tag.sh\ <version>}.| + + \li|Regenerate documentation in each package.| + + \li|Upgrade all dependencies in configure-only mode by executing + \c{etc/upgrade\ -c}.| + + \li|Trigger regeneration of version files (might require several runs + to \"close off\"): + + \ + BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7-asan/ + \ + + If using GCC prior to 8 then might also need explicit version target + upgrades: + + \ + BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/.../hxx{version} + \ + + | + + \li|Regenerate ODB in relevant packages passing upgraded configuration + path explicitly (\c{bdep} is not runnable): + + \ + ./odb.sh ~/work/build2/builds/gcc7-asan/ + \ + + | + + \li|Finish upgrading all dependencies by executing in the upgraded + configuration: + + \ + BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/ + \ + + || + + Verify key tests pass (in particular, the \c{bdep} tests will now be running + against \c{public} services): + + \ + b test: build2/ bpkg/ bdep/ + \ + + \N|We could have queued after this step before preparing + \c{build2-toolchain}. However, splitting this seems to only increase + complexity without any major benefits (it's not hard to update submodules + and regenerated \c{build2-toolchain} if there are any issues with + packages). Plus we can start testing install scripts, etc.| + + Next push the above changes and update all submodules in + \c{build2-toolchain}: + + \ + ./push.sh + + cd build2-toolchain + + git submodule update --remote --checkout + + git status # There should only be 'new commits'; see README-GIT + + git commit -a -m \"Update submodules\" + \ + + As well as (commit after each step if required): + + \ul| + + \li|Regenerate documentation in each package inside as well as in + \c{build2-toolchain} itself.| + + \li|Update ODB by copying relevant files from the previous step (trust + me, this is the easy way).| + + \li|Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to + \c{queue}.|| + + Finally, push the changes: + + \ + git push + \ + +\h#queuing|Queue| + + Prepare packages and the toolchain distribution: + + \ + etc/stage -q -b + \ + + Sort non-alpha packages from \c{cppget.org/queue/1/alpha/} into appropriate + sections (we could probably automate this similar to \c{bdep-release(1)}). + + Note also that we assume all the packages already have the corresponding + ownership information either in \c{queue} or \c{public}. However, if any + new packages were added, then that will have to be added as well. + + Commit and push queue repository: + + \ + cd cppget.org/queue/ + git add . + git ci -m \"Queue build2 toolchain X.Y.Z\" + git push + \ + +\h#build-public|Verify queued packages build with \c{public}| + + This makes sure that the new version can be built with the old toolchain. + + While all the packages should build (except perhaps \c{libhello} which, + being based on the latest \c{bdep-new(1)} output, may use new features), + some tests may fail. In particular, we will be running \c{bpkg} tests using + a previous version of the build system and \c{bdep} tests also using a + previous version of \c{bpkg}. None of that is or will be supported. So + for now we manually review all the failed builds for anything suspicious + and in the future we may skip tests if the tool versions mismatch. + + +\h#install-queue|Test install scripts| + + Test \l{https://download.build2.org/queue/ \c{queue} install scripts}. Here + we just smoke-test each script on its \"primary\" platform and make sure + \c{queue} URLs/repositories are used. + + +\h#upgrade-brep-bpub|Upgrade \c{brep} and \c{bpub} on \c{cppget.org}| + + Upgrade \c{brep} and \c{bpub} using the queued toolchain according + to \c{private/brep-bpub-upgrade.txt}. + + After upgrade, drop all package builds for \c{public} and \c{queue} and + verify there are no regressions. This makes sure that the \c{brep} services + in the new version are usable with the old toolchain. If there are issues, + they have to be fixed by publishing/queuing revisions. + + \N|The \c{bdep} tests that exercise CI and publishing services do it in the + simulation mode. To properly test that these services are compatible with + the old version of the toolchain we would need to CI/publish a real test + package manually.| + + +\h#start-queue|Start \c{queue} builds| + + Update \c{queue} \c{buildtab} based on the \c{stage} CI \c{buildtab} + (normally just a copy sans the sanitized toolchain configurations). + + Adjust \c{stage} and \c{devel} build host configurations to enable the + \c{queue} toolchain. Shift most instances from \c{stage} to \c{queue} + in the hardware class-specific configurations. Regenerate affected + configurations and reboot build hosts: + + \ + cd private/buildos/ + + ./gen-config stage <class> + ./gen-config devel <class> + + ./po-hosts -r -c stage -c devel + \ + + Verify both \c{queue} and \c{public} builds with the queued toolchain, + fixing any breakages with revisions, moving to legacy, and/or queuing + replacements. We can only proceed further once we have a \"resolution\" + for every (newly) broken package. + + +\h#stop-queue|Stop \c{queue} builds| + + Adjust \c{stage} and \c{devel} build host configurations to disable the + \c{queue} toolchain (comment out). Regenerate affected configurations and + reboot build hosts as on the previous step. + +\h1#public|Public| + +\b{Pre-Conditions:} + +\ul| + +\li|A \"resolution\" is queued or published for every (newly) broken package.|| + +\h#public-machines|Update \c{public} \c{buildtab}s and build machines| + + Poweroff the old set of \c{public} build hosts: + + \ + ./po-hosts -c public + \ + + Update \c{public} \c{buildtab}s based on the \c{queue} \c{buildtab} + (normally just a copy). + + Adjust build host configurations and add/remove new/old machines. + + Replace the \c{public} \c{buildos} image on \c{build-cache} with the + one for \c{stage}. + + Comment out the \c{public} toolchain in the build host configuration + (effectively making it a no-toolchain configuration) and power on the new + set of \c{public} build hosts. + + Review deployed machines against the updated \c{public} \c{buildtab} and + remove those that are no longer used: + + \ + cd private/buildos/ + + ./ls-machines -c public + + ~/work/buildos/remove-machine <host> <machine> + \ + + Also review deployed machines against the latest available versions and + upgrade those that are not the latest: + + \ + cd private/buildos/ + + ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" + ./ls-machines -c public + + ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver + \ + + Uncomment the \c{public} toolchain in the build host configuration. The + only remaining step is to reboot (not yet): + + \ + ./po-hosts -r -c public + \ + +\h#pub-dist|Publish distribution| + + Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to \c{public} + and publish the distribution (this also cleans/disables the \c{queue} + toolchain): + + \ + etc/stage -p + \ + +\h#pub-pkg|Publish packages| + + Move packages (manually for now) from the \c{queue} to \c{public} \c{git} + repository, including ownership information. Move old/replaced/FTB + packages either to legacy or delete. + + Commit everything (see the commit log for procedure @@ this needs + automation) and push (which should trigger auto-publish). Note: commit and + push both \c{queue} and \c{public} \c{git} repositories. + + Note that once published, the existing install instructions/download + links are no longer usable, so do not linger. + +\h#start-public|Start \c{public} builds| + + Reboot \c{public} build hosts. They should next bootstrap and proceed + with building all packages. + +\h#install-public|Test install scripts| + + Test \l{https://download.build2.org/ \c{public} install scripts}. Here + we just smoke-test each script on its \"primary\" platform and make sure + \c{public} URLs/repositories are used. + +\h#web|Update web| + + \ul| + + \li|Write release notes, use placeholder for announcement URL (or guess). + Add link from the \c{doc.cli}. Add blog entry.| + + \li|Add any new FAQ entries or any other updates (main page, etc). Any new + documentation to link from the Doc page?| + + \li|Update the Download page. + + \ + cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256` + \ + + | + + \li|Update the Install page. + + \ + cat cppget.org/public/0/toolchain.sha256 + \ + + | + + \li|Regenerate documentation in \c{private/} for all hosts: + + \ + cd private/ + + cd <host>/www/ + ./cli.sh + \ + + | + + \li|Test locally, publish, and test remote: + + \ + cd private/ + ./publish --dry-run + ./publish + \ + + || + + +\h#ann|Announce| + + \ul| + + \li|Write and send the announcement to the mailing lists. + + \ + cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256` + \ + + Add \c{reply-to:} header for \c{users@} announcement.| + + \li|Patch (or verify) the announcement URL in release notes, re-publish.| + + \li|Announce on reddit, slack, and other relevant places.|| + +\h#tag|Commit, tag, and push the rest.| + + Tag \c{build2-toolchain}: + + \ + cd build2-toolchain + git tag -a vX.Y.Z -m \"Tag version X.Y.Z\" + git push --follow-tags + \ + + Add \c{etc/} and \c{private/} back to \c{modules} in \c{etc/git/modules} + (essentially reverse \l{#etc Update \c{etc/git/modules}}). + + Finalize changes, commit, tag, and push \c{style/}, \c{etc/}, and + \c{private/}. + + Snapshot \c{buildos} subvolumes: + + \ + btrfs subvolume snapshot buildos-3 buildos-3-X.Y.Z + btrfs subvolume snapshot buildos-6 buildos-6-X.Y.Z + \ + +\h1#reopen|Reopen| + +\h#version-snapshot|Change to snapshot version| + + Change to the snapshot version in all the released packages. Use + \c{bdep-release(1)} unless a custom versioning script is used: + + \ + bdep release --open --show-push [--open-*] + # review commit + git push ... + \ + + Essentially, the same steps as in \l{#version-release Change to + release version} (but no tagging). + +\h#stage-machines-reopen|Update \c{stage} \c{buildtab}s and build machines| + + Essentially, the same steps as in \l{#public-machines Update \c{public} + \c{buildtab}s and build machines} but for stage. Some differences: + + Clean \c{buildtab}s (both \c{stage} and CI) by removing no longer relevant + configurations, moving some to \c{legacy}, etc. + + More generally, this is the time to do sweeping changes such as renaming + machines/configurations, adjusting classes, etc. This is also the time to + rebalance machines across available hosts. + +\h#restage-reopen|Restage| + + Make symlinks for the new version in \c{private/baseutils} (for both + \c{baseutils} and \c{mingw}; the idea is that we will start with those and + maybe upgrade later). + + Then cleanup and restage: + + \ + rm -r staging/0/* + rm -r staging/repository/1/*/ + + etc/stage -b + \ + +\h#start-stage|Start \c{stage} builds| + + Reboot \c{stage} build hosts. They should next bootstrap and proceed with + building all the staged packages. Make sure all the builds of the new + development snapshot are successful and there is nothing unbuilt. + +\h#commit-reopen|Commit and push \c{etc/} and \c{private/}.| + + Commit and push changes to \c{etc/} and \c{private/}. + +" |