summaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
Diffstat (limited to 'doc')
-rw-r--r--doc/.gitignore2
-rw-r--r--doc/bash-style.cli4
-rwxr-xr-xdoc/cli.sh5
-rw-r--r--doc/release.cli724
4 files changed, 733 insertions, 2 deletions
diff --git a/doc/.gitignore b/doc/.gitignore
index a5873dc..36afaee 100644
--- a/doc/.gitignore
+++ b/doc/.gitignore
@@ -1 +1,3 @@
build2-bash-style.xhtml
+build2-release.xhtml
+
diff --git a/doc/bash-style.cli b/doc/bash-style.cli
index 9e90e8a..d22cc23 100644
--- a/doc/bash-style.cli
+++ b/doc/bash-style.cli
@@ -9,7 +9,7 @@
// - Maximum <pre> line is 70 characters.
//
-"\h1|Contents|"
+"\h1|Table of Contents|"
"\$TOC$"
"
@@ -27,7 +27,7 @@ You just need to be clear on why you are doing it.
See also \l{https://google.github.io/styleguide/shell.xml Google's Bash Style
Guide} as well as \l{https://github.com/progrium/bashstyle Let's do Bash
right!}; we agree with quite a few (but not all) items in there. In particular,
-it provides a lot more rationale compared to this guide.
+the former provides a lot more rationale compared to this guide.
\h1#style|Style|
diff --git a/doc/cli.sh b/doc/cli.sh
index 44d311a..e737ee7 100755
--- a/doc/cli.sh
+++ b/doc/cli.sh
@@ -22,3 +22,8 @@ cli --generate-html --html-suffix .xhtml \
--html-prologue-file doc-prologue.xhtml \
--html-epilogue-file doc-epilogue.xhtml \
--output-prefix build2- bash-style.cli
+
+cli --generate-html --html-suffix .xhtml \
+--html-prologue-file doc-prologue.xhtml \
+--html-epilogue-file doc-epilogue.xhtml \
+--output-prefix build2- release.cli
diff --git a/doc/release.cli b/doc/release.cli
new file mode 100644
index 0000000..6639371
--- /dev/null
+++ b/doc/release.cli
@@ -0,0 +1,724 @@
+// file : doc/release.cli
+// copyright : Copyright (c) 2014-2018 Code Synthesis Ltd
+// license : MIT; see accompanying LICENSE file
+
+"\title=Release Process"
+
+// NOTES
+//
+// - Maximum <pre> line is 70 characters.
+//
+
+"\h1|Table of Contents|"
+"\$TOC$"
+
+"
+\h0#process|Process|
+
+Review the state and services list (currently on paper) for any new additions.
+Consider how/when they are updated/tested during the release process.
+
+
+\h1#stage|Stage|
+
+The staging repository is completely independent which means it must contain
+all the \c{build2} dependencies (plus a few extra packages for testing; see
+the \c{etc/stage} script for the complete list). This in turn means that if
+any of these dependencies are in the unreleased state, then they should go
+through the applicable steps in this section (e.g., updating of \c{NEWS}, etc)
+and then be queued and published (effectively released) as part of the
+\c{build2} release. Generally, however, we should strive to not unnecessarily
+bundle the release of dependencies with the release of \c{build2} to keep
+the process as streamlined as possible.
+
+\N|When unbundling the release of a dependency we need to remove its
+distribution from \c{etc/stage} and add the pre-distributed packages
+(for example, from \c{public}) to \c{staging/repository/1/}.|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|Current \c{stage} build is clean.||
+
+\h#copy|Update copyright if new year|
+
+ Use the \c{git/copyright} script. See the script header for instructions.
+
+\h#etc|Update \c{etc/git/modules}|
+
+ Review for any new modules. Remove \c{etc/} and \c{private/} from
+ \c{modules} to reduce noise during stat.
+
+\h#review|Review \c{@@}|
+
+ Review \c{@@} notes:
+
+ \
+ ./review.sh | less -R
+ \
+
+ At least look for \c{@@\ TMP}
+
+ \
+ ./review.sh | grep TMP
+ \
+
+\h#review-db|Review database schema changes|
+
+ Review database schema changelog differences in \c{bpkg}, \c{bdep}, and
+ \c{brep} compared to the previous release (tag) for any schema/data
+ migration that may be required (if something is missing, then it would also
+ have to be tested).
+
+\h#news|Update \c{NEWS} files|
+
+ Update in all projects (including dependencies) that will have a release
+ version.
+
+ See \c{etc/stage} for the complete list.
+
+
+\h#packaging|Release \c{packaging/} dependencies|
+
+ Release dependencies that have the snapshot version as described in
+ \c{private/build2-packaging.txt}.
+
+ \N|Maybe this should be done during queuing? Why do we release (but not
+ publish) these now and other dependencies later?|
+
+
+\h#dependencies|Finalize all other dependencies|
+
+ Make sure all other unreleased dependencies listed in \c{etc/stage} are
+ ready to be released. Effectively, the only remaining step should be to
+ change the version.
+
+ Do this in the dependency order and finish each off with:
+
+ \
+ git pull && bdep sync -fura && bdep test -ar
+ \
+
+
+\h#upgrade|Upgrade dependencies|
+
+ Upgrade all dependencies in the \c{build2} toolchain build:
+
+ \
+ etc/upgrade
+ \
+
+
+\h#hello|Update \c{hello/} projects|
+
+ These projects should closely track the output of \c{bdep-new(1)}. The
+ recommended procedure is to generate a new project with the same name and
+ then review/resolve the differences with \c{diff\ -ru}.
+
+ Every change in these projects must be accompanied by a revision increment
+ and a re-tag. This should be managed with \c{bdep-release(1)}:
+
+ \
+ # make changes
+ git add .
+ bdep release --revision --show-push
+ # review commit
+ git push ...
+ \
+
+ Once done, run the \c{intro} scripts and review any changes in the output
+ (this information will be helpful on the next step):
+
+ \
+ cd etc
+
+ ./intro2-tldr 2>&1 | tee intro2-tldr.out
+ diff -u intro2-tldr.orig intro2-tldr.out # Or use gitk.
+ mv intro2-tldr.out intro2-tldr.orig
+
+ ./intro2-tour 2>&1 | tee intro2-tour.out
+ diff -u intro2-tour.orig intro2-tour.out # Or use gitk.
+ mv intro2-tour.out intro2-tour.orig
+ \
+
+\h#doc|Review documentation|
+
+ Review the following documentation for (1) sample output changes and (2)
+ still being relevant/making sense.
+
+ \N|Ideally this should be done during development but it's easy to forget.
+ Also, check if there is any new documentation that has been added that is
+ not on the below list.|
+
+ \ul|
+
+ \li|Install guide: 1 & 2.|
+
+ \li|Toolchain introduction: 1 & 2 (use \c{intro} script output).|
+
+ \li|Introduction in the build system manual: 1 (uses \c{bdep-new(1)}
+ output).|
+
+ \li|Testscript manual: 1.||
+
+
+\h#submod|Update all submodules|
+
+ Make sure all working trees are clean. Review \c{sub_modules} in
+ \c{etc/git/modules} for any missing new modules, then:
+
+ \N|\c{etc/} and \c{private/} need to be committed manually.|
+
+ \
+ ./modup.sh
+ ./commit.sh # If changes.
+ ./push.sh
+
+ cd build2-toolchain
+
+ git submodule update --remote --checkout
+ git submodule foreach git submodule update --init --recursive
+
+ git status # There should only be 'new commits'; see README-GIT
+
+ git commit -a -m \"Update submodules\"
+ git push
+ \
+
+\h#stage-machines|Update \c{stage} \c{buildtab}s and build machines|
+
+ Review \c{stage} \c{buildtab} for any configurations to drop (for example,
+ an intermediate version of a compiler).
+
+ Based on these changes update \c{stage} CI \c{buildtab}, which is a subset
+ of the \c{stage} configurations (and is a base for the \c{queue}/\c{public}
+ configurations).
+
+ Review deployed machines against the updated \c{stage} \c{buildtab} and
+ remove those that are no longer used:
+
+ \
+ cd private/buildos/
+
+ ./ls-machines -c stage -c devel
+
+ ~/work/buildos/remove-machine <host> <machine>
+ \
+
+ Also review deployed machines against the latest available versions and
+ upgrade those that are not the latest:
+
+ \
+ cd private/buildos/
+
+ ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\"
+ ./ls-machines -c stage -c devel
+
+ ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver
+ \
+
+
+\h#restage|Restage|
+
+ Confirm \c{stage} and \c{devel} \c{buildos} images, hardware-specific
+ configurations are the same.
+
+ Review \c{staging/0/} and \c{staging/repository/1/} for anything stray.
+
+ Update all submodules in \c{build2-toolchain}:
+
+ \
+ git submodule update --remote --checkout
+ \
+
+ Restage with \c{baseutils}/\c{mingw} regeneration:
+
+ \
+ etc/stage -b
+ \
+
+ Upgrade \c{brep} on \c{stage} and sync latest \c{buildtab}s.
+
+ Verify \c{stage} build is clean, nothing is unbuilt.
+
+
+\h#install-stage|Test install scripts|
+
+ Test \l{https://stage.build2.org/0/ \c{stage} install scripts}, including
+ upgrading, as described in \c{private/install/testing.txt}.
+
+
+\h1#queue|Queue|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|Final \c{stage} build is clean.|
+
+\li|Build with the \c{queue} toolchain is inactive.||
+
+
+\h#queue-repo|Review \c{queue} repository|
+
+ Pull, review, and clean the \c{queue} \c{git} repository for any stale
+ packages or anything (ownership, etc) that may conflict with packages being
+ released.
+
+ Also clean toolchain distribution:
+
+ \
+ rm -rf cppget.org/queue/0/*
+ \
+
+\h#version-release|Change to release version|
+
+ Change to the release version in all the packages being released and tag
+ (see \c{etc/stage} for the list). Use \c{bdep-release(1)} unless a custom
+ versioning/tagging scripts are used:
+
+ \
+ bdep release --no-open --show-push [--alpha|--beta]
+ # review commit
+ git push ...
+ \
+
+ Do this in the dependency order and finish each off with:
+
+ \
+ bdep sync -fura && bdep update
+ \
+
+ For the \c{build2} packages (commit but don't push after each step if
+ required):
+
+ \ul|
+
+ \li|Close schema versions in \c{bpkg}, \c{bdep}, and \c{brep}.|
+
+ \li|Change \c{BUILD2_STAGE} in \c{build2/build2/config.hxx.in} to \c{false}.|
+
+ \li|Change version by updating (including with new modules) and then
+ executing:
+
+ \
+ etc/version
+ ./commit.sh
+ git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\"
+ \
+
+ |
+
+ \li|Tag by executing \c{tag.sh\ <version>}.|
+
+ \li|Regenerate documentation in each package.|
+
+ \li|Upgrade all dependencies in configure-only mode by executing
+ \c{etc/upgrade\ -c}.|
+
+ \li|Trigger regeneration of version files (might require several runs
+ to \"close off\"):
+
+ \
+ BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7-asan/
+ \
+
+ If using GCC prior to 8 then might also need explicit version target
+ upgrades:
+
+ \
+ BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/.../hxx{version}
+ \
+
+ |
+
+ \li|Regenerate ODB in relevant packages passing upgraded configuration
+ path explicitly (\c{bdep} is not runnable):
+
+ \
+ ./odb.sh ~/work/build2/builds/gcc7-asan/
+ \
+
+ |
+
+ \li|Finish upgrading all dependencies by executing in the upgraded
+ configuration:
+
+ \
+ BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/
+ \
+
+ ||
+
+ Verify key tests pass (in particular, the \c{bdep} tests will now be running
+ against \c{public} services):
+
+ \
+ b test: build2/ bpkg/ bdep/
+ \
+
+ \N|We could have queued after this step before preparing
+ \c{build2-toolchain}. However, splitting this seems to only increase
+ complexity without any major benefits (it's not hard to update submodules
+ and regenerated \c{build2-toolchain} if there are any issues with
+ packages). Plus we can start testing install scripts, etc.|
+
+ Next push the above changes and update all submodules in
+ \c{build2-toolchain}:
+
+ \
+ ./push.sh
+
+ cd build2-toolchain
+
+ git submodule update --remote --checkout
+
+ git status # There should only be 'new commits'; see README-GIT
+
+ git commit -a -m \"Update submodules\"
+ \
+
+ As well as (commit after each step if required):
+
+ \ul|
+
+ \li|Regenerate documentation in each package inside as well as in
+ \c{build2-toolchain} itself.|
+
+ \li|Update ODB by copying relevant files from the previous step (trust
+ me, this is the easy way).|
+
+ \li|Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to
+ \c{queue}.||
+
+ Finally, push the changes:
+
+ \
+ git push
+ \
+
+\h#queuing|Queue|
+
+ Prepare packages and the toolchain distribution:
+
+ \
+ etc/stage -q -b
+ \
+
+ Sort non-alpha packages from \c{cppget.org/queue/1/alpha/} into appropriate
+ sections (we could probably automate this similar to \c{bdep-release(1)}).
+
+ Note also that we assume all the packages already have the corresponding
+ ownership information either in \c{queue} or \c{public}. However, if any
+ new packages were added, then that will have to be added as well.
+
+ Commit and push queue repository:
+
+ \
+ cd cppget.org/queue/
+ git add .
+ git ci -m \"Queue build2 toolchain X.Y.Z\"
+ git push
+ \
+
+\h#build-public|Verify queued packages build with \c{public}|
+
+ This makes sure that the new version can be built with the old toolchain.
+
+ While all the packages should build (except perhaps \c{libhello} which,
+ being based on the latest \c{bdep-new(1)} output, may use new features),
+ some tests may fail. In particular, we will be running \c{bpkg} tests using
+ a previous version of the build system and \c{bdep} tests also using a
+ previous version of \c{bpkg}. None of that is or will be supported. So
+ for now we manually review all the failed builds for anything suspicious
+ and in the future we may skip tests if the tool versions mismatch.
+
+
+\h#install-queue|Test install scripts|
+
+ Test \l{https://download.build2.org/queue/ \c{queue} install scripts}. Here
+ we just smoke-test each script on its \"primary\" platform and make sure
+ \c{queue} URLs/repositories are used.
+
+
+\h#upgrade-brep-bpub|Upgrade \c{brep} and \c{bpub} on \c{cppget.org}|
+
+ Upgrade \c{brep} and \c{bpub} using the queued toolchain according
+ to \c{private/brep-bpub-upgrade.txt}.
+
+ After upgrade, drop all package builds for \c{public} and \c{queue} and
+ verify there are no regressions. This makes sure that the \c{brep} services
+ in the new version are usable with the old toolchain. If there are issues,
+ they have to be fixed by publishing/queuing revisions.
+
+ \N|The \c{bdep} tests that exercise CI and publishing services do it in the
+ simulation mode. To properly test that these services are compatible with
+ the old version of the toolchain we would need to CI/publish a real test
+ package manually.|
+
+
+\h#start-queue|Start \c{queue} builds|
+
+ Update \c{queue} \c{buildtab} based on the \c{stage} CI \c{buildtab}
+ (normally just a copy sans the sanitized toolchain configurations).
+
+ Adjust \c{stage} and \c{devel} build host configurations to enable the
+ \c{queue} toolchain. Shift most instances from \c{stage} to \c{queue}
+ in the hardware class-specific configurations. Regenerate affected
+ configurations and reboot build hosts:
+
+ \
+ cd private/buildos/
+
+ ./gen-config stage <class>
+ ./gen-config devel <class>
+
+ ./po-hosts -r -c stage -c devel
+ \
+
+ Verify both \c{queue} and \c{public} builds with the queued toolchain,
+ fixing any breakages with revisions, moving to legacy, and/or queuing
+ replacements. We can only proceed further once we have a \"resolution\"
+ for every (newly) broken package.
+
+
+\h#stop-queue|Stop \c{queue} builds|
+
+ Adjust \c{stage} and \c{devel} build host configurations to disable the
+ \c{queue} toolchain (comment out). Regenerate affected configurations and
+ reboot build hosts as on the previous step.
+
+\h1#public|Public|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|A \"resolution\" is queued or published for every (newly) broken package.||
+
+\h#public-machines|Update \c{public} \c{buildtab}s and build machines|
+
+ Poweroff the old set of \c{public} build hosts:
+
+ \
+ ./po-hosts -c public
+ \
+
+ Update \c{public} \c{buildtab}s based on the \c{queue} \c{buildtab}
+ (normally just a copy).
+
+ Adjust build host configurations and add/remove new/old machines.
+
+ Replace the \c{public} \c{buildos} image on \c{build-cache} with the
+ one for \c{stage}.
+
+ Comment out the \c{public} toolchain in the build host configuration
+ (effectively making it a no-toolchain configuration) and power on the new
+ set of \c{public} build hosts.
+
+ Review deployed machines against the updated \c{public} \c{buildtab} and
+ remove those that are no longer used:
+
+ \
+ cd private/buildos/
+
+ ./ls-machines -c public
+
+ ~/work/buildos/remove-machine <host> <machine>
+ \
+
+ Also review deployed machines against the latest available versions and
+ upgrade those that are not the latest:
+
+ \
+ cd private/buildos/
+
+ ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\"
+ ./ls-machines -c public
+
+ ~/work/build2/buildos/upload-machine <host> .../new-ver .../old-ver
+ \
+
+ Uncomment the \c{public} toolchain in the build host configuration. The
+ only remaining step is to reboot (not yet):
+
+ \
+ ./po-hosts -r -c public
+ \
+
+\h#pub-dist|Publish distribution|
+
+ Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to \c{public}
+ and publish the distribution (this also cleans/disables the \c{queue}
+ toolchain):
+
+ \
+ etc/stage -p
+ \
+
+\h#pub-pkg|Publish packages|
+
+ Move packages (manually for now) from the \c{queue} to \c{public} \c{git}
+ repository, including ownership information. Move old/replaced/FTB
+ packages either to legacy or delete.
+
+ Commit everything (see the commit log for procedure @@ this needs
+ automation) and push (which should trigger auto-publish). Note: commit and
+ push both \c{queue} and \c{public} \c{git} repositories.
+
+ Note that once published, the existing install instructions/download
+ links are no longer usable, so do not linger.
+
+\h#start-public|Start \c{public} builds|
+
+ Reboot \c{public} build hosts. They should next bootstrap and proceed
+ with building all packages.
+
+\h#install-public|Test install scripts|
+
+ Test \l{https://download.build2.org/ \c{public} install scripts}. Here
+ we just smoke-test each script on its \"primary\" platform and make sure
+ \c{public} URLs/repositories are used.
+
+\h#web|Update web|
+
+ \ul|
+
+ \li|Write release notes, use placeholder for announcement URL (or guess).
+ Add link from the \c{doc.cli}. Add blog entry.|
+
+ \li|Add any new FAQ entries or any other updates (main page, etc). Any new
+ documentation to link from the Doc page?|
+
+ \li|Update the Download page.
+
+ \
+ cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256`
+ \
+
+ |
+
+ \li|Update the Install page.
+
+ \
+ cat cppget.org/public/0/toolchain.sha256
+ \
+
+ |
+
+ \li|Regenerate documentation in \c{private/} for all hosts:
+
+ \
+ cd private/
+
+ cd <host>/www/
+ ./cli.sh
+ \
+
+ |
+
+ \li|Test locally, publish, and test remote:
+
+ \
+ cd private/
+ ./publish --dry-run
+ ./publish
+ \
+
+ ||
+
+
+\h#ann|Announce|
+
+ \ul|
+
+ \li|Write and send the announcement to the mailing lists.
+
+ \
+ cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256`
+ \
+
+ Add \c{reply-to:} header for \c{users@} announcement.|
+
+ \li|Patch (or verify) the announcement URL in release notes, re-publish.|
+
+ \li|Announce on reddit, slack, and other relevant places.||
+
+\h#tag|Commit, tag, and push the rest.|
+
+ Tag \c{build2-toolchain}:
+
+ \
+ cd build2-toolchain
+ git tag -a vX.Y.Z -m \"Tag version X.Y.Z\"
+ git push --follow-tags
+ \
+
+ Add \c{etc/} and \c{private/} back to \c{modules} in \c{etc/git/modules}
+ (essentially reverse \l{#etc Update \c{etc/git/modules}}).
+
+ Finalize changes, commit, tag, and push \c{style/}, \c{etc/}, and
+ \c{private/}.
+
+ Snapshot \c{buildos} subvolumes:
+
+ \
+ btrfs subvolume snapshot buildos-3 buildos-3-X.Y.Z
+ btrfs subvolume snapshot buildos-6 buildos-6-X.Y.Z
+ \
+
+\h1#reopen|Reopen|
+
+\h#version-snapshot|Change to snapshot version|
+
+ Change to the snapshot version in all the released packages. Use
+ \c{bdep-release(1)} unless a custom versioning script is used:
+
+ \
+ bdep release --open --show-push [--open-*]
+ # review commit
+ git push ...
+ \
+
+ Essentially, the same steps as in \l{#version-release Change to
+ release version} (but no tagging).
+
+\h#stage-machines-reopen|Update \c{stage} \c{buildtab}s and build machines|
+
+ Essentially, the same steps as in \l{#public-machines Update \c{public}
+ \c{buildtab}s and build machines} but for stage. Some differences:
+
+ Clean \c{buildtab}s (both \c{stage} and CI) by removing no longer relevant
+ configurations, moving some to \c{legacy}, etc.
+
+ More generally, this is the time to do sweeping changes such as renaming
+ machines/configurations, adjusting classes, etc. This is also the time to
+ rebalance machines across available hosts.
+
+\h#restage-reopen|Restage|
+
+ Make symlinks for the new version in \c{private/baseutils} (for both
+ \c{baseutils} and \c{mingw}; the idea is that we will start with those and
+ maybe upgrade later).
+
+ Then cleanup and restage:
+
+ \
+ rm -r staging/0/*
+ rm -r staging/repository/1/*/
+
+ etc/stage -b
+ \
+
+\h#start-stage|Start \c{stage} builds|
+
+ Reboot \c{stage} build hosts. They should next bootstrap and proceed with
+ building all the staged packages. Make sure all the builds of the new
+ development snapshot are successful and there is nothing unbuilt.
+
+\h#commit-reopen|Commit and push \c{etc/} and \c{private/}.|
+
+ Commit and push changes to \c{etc/} and \c{private/}.
+
+"