diff options
Diffstat (limited to 'doc/manual.cli')
| -rw-r--r-- | doc/manual.cli | 1838 |
1 files changed, 1617 insertions, 221 deletions
diff --git a/doc/manual.cli b/doc/manual.cli index 4326a5b..2fa3248 100644 --- a/doc/manual.cli +++ b/doc/manual.cli @@ -41,12 +41,24 @@ that are executed on the build host. Inside virtual machines/containers, agent. Virtual machines and containers running a \c{bbot} instance in the worker mode are collectively called \i{build machines}. +In addition to a build machine, a build task may also require one or more +\i{auxiliary machines} which provide additional components that are required +for building or testing a package and that are impossible or impractical to +provide as part of the build machine itself. + Let's now examine the workflow in the other direction, that is, from a worker -to a controller. Once a build machine is booted (by the agent), the worker -inside connects to the TFTP server running on the build host and downloads the -\i{build task manifest}. It then proceeds to perform the build task and -uploads the \i{build result manifest} (which includes build logs) to the TFTP -server. +to a controller. Once a build machine (plus auxiliary machines, if any) are +booted (by the agent), the worker inside the build machine connects to the +TFTP server running on the build host and downloads the \i{build task +manifest}. It then proceeds to perform the build task and uploads the \i{build +artifacts archive}, if any, followed by the \i{build result manifest} (which +includes build logs) to the TFTP server. + +Unlike build machines, auxiliary machines are not expected to run \c{bbot}. +Instead, on boot, they are expected to upload to the TFTP server a list of +environment variables to propagate to the build machine (see the +\c{auxiliary-environment} task manifest value as well as \l{#arch-worker +Worker Logic} for details). Once an agent receives a build task for a specific build machine, it goes through the following steps. First, it creates a directory on its TFTP server @@ -79,23 +91,29 @@ agents. In this case we say that the \i{controller act as an agent}. The controller may also be configured to monitor build sources, such as SCM repositories, directly in which case it generates build tasks itself. -In this architecture the build results are propagated up the chain: from a -worker, to its agent, to its controller, and so on. A controller that is the -final destination of a build result uses email to notify interested parties of -the outcome. For example, \c{brep} would send a notification to the package -owner if the build failed. Similarly, a \c{bbot} controller that monitors a -\cb{git} repository would send an email to a committer if their commit caused a -build failure. The email would include a link (normally HTTP/HTTPS) to the -build logs hosted by the controller. +In this architecture the build results and optional build artifacts are +propagated up the chain: from a worker, to its agent, to its controller, and +so on. A controller that is the final destination of a build result uses email +to notify interested parties of the outcome. For example, \c{brep} would send +a notification to the package owner if the build failed. Similarly, a \c{bbot} +controller that monitors a \cb{git} repository would send an email to a +committer if their commit caused a build failure. The email would include a +link (normally HTTP/HTTPS) to the build logs hosted by the controller. The +build artifacts, such as generated binary distribution packages, are normally +made available for the interested parties to download. See \l{brep#upload +Build Artifacts Upload} for details on the \c{brep} controller's +implementation of the build artifacts upload handling. \h#arch-machine-config|Configurations| -The \c{bbot} architecture distinguishes between a \i{machine configuration} -and a \i{build configuration}. The machine configuration captures the -operating system, installed compiler toolchain, and so on. The same build -machine may be used to \"generate\" multiple \i{build configurations}. For -example, the same machine can normally be used to produce 32/64-bit and -debug/optimized builds. +The \c{bbot} architecture distinguishes between a \i{build machine +configuration}, \i{build target configuration}, and a \i{build package +configuration}. The machine configuration captures the operating system, +installed compiler toolchain, and so on. The same build machine may be used to +\"generate\" multiple \i{build target configurations}. For example, the same +machine can normally be used to produce debug/optimized builds. + +\h2#arch-machine-config-build-machine|Build Machine Configuration| The machine configuration is \i{approximately} encoded in its \i{machine name}. The machine name is a list of components separated with \c{-}. @@ -106,31 +124,31 @@ component. The encoding is approximate in a sense that it captures only what's important to distinguish in a particular \c{bbot} deployment. -The first component normally identifies the operating system and has the -following recommended form: +The first three components normally identify the architecture, operating +system, and optional variant. They have the following recommended form: \ -[<arch>_][<class>_]<os>[_<version>] +<arch>-[<class>_]<os>[_<version>][-<variant>] \ For example: \ -windows -windows_10 -windows_10.1607 -i686_windows_xp -bsd_freebsd_10 -linux_centos_6.2 -linux_ubuntu_16.04 -macos_10.12 +x86_64-windows +x86_64-windows_10 +x86_64-windows_10.1607 +x86_64-windows_10-devmode +x86_64-bsd_freebsd_10 +x86_64-linux_ubuntu_16.04 +x86_64-linux_rhel_9.2-bindist +aarch64-macos_10.12 \ -The second component normally identifies the installed compiler toolchain and +The last component normally identifies the installed compiler toolchain and has the following recommended form: \ -<id>[<version>][<vendor>][<runtime>] +<id>[_<version>][_<vendor>][_<runtime>] \ For example: @@ -140,34 +158,100 @@ gcc gcc_6 gcc_6.3 gcc_6.3_mingw_w64 +clang_3.9 clang_3.9_libc++ -clang_3.9_libstdc++ msvc_14 -msvc_14u3 -icc +msvc_14.3 +clang_15.0_msvc_msvc_17.6 +clang_16.0_llvm_msvc_17.6 \ Some examples of complete machine names: \ -windows_10-msvc_14u3 -macos_10.12-clang_10.0 -linux_ubuntu_16.04-gcc_6.3 +x86_64-windows_10-msvc_14.3 +x86_64-macos_10.12-clang_10.0 +aarch64-linux_ubuntu_16.04-gcc_6.3 +aarch64-linux_rhel_9.2-bindist-gcc_11 \ -Similarly, the build configuration is encoded in a \i{configuration name} -using the same format. As described in \l{#arch-controller Controller Logic}, -build configurations are generated from machine configurations. As a result, -it usually makes sense to have the first component identify the operating -systems and the second component \- the toolchain with the rest identifying a -particular build configuration variant, for example, optimized, sanitized, -etc. For example: +\h2#arch-machine-config-build-target-config|Build Target Configuration| + +Similarly, the build target configuration is encoded in a \i{configuration +name} using the same overall format. As described in \l{#arch-controller +Controller Logic}, target configurations are generated from machine +configurations. As a result, it usually makes sense to have the first +component identify the operating systems and the second component \- the +compiler toolchain with the rest identifying a particular target configuration +variant, for example, optimized, sanitized, etc: \ -windows-vc_14-O2 -linux-gcc_6-O3_asan +[<class>_]<os>[_<version>]-<toolchain>[-<variant>] \ +For example: + +\ +windows_10-msvc_17.6 +windows_10-msvc_17.6-O2 +windows_10-msvc_17.6-static_O2 +windows_10-msvc_17.6-relocatable +windows_10-clang_16.0_llvm_msvc_17.6_lld +linux_debian_12-clang_16_libc++-static_O3 +\ + +Note that there is no \c{<arch>} component in a build target configuration: +this information is best conveyed as part of \c{<target>} as described in +\l{#arch-controller Controller Logic}. + +\h2#arch-machine-config-build-package-config|Build Package Configuration| + +A package can be built in multiple package configurations per target +configuration. A build package configuration normally specifies the options +and/or the package configuration variables that need to be used for the +build. It may also include the information regarding the dependency packages +which need to additionally be configured. The build package configurations +originate from the package manifest \c{*-build-config}, \c{*-builds}, +\c{*-build-include}, and \c{*-build-exclude} values. See +\l{bpkg#manifest-package Package Manifest} for more information on these +values. + + +\h2#arch-machine-config-auxiliary|Auxiliary Machines and Configurations| + +Besides the build machine and the build configuration that is derived from it, +a package build may also involve one or more \i{auxiliary machines} and the +corresponding \i{auxiliary configurations}. + +An auxiliary machine provides additional components that are required for +building or testing a package and that are impossible or impractical to +provide as part of the build machine itself. For example, a package may need +access to a suitably configured database, such as PostgreSQL, in order to run +its tests. + +The auxiliary machine name follows the same overall format as the build +machine name except that the last component captures the information about the +additional component in question rather than the compiler toolchain. For +example: + +\ +x86_64-linux_debian_12-postgresql_16 +aarch64-linux_debian_12-mysql_8 +\ + +The auxiliary configuration name is automatically derived from the machine +name by removing the \c{<arch>} component. For example: + +\ +linux_debian_12-postgresql_16 +linux_debian_12-mysql_8 +\ + +\N|Note that there is no generation of multiple auxiliary configurations from +the same auxiliary machine since that would require some communication of the +desired configuration variant to the machine.| + + \h#arch-machine-header|Machine Header Manifest| @@ TODO: need ref to general manifest overview in bpkg, or, better yet, @@ -182,26 +266,41 @@ followed by the detailed description of each value in subsequent sections. id: <machine-id> name: <machine-name> summary: <string> +[role]: build|auxiliary +[ram-minimum]: <kib> +[ram-maximum]: <kib> \ For example: \ -id: windows_10-msvc_14-1.3 -name: windows_10-msvc_14 +id: x86_64-windows_10-msvc_14-1.3 +name: x86_64-windows_10-msvc_14 summary: Windows 10 build 1607 with VC 14 update 3 \ +\ +id: aarch64-linux_debian_12-postgresql_16-1.0 +name: aarch64-linux_debian_12-postgresql_16 +summary: Debian 12 with PostgreSQL 16 test user/database +role: auxiliary +ram-minimum: 2097152 +ram-maximum: 4194304 +\ + \h2#arch-machine-header-id|\c{id}| \ id: <machine-id> \ -The uniquely machine version/revision/build identifies. For virtual machines +The unique machine version/revision/build identifier. For virtual machines this can be the disk image checksum. For a container this can be UUID that is re-generated every time a container filesystem is altered. +Note that we assume that a different machine identifier is assigned on any +change that may affect the build result. + \h2#arch-machine-header-name|\c{name}| @@ -221,11 +320,34 @@ summary: <string> The one-line description of the machine. +\h2#arch-machine-header-role|\c{role}| + +\ +[role]: build|auxiliary +\ + +The machine role. If unspecified, then \c{build} is assumed. + + +\h2#arch-machine-header-ram|\c{ram-minimum}, \c{ram-maximum}| + +\ +[ram-minimum]: <kib> +[ram-maximum]: <kib> +\ + +The minimum and the maximum amount of RAM in KiB that the machine requires. +The maximum amount is interpreted as the amount beyond which there will be no +benefit. If unspecified, then it is assumed the machine will run with any +minimum amount a deployment will provide and will always benefit from more +RAM, respectively. Neither value should be \c{0}. + + \h#arch-machine|Machine Manifest| The build machine manifest contains the complete description of a build machine on the build host (see the Build OS documentation for their origin and -location). The machine manifest starts with the machine manifest header with +location). The machine manifest starts with the machine header manifest with all the header values appearing before any non-header values. The non-header part of manifest synopsis is presented next followed by the detailed description of each value in subsequent sections. @@ -304,7 +426,8 @@ changes: 1.2: increased disk size to 30GB Or: \ -changes:\ +changes: +\\ 1.1 - initial version @@ -330,14 +453,24 @@ version: <package-version> repository-url: <repository-url> [repository-type]: pkg|git|dir [trust]: <repository-fp> -[test-exclude]: <package-name>/<package-version> +[requires]: <package-requirements> +[tests]: <dependency-package> +[examples]: <dependency-package> +[benchmarks]: <dependency-package> +[dependency-checksum]: <checksum> machine: <machine-name> +[auxiliary-machine]: <machine-name> +[auxiliary-machine-<name>]: <machine-name> target: <target-triplet> [environment]: <environment-name> -[config]: <config-args> +[auxiliary-environment]: <environment-vars> +[target-config]: <tgt-config-args> +[package-config]: <pkg-config-args> +[host]: true|false [warning-regex]: <warning-regex> [interactive]: <breakpoint> +[worker-checksum]: <checksum> \ @@ -378,6 +511,7 @@ The repository type (see \c{repository-url} for details). Alternatively, the repository type can be specified as part of the URL scheme. See \l{bpkg-repository-types(1)} for details. + \h2#arch-task-trust|\c{trust}| \ @@ -396,19 +530,27 @@ some agents may only trust their internally-specified fingerprints to prevent the \"man in the middle\" attacks. -\h2#arch-task-test-exclude|\c{test-exclude}| +\h2#arch-task-requires-tests-examples-benchmarks|\c{requires, tests, examples, benchmarks}| + +The primary package manifest values that need to be known by the \c{bbot} +worker before it retrieves the primary package manifest. See +\l{bpkg#manifest-package Package Manifest} for more information on these +values. + +The controller copies these values from the primary package manifest, except +those \c{tests}, \c{examples}, and \c{benchmarks} values which should be +excluded from building due to their \c{builds}, \c{build-include}, and +\c{build-exclude} manifest values. + + +\h2#arch-task-dependency-checksum|\c{dependency-checksum}| \ -[test-exclude]: <package-name>/<package-version> +[dependency-checksum]: <checksum> \ -The separate test, example, or benchmark package to exclude from building -together with the primary package. This value may be specified multiple times. - -The controller adds this value for packages specified via the \c{tests}, -\c{examples}, and \c{benchmarks} primary package manifest values which should -be excluded from building due to their \c{builds}, \c{build-include}, and -\c{build-exclude} manifest values. +The package dependency checksum received as a part of the previous build task +result (see \l{#arch-result Result Manifest}). \h2#arch-task-machine|\c{machine}| @@ -420,6 +562,21 @@ machine: <machine-name> The name of the build machine to use. +\h2#arch-task-auxiliary-machine|\c{auxiliary-machine}| + +\ +[auxiliary-machine]: <machine-name> +[auxiliary-machine-<name>]: <machine-name> +\ + +The names of the auxiliary machines to use. These values correspond to the +\c{build-auxiliary} and \c{build-auxiliary-<name>} values in the package +manifest. While there each value specifies an auxiliary configuration pattern, +here it specifies the concrete auxiliary machine name that was picked by the +controller from the list of available auxiliary machines (sent as part of the +task request) that match this pattern. + + \h2#arch-task-target|\c{target}| \ @@ -445,30 +602,113 @@ The name of the build environment to use. See \l{#arch-worker Worker Logic} for details. -\h2#arch-task-config|\c{config}| +\h2#arch-task-auxiliary-environment|\c{auxiliary-environment}| + +\ +[auxiliary-environment]: <environment-vars> +\ + +The environment variables describing the auxiliary machines. If any +\c{auxiliary-machine*} values are specified, then after starting such +machines, the agent prepares a combined list of environment variables that +were uploaded by such machines and passes it in this value to the worker. + +The format of this value is a list of environment variable assignments +one per line, in the form: + +\ +<name>=<value> +\ + +Whitespaces before \c{<name>}, around \c{=}, and after \c{<value>} as well as +blank lines and lines that start with \c{#} are ignored. The \c{<name>} part +must only contain capital alphabetic, numeric, and \c{_} characters. The +\c{<value>} part as a whole can be single ('\ ') or double (\"\ \") +quoted. For example: + +\ +DATABASE_HOST=192.168.0.1 +DATABASE_PORT=1245 +DATABASE_USER='John \"Johnny\" Doe' +DATABASE_NAME=\" test database \" +\ + +If the corresponding machine is specified as \c{auxiliary-machine-<name>}, +then its environment variables are prefixed with capitalized \c{<name>_}. For +example: + +\ +auxiliary-machine-pgsql: x86_64-linux_debian_12-postgresql_16 +auxiliary-environment: +\\ +PGSQL_DATABASE_HOST=192.168.0.1 +PGSQL_DATABASE_PORT=1245 +... +\\ +\ + + +\h2#arch-task-target-config|\c{target-config}| + +\ +[target-config]: <tgt-config-args> +\ + +The additional target configuration options and variables. A single level of +quotes (either single or double) is removed in each value before being passed +to \c{bpkg}. For example, the following value: + +\ +target-config: config.cc.coptions=\"-O3 -stdlib='libc++'\" +\ + +Will be passed to \c{bpkg} as the following (single) argument: \ -[config]: <config-args> +config.cc.coptions=-O3 -stdlib='libc++' \ -The additional configuration options and variables. A single level of quotes +Values can be separated with spaces or newlines. See \l{#arch-controller +Controller Logic} for details. + + +\h2#arch-task-package-config|\c{package-config}| + +\ +[package-config]: <pkg-config-args> +\ + +The primary package manifest \c{*-build-config} value for the build +configuration the build task is issued for. See \l{bpkg#manifest-package +Package Manifest} for more information on this value. A single level of quotes (either single or double) is removed in each value before being passed to \c{bpkg}. For example, the following value: \ -config: config.cc.coptions=\"-O3 -stdlib='libc++'\" +package-config: \"?libcurl ~7.76.0\" \ Will be passed to \c{bpkg} as the following (single) argument: \ -config.cc.coptions=-O3 -stdlib='libc++' +?libcurl ~7.76.0 \ Values can be separated with spaces or newlines. See \l{#arch-controller Controller Logic} for details. +\h2#arch-task-host|\c{host}| + +\ +[host]: true|false +\ + +If \c{true}, then the build target configuration is self-hosted. If not +specified, \c{false} is assumed. See \l{#arch-controller Controller Logic} for +details. + + \h2#arch-task-warning-regex|\c{warning-regex}| \ @@ -516,8 +756,19 @@ specified \c{interactive-mode} with either the \c{true} or \c{both} value in the task request. The breakpoint can either be a primary step id of the worker script or the -special \c{error} or \c{warning} value. See \l{#arch-worker Worker Logic} for -details. +special \c{error} or \c{warning} value. There is also the special \c{none} +value which never interrupts the task execution. See \l{#arch-worker Worker +Logic} for details. + + +\h2#arch-task-worker-checksum|\c{worker-checksum}| + +\ +[worker-checksum]: <checksum> +\ + +The worker checksum received as a part of the previous build task result (see +\l{#arch-result Result Manifest}). \h#arch-result|Result Manifest| @@ -535,15 +786,26 @@ status: <status> [update-status]: <status> [test-status]: <status> [install-status]: <status> +[bindist-status]: <status> +[sys-install-status]: <status> [test-installed-status]: <status> +[sys-uninstall-status]: <status> [uninstall-status]: <status> +[upload-status]: <status> [configure-log]: <text> [update-log]: <text> [test-log]: <text> [install-log]: <text> +[bindist-log]: <text> +[sys-install-log]: <text> [test-installed-log]: <text> +[sys-uninstall-log]: <text> [uninstall-log]: <text> +[upload-log]: <text> + +[worker-checksum]: <checksum> +[dependency-checksum]: <checksum> \ @@ -574,11 +836,13 @@ status: <status> The overall (cumulative) build result status. Valid values are: \ +skip # Package update and subsequent operations were skipped. success # All operations completed successfully. warning # One or more operations completed with warnings. error # One or more operations completed with errors. abort # One or more operations were aborted. abnormal # One or more operations terminated abnormally. +interrupt # Task execution has been interrupted. \ The \c{abort} status indicates that the operation has been aborted by @@ -591,9 +855,18 @@ include 'log:' with commands that start VM, for completeness?). The \c{abnormal} status indicates that the operation has terminated abnormally, for example, due to the package manager or build system crash. +The \c{interrupt} status indicates that the task execution has been +interrupted, for example, to reassign resources to a higher priority task. + Note that the overall \c{status} value should appear before any per-operation \c{*-status} values. +The \c{skip} status indicates that the received from the controller build task +checksums have not changed and the task execution has therefore been skipped +under the assumption that it would have produced the same result. See +\c{agent-checksum}, \c{worker-checksum}, and \c{dependency-checksum} for +details. + \h2#arch-result-x-status|\c{*-status}| @@ -611,8 +884,12 @@ configure update test install +bindist +sys-install test-installed +sys-uninstall uninstall +upload \ @@ -628,13 +905,38 @@ the list of supported operation names refer to the \c{*-status} value description. +\h2#arch-result-dependency-checksum|\c{dependency-checksum}| + +\ +[dependency-checksum]: <checksum> +\ + +The package dependency checksum obtained as a byproduct of the package +configuration operation. See \l{bpkg-pkg-build(1)} command's +\c{--rebuild-checksum} option for details. + + +\h2#arch-result-worker-checksum|\c{worker-checksum}| + +\ +[worker-checksum]: <checksum> +\ + +The version of the worker logic used to perform the package build task. + + \h#arch-task-req|Task Request Manifest| An agent (or controller acting as an agent) sends a task request to its controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request -starts with the task request manifest followed by a list of machine manifests. -The task request manifest synopsis is presented next followed by the detailed -description of each value in subsequent sections. +starts with the task request manifest followed by a list of machine header +manifests. The task request manifest synopsis is presented next followed by +the detailed description of each value in subsequent sections. + +\N|The controller is expected to pick each offered machine header manifest +only once. If an agent is capable of running multiple instances of the same +machine, then it must send the matching number of machine header manifests for +such a machine.| \ agent: <name> @@ -643,6 +945,7 @@ toolchain-version: <standard-version> [interactive-mode]: false|true|both [interactive-login]: <login> [fingerprint]: <agent-fingerprint> +[auxiliary-ram]: <kib> \ @@ -709,6 +1012,18 @@ authentication in which case it should respond with the 401 (unauthorized) HTTP status code. +\h2#arch-task-req-auxiliary-ram|\c{auxiliary-ram}| + +\ +[auxiliary-ram]: <kib> +\ + +The amount of RAM in KiB that is available for running auxiliary machines. If +unspecified, then assume there is no hard limit (that is, the agent can +allocate up to the host's available RAM minus the amount required to run the +build machine). + + \h#arch-task-res|Task Response Manifest| A controller sends the task response manifest in response to the task request @@ -722,6 +1037,8 @@ subsequent sections. session: <id> [challenge]: <text> [result-url]: <url> +[*-upload-url]: <url> +[agent-checksum]: <checksum> \ @@ -759,6 +1076,28 @@ private key and then \c{base64}-encoding the result. The URL to POST (upload) the result request to. +\h2#arch-task-res-upload-url|\c{*-upload-url}| + +\ +[*-upload-url]: <url> +\ + +The URLs to upload the build artifacts to, if any, via the HTTP \c{POST} +method using the \c{multipart/form-data} content type (see \l{brep#upload +Build Artifacts Upload} for details on the upload protocol). The substring +matched by \c{*} in \c{*-upload-url} denotes the upload type. + + +\h2#arch-task-res-agent-checksum|\c{agent-checksum}| + +\ +[agent-checksum]: <checksum> +\ + +The agent checksum received as a part of the previous build task result +request (see \l{#arch-result-req Result Request Manifest}). + + \h#arch-result-req|Result Request Manifest| On completion of a task an agent (or controller acting as an agent) sends the @@ -772,6 +1111,7 @@ description of each value in subsequent sections. \ session: <id> [challenge]: <text> +[agent-checksum]: <checksum> \ @@ -795,6 +1135,15 @@ response. It must be present only if the \c{challenge} value was present in the task response. +\h2#arch-result-req-agent-checksum|\c{agent-checksum}| + +\ +[agent-checksum]: <checksum> +\ + +The version of the agent logic used to perform the package build task. + + \h#arch-worker|Worker Logic| The \c{bbot} worker builds each package in a \i{build environment} that is @@ -802,20 +1151,24 @@ established for a particular build target. The environment has three components: the execution environment (environment variables, etc), build system modules, as well as configuration options and variables. -Setting up of the environment is performed by an executable (script, batch -file, etc). Specifically, upon receiving a build task, if it specifies the -environment name then the worker looks for the environment setup executable -with this name in a specific directory and for the executable called -\c{default} otherwise. Not being able to locate the environment executable is -an error. - -Once the environment setup executable is determined, the worker re-executes -itself as that executable passing to it as command line arguments the target -name, the path to the \c{bbot} worker to be executed once the environment is -setup, and any additional options that need to be propagated to the re-executed -worker. The environment setup executable is executed in the build directory as -its current working directory. The build directory contains the build task -\c{task.manifest} file. +Setting up of the execution environment is performed by an executable (script, +batch file, etc). Specifically, upon receiving a build task, if it specifies +the environment name then the worker looks for the environment setup +executable with this name in a specific directory and for the executable +called \c{default} otherwise. Not being able to locate the environment +executable is an error. + +In addition to the environment executable, if the task requires any auxiliary +machines, then the \c{auxiliary-environment} value from the task manifest is +incorporated into the execution environment. + +Specifically, once the environment setup executable is determined, the worker +re-executes itself in the auxiliary environment and as that executable passing +to it as command line arguments the target name, the path to the \c{bbot} +worker to be executed once the environment is setup, and any additional +options that need to be propagated to the re-executed worker. The environment +setup executable is executed in the build directory as its current working +directory. The build directory contains the build task \c{task.manifest} file. The environment setup executable sets up the necessary execution environment for example by adjusting \c{PATH} or running a suitable \c{vcvars} batch file. @@ -828,16 +1181,56 @@ modules (\c{<env-modules>}) and the list of configuration options and variables The re-executed \c{bbot} worker then proceeds to test the package from the repository by executing the following commands, collectively called a \i{worker script}. Each command has a unique \i{step id} that can be used as a -breakpoint as well as a prefix in the \c{<config-args>}, +breakpoint and normally as a prefix in the \c{<tgt-config-args>}, \c{<env-config-args>}, and \c{<env-modules>} values as discussed in -\l{#arch-controller Controller Logic}. Some step ids have fallback step ids -(listed in parenthesis) which are used in the absence of the primary step id -values. The \c{<>}-values are from the task manifest and the environment. - -\ -# bpkg.create +\l{#arch-controller Controller Logic} as well as in the \c{<pkg-config-args>} +values (see below). The \c{<>}-values are from the task manifest and the +environment though some are assigned by the worker during the script execution +(configuration directories, UUIDs, etc). In particular, the +\c{<pkg-config-args>} (prefixed global options and variables), +\c{<pkg-config-opts>} (unprefixed options), \c{<pkg-config-vars>} (unprefixed +variables), \c{<dependency-name>}, \c{<dependency-version-constraint>}, and +\c{<dep-config-vars>} values result from parsing the +\l{#arch-task-package-config \c{package-config}} task manifest value. + +Some prefix step ids have fallback step ids which are used in the absence of +the primary step id values. If the prefix step id differs from the breakpoint +step id and/or has the fallback step ids, then they are listed in parenthesis: +the prefix id before the colon and the fallback ids after it. + +Some commands have no target configuration or environment options or +variables. Such commands have only breakpoint step ids associated, which are +listed in square brackets. + +Note that the worker script varies for different primary package types. The +\c{bbot} worker classifies the primary package based on the configuration type +in which it is built: \c{module} (build system module packages), \c{host} +(packages such as source code generators, marked with the \c{requires: host} +manifest value; see \l{bpkg#manifest-package Package Manifest} for details), +and \c{target} (all other packages). + +Note also that the \c{*.configure.build} step configures potentially multiple +packages (primary package, tests, etc) in potentially multiple configurations +by always using the \c{bpkg.global.configure.build} prefix step id for global +(as opposed to package-specific) \l{bpkg-pkg-build(1)} options. The +\c{bpkg.global.configure.build} prefix id has no fallback ids. + +Note finally that if no configuration variables are specified in the main +package configuration, then the worker adds the +\c{config.<name>.develop=false} configuration variable for the main package at +the \c{bpkg.configure.build} step to trigger its package skeleton creation and +loading. It also adds this variable for external test packages at this step +and for the same purpose. This makes sure that these packages can be used as +dependencies of dependents with configuration clauses. To keep the below +listings concise, these variables are not shown. + +Worker script for \c{target} packages: + +\ +# bpkg.create (bpkg.target.create : b.create, bpkg.create) # -bpkg -V create <env-modules> <env-config-args> <config-args> +bpkg -V create <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> # bpkg.configure.add # @@ -847,9 +1240,20 @@ bpkg -v add <repository-url> # bpkg -v fetch --trust <repository-fp> -# bpkg.configure.build +# bpkg.configure.build ( +# bpkg.global.configure.build, +# (bpkg.target.configure.build : b.configure, bpkg.configure.build)) # -bpkg -v build --yes --configure-only <package-name>/<package-version> +bpkg -v build --configure-only \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<pkg-config-opts>] \\ + [{ <pkg-config-vars> }+] <package-name>/<package-version> \\ + [([{ <test-config-vars> }+] \\ + <test-package-name>[ <test-version-constraint>])...] \\ + [([{ <dep-config-vars> }+] \\ + (?|sys:)<dependency-name> \\ + [<dependency-version-constraint>])...] \\ + [?sys:<dependency-name>[ <dependency-version-constraint>]...] # bpkg.update # @@ -863,138 +1267,1010 @@ bpkg -v update <package-name> bpkg -v test <package-name> } -# For each package referred to by the tests, examples, or benchmarks -# package manifest values and not excluded by the test-exclude task -# manifest values: +# For each (runtime) tests, examples, or benchmarks package referred +# to by the task manifest: # { - # bpkg.test-separate.configure.build (bpkg.configure.build) - # - bpkg -v build --yes --configure-only \\ - '<package-name> [<version-constraint>]' - - # bpkg.test-separate.update (bpkg.update) + # bpkg.test-separate.update ( : bpkg.update) # bpkg -v update <package-name> - # bpkg.test-separate.test (bpkg.test) + # bpkg.test-separate.test ( : bpkg.test) # bpkg -v test <package-name> } -# If config.install.root is specified: +# If the install operation is supported by the package, +# config.install.root is specified, and no +# bpkg.bindist.{debian,fedora,archive} step is enabled: # { # bpkg.install # bpkg -v install <package-name> + # If bbot.install.ldconfig step is enabled: + # + { + # bbot.install.ldconfig + # + sudo ldconfig + } +} + +# If the install operation is supported by the package and +# bpkg.bindist.{debian,fedora,archive} step is enabled: +# +{ + # bpkg.bindist.{debian,fedora,archive} + # + bpkg -v bindist --distribution <distribution> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + <package-name> +} + +# If the install operation is supported by the package and +# bbot.sys-install step is enabled: +# +{ + # If <distribution> is 'debian': + # + { + # bbot.sys-install.apt-get.update + # + sudo apt-get update + + # bbot.sys-install.apt-get.install + # + sudo apt-get install <distribution-package-file>... + } + # + # Otherwise, if <distribution> is 'fedora': + # + { + # bbot.sys-install.dnf.install + # + sudo dnf install <distribution-package-file>... + } + # + # Otherwise, if <distribution> is 'archive': + # + { + # For each package file: + # + { + # bbot.sys-install.tar.extract + # + [sudo] tar -xf <distribution-package-file> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + } + + # If bbot.sys-install.ldconfig step is enabled: + # + { + # bbot.sys-install.ldconfig + # + sudo ldconfig + } + } +} + +# If the main package is installed either from source or from the +# binary distribution package: +# +{ # If the package contains subprojects that support the test # operation: # { - # b.test-installed.create + # b.test-installed.create ( : b.create) # - b -V create <env-modules> <env-config-args> <config-args> + b -V create <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> - # b.test-installed.configure + # For each test subproject: # - b -v configure + { + # b.test-installed.configure ( : b.configure) + # + b -v configure [<pkg-config-vars>] + } # b.test-installed.test # b -v test } - # If any of the tests, examples, or benchmarks package manifest - # values are specified and are not all excluded by the test-exclude - # task manifest values: + # If task manifest refers to any (runtime) tests, examples, or + # benchmarks packages: # { - # bpkg.test-installed.create (bpkg.create) + # bpkg.test-separate-installed.create ( + # bpkg.test-separate-installed.create_for_target : + # bpkg.test-separate-installed.create) # - bpkg -V create <env-modules> <env-config-args> <config-args> + bpkg -V create <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> - # bpkg.test-installed.configure.add (bpkg.configure.add) + # bpkg.test-separate-installed.configure.add ( + # : bpkg.configure.add) # bpkg -v add <repository-url> - # bpkg.test-installed.configure.fetch (bpkg.configure.fetch) + # bpkg.test-separate-installed.configure.fetch ( + # : bpkg.configure.fetch) # bpkg -v fetch --trust <repository-fp> - # For each package referred to by the tests, examples, or - # benchmarks package manifest values and not excluded by the - # test-exclude task manifest values: + # bpkg.test-separate-installed.configure.build ( + # bpkg.global.configure.build, + # (bpkg.test-separate-installed.configure.build_for_target : + # bpkg.test-separate-installed.configure.build)) + # + bpkg -v build --configure-only \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + ([{ <test-config-vars> }+] \\ + <test-package-name>[ <test-version-constraint>])... \\ + ?sys:<package-name>/<package-version> \\ + [?sys:<dependency-name>[ <dependency-version-constraint>]...] + + # For each (runtime) tests, examples, or benchmarks package + # referred to by the task manifest: # { - # bpkg.test-separate-installed.configure.build ( - # bpkg.configure.build) - # - bpkg -v build --yes --configure-only \\ - '<package-name> [<version-constraint>]' - - # bpkg.test-separate-installed.update (bpkg.update) + # bpkg.test-separate-installed.update ( : bpkg.update) # bpkg -v update <package-name> - # bpkg.test-separate-installed.test (bpkg.test) + # bpkg.test-separate-installed.test ( : bpkg.test) # bpkg -v test <package-name> } } +} + +# If the main package is installed from the binary distribution package: +# +{ + # If <distribution> is 'debian': + # + { + # bbot.sys-uninstall.apt-get.remove + # + sudo apt-get remove <distribution-package-name>... + } + # + # Otherwise, if <distribution> is 'fedora': + # + { + # bbot.sys-uninstall.dnf.remove + # + sudo dnf remove <distribution-package-name>... + } + # + # Otherwise, if <distribution> is 'archive': + # + { + # Noop. + } +} +# If the main package is installed from source: +# +{ # bpkg.uninstall # bpkg -v uninstall <package-name> } +# If the install operation is supported by the package and +# bbot.bindist.upload step is enabled: +# +{ + # Move the generated binary distribution files to the + # upload/bindist/<distribution>/ directory. +} + +# If bbot.upload step is enabled and upload/ directory is not empty: +# +{ + # bbot.upload.tar.create + # + tar -cf upload.tar upload/ + + # bbot.upload.tar.list + # + tar -tf upload.tar upload/ +} + # end # # This step id can only be used as a breakpoint. \ -For details on configuring and testing installation refer to -\l{#arch-controller Controller Logic}. +Worker script for \c{host} packages: -If the package is a build system module, then it is built and tested (using -the bundled tests) in a separate configuration that mimics the one used to -build \c{build2} itself. Note that the configuration and environment options -and variables are not passed to commands that may affect this configuration. -Such commands, therefore, have associated step ids that can only be used -as breakpoints (listed in square brackets): +\ +# If configuration is self-hosted: +# +{ + # bpkg.create (bpkg.host.create : b.create, bpkg.create) + # + bpkg -V create --type host -d <host-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> +} +# +# Otherwise: +# +{ + # [bpkg.create] + # + b -V create(<host-conf>, cc) config.config.load=~host + bpkg -v create --existing --type host -d <host-conf> +} + +# bpkg.configure.add +# +bpkg -v add -d <host-conf> <repository-url> + +# bpkg.configure.fetch +# +bpkg -v fetch -d <host-conf> --trust <repository-fp> + +# If configuration is self-hosted and config.install.root is specified: +# +{ + # bpkg.create (bpkg.target.create : b.create, bpkg.create) + # + bpkg -V create -d <install-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # [bpkg.link] + # + bpkg -v link -d <install-conf> <host-conf> + + # bpkg.configure.add + # + bpkg -v add -d <install-conf> <repository-url> + + # bpkg.configure.fetch + # + bpkg -v fetch -d <install-conf> --trust <repository-fp> +} + +# If task manifest refers to any build-time tests, examples, or +# benchmarks packages: +# +{ + # bpkg.create (bpkg.target.create : b.create, bpkg.create) + # + bpkg -V create -d <target-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # [bpkg.create] + # + b -V create(<module-conf>, cc) config.config.load=~build2 + bpkg -v create --existing --type build2 -d <module-conf> + + # [bpkg.link] + # + bpkg -v link -d <target-conf> <host-conf> + bpkg -v link -d <target-conf> <module-conf> + bpkg -v link -d <host-conf> <module-conf> + + # If configuration is self-hosted and config.install.root is + # specified: + # + { + # [bpkg.link] + # + bpkg -v link -d <install-conf> <module-conf> + } + + # bpkg.configure.add + # + bpkg -v add -d <target-conf> <repository-url> + + # bpkg.configure.fetch + # + bpkg -v fetch -d <target-conf> --trust <repository-fp> +} + +# bpkg.configure.build (bpkg.global.configure.build) +# +# Notes: +# +# - Some parts may be omitted. +# +# - Parts related to different configurations have different prefix +# step ids: +# +# bpkg.host.configure.build for <host-uuid> +# bpkg.target.configure.build for <install-uuid> +# bpkg.target.configure.build for <target-uuid> +# +# - All parts have the same fallback step ids: b.configure and +# bpkg.configure.build. +# +bpkg -v build --configure-only \\ +<env-config-args> <tgt-config-args> <pkg-config-args> \\ +[<pkg-config-opts>] \\ +\\ +{ --config-uuid <host-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<pkg-config-vars>] }+ \\ +<package-name>/<package-version> \\ +\\ +{ --config-uuid <install-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<pkg-config-vars>] }+ \\ +<package-name>/<package-version> \\ +\\ +({ --config-uuid <host-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<test-config-vars>] }+ \\ + <runtime-test-package-name>[ <test-version-constraint>])... \\ +\\ +({ --config-uuid <target-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<test-config-vars>] }+ \\ + <buildtime-test-package-name>[ <test-version-constraint>])... \\ +\\ +({ --config-uuid <host-uuid> [--config-uuid <install-uuid>] \\ + [<dep-config-vars>] }+ \\ + (?|sys:)<dependency-name>[ <dependency-version-constraint>])... \\ +\\ +[?sys:<dependency-name>[ <dependency-version-constraint>]...] + +# bpkg.update +# +bpkg -v update -d <host-conf> <package-name> + +# If the test operation is supported by the package: +# +{ + # bpkg.test + # + bpkg -v test -d <host-conf> <package-name> +} + +# If configuration is self-hosted, then for each runtime tests, +# examples, or benchmarks package referred to by the task manifest: +# +{ + # bpkg.test-separate.update ( : bpkg.update) + # + bpkg -v update -d <host-conf> <package-name> + + # bpkg.test-separate.test ( : bpkg.test) + # + bpkg -v test -d <host-conf> <package-name> +} + +# For each build-time tests, examples, or benchmarks package referred +# to by the task manifest: +# +{ + # bpkg.test-separate.update ( : bpkg.update) + # + bpkg -v update -d <target-conf> <package-name> + + # bpkg.test-separate.test ( : bpkg.test) + # + bpkg -v test -d <target-conf> <package-name> +} + +# If configuration is self-hosted, the install operation is supported +# by the package, config.install.root is specified, and no +# bpkg.bindist.{debian,fedora,archive} step is enabled: +# +{ + # bpkg.install + # + bpkg -v install -d <install-conf> <package-name> + + # If bbot.install.ldconfig step is enabled: + # + { + # bbot.install.ldconfig + # + sudo ldconfig + } +} + +# If configuration is self-hosted, the install operation is supported +# by the package, and bpkg.bindist.{debian,fedora,archive} step is +# enabled: +# +{ + # bpkg.bindist.{debian,fedora,archive} + # + bpkg -v bindist --distribution <distribution> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + <package-name> +} + +# If the install operation is supported by the package and +# bbot.sys-install step is enabled: +# +{ + # If <distribution> is 'debian': + # + { + # bbot.sys-install.apt-get.update + # + sudo apt-get update + + # bbot.sys-install.apt-get.install + # + sudo apt-get install <distribution-package-file>... + } + # + # Otherwise, if <distribution> is 'fedora': + # + { + # bbot.sys-install.dnf.install + # + sudo dnf install <distribution-package-file>... + } + # + # Otherwise, if <distribution> is 'archive': + # + { + # For each package file: + # + { + # bbot.sys-install.tar.extract + # + [sudo] tar -xf <distribution-package-file> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + } + + # If bbot.sys-install.ldconfig step is enabled: + # + { + # bbot.sys-install.ldconfig + # + sudo ldconfig + } + } +} + +# If the main package is installed either from source or from the +# binary distribution package: +# +{ + # If the package contains subprojects that support the test + # operation: + # + { + # b.test-installed.create ( : b.create) + # + b -V create <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # For each test subproject: + # + { + # b.test-installed.configure ( : b.configure) + # + b -v configure [<pkg-config-vars>] + } + + # b.test-installed.test + # + b -v test + } + + # If task manifest refers to any tests, examples, or benchmarks + # packages: + # + { + # bpkg.test-separate-installed.create ( + # bpkg.test-separate-installed.create_for_host : + # bpkg.test-separate-installed.create) + # + bpkg -V create --type host -d <host-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # If task manifest refers to any runtime tests, examples, or + # benchmarks packages: + # + { + # bpkg.test-separate-installed.configure.add ( + # : bpkg.configure.add) + # + bpkg -v add -d <host-conf> <repository-url> + + # bpkg.test-separate-installed.configure.fetch ( + # : bpkg.configure.fetch) + # + bpkg -v fetch -d <host-conf> --trust <repository-fp> + } + + # If task manifest refers to any build-time tests, examples, or + # benchmarks packages: + # + { + # bpkg.test-separate-installed.create ( + # bpkg.test-separate-installed.create_for_host : + # bpkg.test-separate-installed.create) + # + bpkg -V create -d <target-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # [bpkg.test-separate-installed.create] + # + b -V create(<module-conf>, cc) config.config.load=~build2 + bpkg -v create --existing --type build2 -d <module-conf> + + # [bpkg.test-separate-installed.link] + # + bpkg -v link -d <target-conf> <host-conf> + bpkg -v link -d <target-conf> <module-conf> + bpkg -v link -d <host-conf> <module-conf> + + # bpkg.test-separate-installed.configure.add ( + # : bpkg.configure.add) + # + bpkg -v add -d <target-conf> <repository-url> + + # bpkg.test-separate-installed.configure.fetch ( + # : bpkg.configure.fetch) + # + bpkg -v fetch -d <target-conf> --trust <repository-fp> + } + + # bpkg.test-separate-installed.configure.build ( + # bpkg.global.configure.build, + # (bpkg.test-separate-installed.configure.build_for_host : + # bpkg.test-separate-installed.configure.build)) + # + # Note that any of the runtime or build-time tests related parts + # (but not both) may be omitted. + # + bpkg -v build --configure-only \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + \\ + ({ --config-name <host-conf> [<test-config-vars>] }+ \\ + <runtime-test-package-name>[ <test-version-constraint>])... \\ + \\ + ({ --config-name <target-conf> [<test-config-vars>] }+ \\ + <buildtime-test-package-name>[ <test-version-constraint>])... \\ + \\ + ?sys:<package-name>/<package-version> \\ + \\ + [?sys:<dependency-name>[ <dependency-version-constraint>]...] + + # For each tests, examples, or benchmarks package referred + # to by the task manifest: + # + { + # bpkg.test-separate-installed.update ( : bpkg.update) + # + bpkg -v update <package-name> + + # bpkg.test-separate-installed.test ( : bpkg.test) + # + bpkg -v test <package-name> + } + } +} + +# If the main package is installed from the binary distribution package: +# +{ + # If <distribution> is 'debian': + # + { + # bbot.sys-uninstall.apt-get.remove + # + sudo apt-get remove <distribution-package-name>... + } + # + # Otherwise, if <distribution> is 'fedora': + # + { + # bbot.sys-uninstall.dnf.remove + # + sudo dnf remove <distribution-package-name>... + } + # + # Otherwise, if <distribution> is 'archive': + # + { + # Noop. + } +} + +# If the main package is installed from source: +# +{ + # bpkg.uninstall + # + bpkg -v uninstall -d <install-conf> <package-name> +} + +# If the install operation is supported by the package and +# bbot.bindist.upload step is enabled: +# +{ + # Move the generated binary distribution files to the + # upload/bindist/<distribution>/ directory. +} + +# If bbot.upload step is enabled and upload/ directory is not empty: +# +{ + # bbot.upload.tar.create + # + tar -cf upload.tar upload/ + + # bbot.upload.tar.list + # + tar -tf upload.tar upload/ +} + +# end +# +# This step id can only be used as a breakpoint. +\ + +Worker script for \c{module} packages: \ -# [bpkg.module.create] +# If configuration is self-hosted: +# +{ + # bpkg.create (bpkg.module.create) + # + b -V create(<module-conf>, <env-modules>) config.config.load=~build2 \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + bpkg -v create --existing --type build2 -d <module-conf> +} +# +# Otherwise: # -b -V create config.config.load=~build2 -bpkg -v create --existing +{ + # [bpkg.create] + # + b -V create(<module-conf>, cc) config.config.load=~build2 + bpkg -v create --existing --type build2 -d <module-conf> +} -# bpkg.module.configure.add (bpkg.configure.add) +# bpkg.configure.add # -bpkg -v add <repository-url> +bpkg -v add -d <module-conf> <repository-url> -# bpkg.module.configure.fetch (bpkg.configure.fetch) +# bpkg.configure.fetch # -bpkg -v fetch --trust <repository-fp> +bpkg -v fetch -d <module-conf> --trust <repository-fp> + +# If configuration is self-hosted and config.install.root is specified: +# +{ + # bpkg.create (bpkg.module.create) + # + b -V create(<install-conf>, <env-modules>) \\ + config.config.load=~build2 \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + bpkg -v create --existing --type build2 -d <install-conf> + + # bpkg.configure.add + # + bpkg -v add -d <install-conf> <repository-url> -# [bpkg.module.configure.build] + # bpkg.configure.fetch + # + bpkg -v fetch -d <install-conf> --trust <repository-fp> +} + +# If task manifest refers to any (build-time) tests, examples, or +# benchmarks packages: # -bpkg -v build --yes --configure-only <package-name>/<package-version> +{ + # bpkg.create (bpkg.target.create : b.create, bpkg.create) + # + bpkg -V create -d <target-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> -# [bpkg.module.update] + # [bpkg.create] + # + b -V create(<host-conf>, cc) config.config.load=~host + bpkg -v create --existing --type host -d <host-conf> + + # [bpkg.link] + # + bpkg -v link -d <target-conf> <host-conf> + bpkg -v link -d <target-conf> <module-conf> + bpkg -v link -d <host-conf> <module-conf> + + # bpkg.configure.add + # + bpkg -v add -d <target-conf> <repository-url> + + # bpkg.configure.fetch + # + bpkg -v fetch -d <target-conf> --trust <repository-fp> +} + +# bpkg.configure.build (bpkg.global.configure.build) # -bpkg -v update <package-name> +# Notes: +# +# - Some parts may be omitted. +# +# - Parts related to different configurations have different prefix +# step ids: +# +# bpkg.module.configure.build for <module-uuid> +# bpkg.target.configure.build for <install-uuid> +# bpkg.target.configure.build for <target-uuid> +# +# - All parts have the same fallback step ids: b.configure and +# bpkg.configure.build. +# +bpkg -v build --configure-only \\ +<env-config-args> <tgt-config-args> <pkg-config-args> \\ +[<pkg-config-opts>] \\ +\\ +{ --config-uuid <module-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<pkg-config-vars>] }+ \\ +<package-name>/<package-version> \\ +\\ +{ --config-uuid <install-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<pkg-config-vars>] }+ \\ +<package-name>/<package-version> \\ +\\ +({ --config-uuid <target-uuid> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + [<test-config-vars>] }+ \\ + <buildtime-test-package-name>[ <test-version-constraint>])... \\ +\\ +({ --config-uuid <host-uuid> [--config-uuid <install-uuid>] \\ + [<dep-config-vars>] }+ \\ + (?|sys:)<dependency-name>[ <dependency-version-constraint>])... \\ +\\ +[?sys:<dependency-name>[ <dependency-version-constraint>]...] + +# bpkg.update +# +bpkg -v update -d <module-conf> <package-name> # If the test operation is supported by the package: # { - # [bpkg.module.test] + # bpkg.test # - bpkg -v test <package-name> + bpkg -v test -d <module-conf> <package-name> +} + +# For each (build-time) tests, examples, or benchmarks package referred +# to by the task manifest: +# +{ + # bpkg.test-separate.update ( : bpkg.update) + # + bpkg -v update -d <target-conf> <package-name> + + # bpkg.test-separate.test ( : bpkg.test) + # + bpkg -v test -d <target-conf> <package-name> +} + +# If configuration is self-hosted, the install operation is supported +# by the package, config.install.root is specified, and no +# bpkg.bindist.{debian,fedora,archive} step is enabled: +# +{ + # bpkg.install + # + bpkg -v install -d <install-conf> <package-name> + + # If bbot.install.ldconfig step is enabled: + # + { + # bbot.install.ldconfig + # + sudo ldconfig + } +} + +# If configuration is self-hosted, the install operation is supported +# by the package, and bpkg.bindist.{debian,fedora,archive} step is +# enabled: +# +{ + # bpkg.bindist.{debian,fedora,archive} + # + bpkg -v bindist --distribution <distribution> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + <package-name> +} + +# If the install operation is supported by the package and +# bbot.sys-install step is enabled: +# +{ + # If <distribution> is 'debian': + # + { + # bbot.sys-install.apt-get.update + # + sudo apt-get update + + # bbot.sys-install.apt-get.install + # + sudo apt-get install <distribution-package-file>... + } + # + # Otherwise, if <distribution> is 'fedora': + # + { + # bbot.sys-install.dnf.install + # + sudo dnf install <distribution-package-file>... + } + # + # Otherwise, if <distribution> is 'archive': + # + { + # For each package file: + # + { + # bbot.sys-install.tar.extract + # + [sudo] tar -xf <distribution-package-file> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + } + + # If bbot.sys-install.ldconfig step is enabled: + # + { + # bbot.sys-install.ldconfig + # + sudo ldconfig + } + } } + +# If the main package is installed either from source or from the +# binary distribution package: +# +{ + # If task manifest refers to any (build-time) tests, examples, or + # benchmarks packages: + # + { + # [bpkg.test-separate-installed.create] + # + b -V create(<module-conf>, cc) config.config.load=~build2 + bpkg -v create --existing --type build2 -d <module-conf> + + # bpkg.test-separate-installed.create ( + # bpkg.test-separate-installed.create_for_module : + # bpkg.test-separate-installed.create) + # + bpkg -V create -d <target-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # bpkg.test-separate-installed.create ( + # bpkg.test-separate-installed.create_for_module : + # bpkg.test-separate-installed.create) + # + bpkg -V create --type host -d <host-conf> <env-modules> \\ + <env-config-args> <tgt-config-args> <pkg-config-args> + + # [bpkg.test-separate-installed.link] + # + bpkg -v link -d <target-conf> <host-conf> + bpkg -v link -d <target-conf> <module-conf> + bpkg -v link -d <host-conf> <module-conf> + + # bpkg.test-separate-installed.configure.add ( + # : bpkg.configure.add) + # + bpkg -v add -d <target-conf> <repository-url> + + # bpkg.test-separate-installed.configure.fetch ( + # : bpkg.configure.fetch) + # + bpkg -v fetch -d <target-conf> --trust <repository-fp> + + # bpkg.test-separate-installed.configure.build ( + # bpkg.global.configure.build, + # (bpkg.test-separate-installed.configure.build_for_module : + # bpkg.test-separate-installed.configure.build)) + # + bpkg -v build --configure-only \\ + <env-config-args> <tgt-config-args> <pkg-config-args> \\ + \\ + ({ --config-name <target-conf> [<test-config-vars>] }+ \\ + <buildtime-test-package-name>[ <test-version-constraint>])... \\ + \\ + ?sys:<package-name>/<package-version> \\ + \\ + [?sys:<dependency-name>[ <dependency-version-constraint>]...] + + # For each (build-time) tests, examples, or benchmarks package + # referred to by the task manifest: + # + { + # bpkg.test-separate-installed.update ( : bpkg.update) + # + bpkg -v update -d <target-conf> <package-name> + + # bpkg.test-separate-installed.test ( : bpkg.test) + # + bpkg -v test -d <target-conf> <package-name> + } + } +} + +# If the main package is installed from the binary distribution package: +# +{ + # If <distribution> is 'debian': + # + { + # bbot.sys-uninstall.apt-get.remove + # + sudo apt-get remove <distribution-package-name>... + } + # + # Otherwise, if <distribution> is 'fedora': + # + { + # bbot.sys-uninstall.dnf.remove + # + sudo dnf remove <distribution-package-name>... + } + # + # Otherwise, if <distribution> is 'archive': + # + { + # Noop. + } +} + +# If the main package is installed from source: +# +{ + # bpkg.uninstall + # + bpkg -v uninstall -d <install-conf> <package-name> +} + +# If the install operation is supported by the package and +# bbot.bindist.upload step is enabled: +# +{ + # Move the generated binary distribution files to the + # upload/bindist/<distribution>/ directory. +} + +# If bbot.upload step is enabled and upload/ directory is not empty: +# +{ + # bbot.upload.tar.create + # + tar -cf upload.tar upload/ + + # bbot.upload.tar.list + # + tar -tf upload.tar upload/ +} + +# end +# +# This step id can only be used as a breakpoint. \ +For details on configuring and testing installation refer to +\l{#arch-controller Controller Logic}. + If a primary or test package comes from a version control-based repository, then its \c{dist} meta-operation is also tested as a part of the \c{bpkg[.*].configure.build} steps by re-distributing the source directory in @@ -1040,82 +2316,183 @@ shift exec \"$@\" cc config.c=\"gcc-9 $mode\" config.cxx=\"g++-9 $mode\" \ +\h2#arch-worker-bindist-result|Bindist Result Manifest| + +At the \c{bbot.bindist.upload} step the worker also creates the +\c{bindist-result.json} and \c{bindist-result.manifest} files in the +\c{upload/bindist/<distribution>/} directory, next to the generated binary +distribution package files. The \c{bindist-result.json} file contains the +structured JSON output of the \l{bpkg-pkg-bindist(1)} command. The +\c{bindist-result.manifest} file contains the subset of the information from +\c{bindist-result.json}. Specifically, it starts with the binary distribution +package header manifest followed by a list of package file manifests. The +manifest values are: + +\ +distribution: +architecture: +os-release-name-id: +os-release-version-id: +package-name: +package-version: +[package-system-version]: + +package-file-type: +package-file-path: +[package-file-system-name]: +\ + +The manifest values derive from the corresponding JSON object values and +preserve their semantics. The only differences are that the +\c{os-release-version-id} value may not be absent and the +\c{package-file-path} values are relative to the +\c{upload/bindist/<distribution>/} directory and are in the POSIX +representation. See \l{bpkg-pkg-bindist(1)} for the JSON values semantics. + \h#arch-controller|Controller Logic| A \c{bbot} controller that issues own build tasks maps available build -machines (as reported by agents) to \i{build configurations} according to the -\c{buildtab} configuration file. Blank lines and lines that start with \c{#} -are ignored. All other lines in this file have the following format: +machines (as reported by agents) to \i{build target configurations} according +to the \c{buildtab} configuration file. Blank lines and lines that start with +\c{#} are ignored. All other lines in this file have the following format: \ -<machine-pattern> <config> <target>[/<environment>] <classes> [<config-arg>]* [<warning-regex>]* +<machine-pattern> <target-config> <target>[/<environment>] <classes> [<tgt-config-arg>]* [<warning-regex>]* -<config-arg> = [<prefix>:](<variable>|<option>) -<prefix> = <tool>[.<phase>][.<operation>[.<command>]] +<tgt-config-arg> = [[+|-]<prefix>:](<variable>|<option>) | \\ + (+|-)<prefix>: +<prefix> = <tool>[.<cfg-type>][.<phase>][.<operation>[.<command>]] \ Where \c{<machine-pattern>} is filesystem wildcard pattern that is matched -against available machine names, \c{<config>} is the configuration name, -\c{<target>} is the build target, optional \c{<environment>} is the build -environment name, \c{<classes>} is a space-separated list of configuration -classes that is matched against the package \c{builds} values, optional -\c{<config-arg>} list is additional configuration options and variables, and -optional \c{<warning-regex>} list is additional regular expressions that -should be used to detect warnings in the logs. - -The build configurations can belong to multiple classes with their names -reflecting some common configuration aspects, such as the operating system, -compiler, build options, etc. Predefined class names are \c{default}, \c{all}, -and \c{none}. The default configurations are built by default. A configuration -must also belong to the \c{all} unless it is hidden. Valid custom class names -must contain only alpha-numeric characters, \c{_}, \c{+}, \c{-}, and \c{.}, -except as the first character for the last three. Class names that start with -\c{_} are reserved for the future hidden/special class functionality. - -Regular expressions must start with \c{~}, to be distinguished from -configuration options and variables. Note that the \c{<config-arg>} and -\c{<warning-regex>} lists have the same quoting semantics as in the \c{config} -and the \c{warning-regex} value in the build task manifest. The matched -machine name, the target, the environment name, configuration -options/variables, and regular expressions are included into the build task -manifest. - -Values in the \c{<config-arg>} list can be opionally prefixed with the \i{step -id} or a leading portion thereof to restrict it to a specific step, operation, -phase, or tool in the \i{worker script} (see \l{#arch-worker Worker -Logic}). Unprefixed values only apply to the \c{bpkg.create}, -\c{b.test-installed.create}, and \c{bpkg.test-installed.create} steps. Note -that options with values can only be specified using the single argument +against available machine names, \c{<target-config>} is the target +configuration name, \c{<target>} is the build target, optional +\c{<environment>} is the build environment name, \c{<classes>} is a +space-separated list of configuration classes that is matched against the +package configuration \c{*-builds} values, optional \c{<tgt-config-arg>} list +is additional configuration options and variables, and optional +\c{<warning-regex>} list is additional regular expressions that should be used +to detect warnings in the logs. + +The build target configurations can belong to multiple classes with their +names reflecting some common configuration aspects, such as the operating +system, compiler, build options, etc. Predefined class names are \c{default}, +\c{all}, \c{hidden}, \c{none}, \c{host}, and \c{build2}. The default target +configurations are built by default. A configuration must also belong to the +\c{all}, \c{hidden}, or some special-purpose configuration class. The latter +is intended for testing some optional functionality which packages are not +expected to provide normally (for example, relocatable installation). A +configuration that is self-hosted must also belong to the \c{host} class and, +if it is also self-hosted for build system modules, to the \c{build2} +class. Valid custom class names must contain only alpha-numeric characters, +\c{_}, \c{+}, \c{-}, and \c{.}, except as the first character for the last +three. Class names that start with \c{_} are reserved for the future +hidden/special class functionality. + +Regular expressions must start with \c{~}, to be distinguished from target +configuration options and variables. Note that the \c{<tgt-config-arg>} and +\c{<warning-regex>} lists have the same quoting semantics as in the +\c{target-config} and the \c{warning-regex} value in the build task +manifest. The matched machine name, the target, the environment name, +configuration options/variables, and regular expressions are included into the +build task manifest. + +Values in the \c{<tgt-config-arg>} list can be optionally prefixed with the +\i{step id} or a leading portion thereof to restrict it to a specific step, +operation, phase, or tool in the \i{worker script} (see \l{#arch-worker Worker +Logic}). The prefix can optionally begin with the \c{+} or \c{-} character (in +this case the argument can be omitted) to enable or disable the respective +step. The steps which can be enabled or disabled are: + +\ +bpkg.update +bpkg.test +bpkg.test-separate.update +bpkg.test-separate.test + +# Disabled if bpkg.bindist.* is enabled. +# +bpkg.install + +# Disabled by default. +# +bbot.install.ldconfig + +# Disabled by default. +# +bpkg.bindist.{debian,fedora,archive} + +# Disabled if bpkg.bindist.* is disabled. +# +bbot.sys-install + +# Disabled by default. +# +bbot.sys-install.ldconfig + +b.test-installed.test +bpkg.test-separate-installed.update +bpkg.test-separate-installed.test + +# Disabled by default. +# +bbot.bindist.upload + +bbot.upload +\ + +Note that the \c{bpkg.bindist.*} steps are mutually exclusive and only the +last step status change via the \c{(+|-)bpkg.bindist.*} prefix is considered. + +Unprefixed values only apply to the \c{*.create[_for_*]} steps. Note that +options with values can only be specified using the single argument notation. For example: \ -bpkg:--fetch-timeout=600 bpkg.configure.fetch:--fetch-timeout=60 b:-j1 +bpkg:--fetch-timeout=600 \\ +bpkg.configure.fetch:--fetch-timeout=60 \\ ++bpkg.bindist.debian: \\ +b:-j1 \ Note that each machine name is matched against every pattern and all the -patterns that match produce a configuration. If a machine does not match any -pattern, then it is ignored (meaning that this controller is not interested in -testing its packages with this machine). If multiple machines match the same -pattern, then only a single configuration using any of the machines is -produced (meaning that this controller considers these machines equivalent). - -As an example, let's say we have a machine named \c{windows_10-vc_14u3}. If we +patterns that match produce a target configuration. If a machine does not +match any pattern, then it is ignored (meaning that this controller is not +interested in testing its packages with this machine). If multiple machines +match the same pattern, then only a single target configuration using any of +the machines is produced (meaning that this controller considers these +machines equivalent). + +As an example, let's say we have a machine named \c{windows_10-vc_14.3}. If we wanted to test both 32 and 64-bit as well as debug and optimized builds, then -we could have generated the following configurations: +we could have generated the following target configurations: + +\ +windows*-msvc_14* windows-msvc_14-Z7 i686-microsoft-win32-msvc14.0 \"all default msvc i686 debug\" config.cc.coptions=/Z7 config.cc.loptions=/DEBUG ~\"warning C4\d{3}: \" + +windows*-msvc_14* windows-msvc_14-O2 i686-microsoft-win32-msvc14.0 \"all default msvc i686 optimized\" config.cc.coptions=/O2 ~\"warning C4\d{3}: \" + +windows*-msvc_14* windows-msvc_14-Z7 x86_64-microsoft-win32-msvc14.0 \"all default msvc x86_64 debug\" config.cc.coptions=/Z7 config.cc.loptions=/DEBUG ~\"warning C4\d{3}: \" +windows*-msvc_14* windows-msvc_14-O2 x86_64-microsoft-win32-msvc14.0 \"all default msvc x86_64 optimized\" config.cc.coptions=/O2 ~\"warning C4\d{3}: \" \ -windows*-msvc_14* windows-msvc_14-32-Z7 i686-microsoft-win32-msvc14.0 \"all default msvc i686 debug\" config.cc.coptions=/Z7 config.cc.loptions=/DEBUG ~\"warning C4\d{3}: \" -windows*-msvc_14* windows-msvc_14-32-O2 i686-microsoft-win32-msvc14.0 \"all default msvc i686 optimized\" config.cc.coptions=/O2 ~\"warning C4\d{3}: \" +In the above example we could handle both \c{i686} and \c{x86_64} +architectures with the same machine but this may not always be possible +and we may have to use different machines for different configuration/target +combinations. For example: + +\ +x86_64_linux_debian_11*-gcc_12.2 linux_debian_11-gcc_12.2 i686-linux-gnu ... -windows*-msvc_14* windows-msvc_14-64-Z7 x86_64-microsoft-win32-msvc14.0 \"all default msvc x86_64 debug\" config.cc.coptions=/Z7 config.cc.loptions=/DEBUG ~\"warning C4\d{3}: \" +x86_64_linux_debian_11*-gcc_12.2 linux_debian_11-gcc_12.2 x86_64-linux-gnu ... -windows*-msvc_14* windows-msvc_14-64-O2 x86_64-microsoft-win32-msvc14.0 \"all default msvc x86_64 optimized\" config.cc.coptions=/O2 ~\"warning C4\d{3}: \" +aarch64_linux_debian_11*-gcc_12.2 linux_debian_11-gcc_12.2 aarch64-linux-gnu ... \ As another example, let's say we have \c{linux_fedora_25-gcc_6} and \c{linux_ubuntu_16.04-gcc_6}. If all we cared about is testing GCC 6 64-bit -builds on Linux, then our configurations could look like this: +builds on Linux, then our target configurations could look like this: \ linux*-gcc_6 linux-gcc_6-g x86_64-linux-gnu \"all default gcc debug\" config.cc.coptions=-g @@ -1123,11 +2500,12 @@ linux*-gcc_6 linux-gcc_6-g x86_64-linux-gnu \"all default gcc debug\" config.cc. linux*-gcc_6 linux-gcc_6-O3 x86_64-linux-gnu \"all default gcc optimized\" config.cc.coptions=-O3 \ -A build configuration class can derive from another class in which case -configurations that belong to the derived class are treated as also belonging -to the base class (or classes, recursively). The derived and base class names -are separated with \c{:} (no leading or trailing spaces allowed) and the base -must be present in the first mentioning of the derived class. For example: +A build target configuration class can derive from another class in which case +target configurations that belong to the derived class are treated as also +belonging to the base class (or classes, recursively). The derived and base +class names are separated with \c{:} (no leading or trailing spaces allowed) +and the base must be present in the first mentioning of the derived class. For +example: \ linux*-gcc_6 linux-gcc_6-g x86_64-linux-gnu \"all gcc-6+ debug\" config.cc.coptions=-g @@ -1151,15 +2529,17 @@ linux*-gcc_6 linux-gcc_6 x86_64-linux-gnu \"all gcc-6+ \" linux*-gcc_8 linux-gcc_8 x86_64-linux-gnu \"all gcc-8+:gcc-7+\" \ -If the \c{<config-arg>} list contains the \c{config.install.root} variable -that applies to the \c{bpkg.create} step, then in addition to building and -possibly running tests, the \c{bbot} worker will also test installing and -uninstalling each package. Furthermore, if the package contains subprojects -that support the test operation and/or refers to other packages via the -\c{tests}, \c{examples}, or \c{benchmarks} manifest values which are not -excluded by the \c{test-exclude} task manifest values, then the worker will -additionally build such subprojects/packages against the installation and run -their tests (test installed and test separate installed phases). +If the \c{<tgt-config-arg>} list contains the \c{config.install.root} variable +that applies to the \c{bpkg.target.create} or, as a fallback, \c{b.create} or +\c{bpkg.create} steps, then in addition to building and possibly running +tests, the \c{bbot} worker will also test installing and uninstalling each +package (unless replaced with the \c{bbot.sys-install} step). Furthermore, if +the package contains subprojects that support the test operation and/or refers +to other packages via the \c{tests}, \c{examples}, or \c{benchmarks} manifest +values which are not excluded by the \c{bbot} controller, then the worker will +additionally build such subprojects/packages against the installation (created +either from source or from the binary distribution package) and run their +tests (test installed and test separate installed phases). Two types of installations can be tested: \i{system} and \i{private}. A system installation uses a well-known location, such as \c{/usr} or \c{/usr/local}, @@ -1170,8 +2550,8 @@ preferable, it may not be always usable because of the potential conflicts with the already installed software, for example, by the system package manager. -As an example, the following two configurations could be used to test system -and private installations: +As an example, the following two target configurations could be used to test +system and private installations: \ linux*-gcc* linux-gcc-sysinstall x86_64-linux-gnu \"all default gcc\" config.install.root=/usr config.install.sudo=sudo @@ -1179,7 +2559,23 @@ linux*-gcc* linux-gcc-sysinstall x86_64-linux-gnu \"all default gcc\" config.ins linux*-gcc* linux-gcc-prvinstall x86_64-linux-gnu \"all default gcc\" config.install.root=/tmp/install config.cc.poptions=-I/tmp/install/include config.cc.loptions=-L/tmp/install/lib config.bin.rpath=/tmp/install/lib \ -Note also that while building and running tests against the installation the -worker makes the \c{bin} subdirectory of \c{config.install.root} the first -entry in the \c{PATH} environment variable. +Note also that while building and running tests against the installation +created either from source or from the archive distribution package the worker +makes the \c{bin} subdirectory of \c{config.install.root} the first entry in +the \c{PATH} environment variable, except for build system modules which +supposedly don't install any executables. As was mentioned earlier, normally +the \c{config.install.root} variable is expected to be prefixed with the +\c{bpkg.target.create} or, as a fallback, \c{b.create} or \c{bpkg.create} step +ids. However, for testing of the relocatable installations it can be desirable +to extract the archive distribution package content at the +\c{bbot.sys-install.tar.extract} step into a different installation +directory. If that's the case, then this directory needs to also be specified +as \c{bbot.sys-install:config.install.root}. If specified, this directory will +be preferred as a base for forming the \c{bin/} directory path. + +The \c{bbot} controller normally issues the build task by picking an unbuilt +package configuration and one of the produced (via the machine names match) +target configurations, which is not excluded from building due to this package +configuration \c{*-builds}, \c{*-build-include}, and \c{*-build-exclude} +manifest values. " |