diff options
| -rw-r--r-- | doc/manual.cli | 605 |
1 files changed, 368 insertions, 237 deletions
diff --git a/doc/manual.cli b/doc/manual.cli index c565278..aa0f92d 100644 --- a/doc/manual.cli +++ b/doc/manual.cli @@ -155,388 +155,519 @@ windows-vc_14-32-debug linux-gcc_6-cross-arm-eabi \ -\h#arch-machine-header-manifest|Machine Header Manifest| +\h#arch-machine-header|Machine Header Manifest| -\ -SYNOPSIS +@@ TODO: need ref to general manifest overview in bpkg, or, better yet, +move it to libbutl and ref to that from both places. -id: <machine-id> -name: <machine-name> +The build machine header manifest contains basic information about a build +machine on the build host. A list of machine header manifests is sent by +\c{bbot} agents to controllers. The manifest synopsis is presented next +followed by the detailed description of each value in subsequent sections. + +\ +id: <machine-id> +name: <machine-name> summary: <string> \ -The build machine header manifest contains basic information about a build -machine on the build host. A list of machine header manifests is sent by -\c{bbot} agents to controllers. +For example: -\dl| +\ +id: windows_10-msvc_14-1.3 +name: windows_10-msvc_14 +summary: Windows 10 build 1607 with VC 14 update 3 +\ -\li|\n\c{id: <machine-id>}\n +\h2#arch-machine-header-id|\c{id}| - The \i{machine-id} uniquely identifies a machine version/revision/build. - For virtual machines this can be the disk image checksum. For a container - this can be UUID that is re-generated every time a container filesystem - is altered.| +\ +id: <machine-id> +\ -\li|\n\c{name: <machine-name>}\n +The uniquely machine version/revision/build identifies. For virtual machines +this can be the disk image checksum. For a container this can be UUID that is +re-generated every time a container filesystem is altered. - The machine name as described above.| -\li|\n\c{summary: <string>}\n +\h2#arch-machine-header-name|\c{name}| - A one-line description of the machine. For example: +\ +name: <machine-name> +\ - \ - id: windows_10-msvc_14-1.3 - name: windows_10-msvc_14 - summary: Windows 10 build 1607 with VC 14 update 3 - \ +The machine name. -|| -\h#arch-machine-manifest|Machine Manifest| +\h2#arch-machine-header-summary|\c{summary}| \ -SYNOPSIS - -id: <machine-id> -name: <machine-name> summary: <string> - -type: <machine-type> -mac: <macaddr> -options: <machine-options> \ +The one-line description of the machine. + + +\h#arch-machine|Machine Manifest| + The build machine manifest contains the complete description of a build machine on the build host (see the Build OS documentation for their origin and -location). The machine manifest starts with the machine manifest header. All -the header values must appear before any non-header values. +location). The machine manifest starts with the machine manifest header with +all the header values appearing before any non-header values. The non-header +part of manifest synopsis is presented next followed by the detailed +description of each value in subsequent sections. -\dl| +\ +type: kvm|nspawn +[mac]: <addr> +[options]: <machine-options> +\ -\li|\n\c{type: <machine-type>}\n - The machine type. Valid values are \c{kvm} (QEMU/KVM virtual machine) and - \c{nspawn} (\c{systemd-nspawn} container).| +\h2#arch-machine-type|\c{type}| -\li|\n\c{mac: <macaddr>}\n +\ +type: kvm|nspawn +\ - Optional fixed MAC address for the machine in the hexadecimal, - comma-separated format. For example: +The machine type. Valid values are \c{kvm} (QEMU/KVM virtual machine) and +\c{nspawn} (\c{systemd-nspawn} container). - \ - mac: de:ad:be:ef:de:ad - \ - If it is not specified, then a random address is generated on the first - machine bootstrap which is then reused for each build/re-bootstrap. Note - that it you specify a fixed address, then the machine can only be used by a - single \c{bbot} agent.| +\h2#arch-machine-mac|\c{mac}| +\ +[mac]: <addr> +\ -\li|\n\c{options: <machine-options>}\n +The fixed MAC address for the machine. Must be in the hexadecimal, +comma-separated format. For example: - Optional list of machine options. The exact semantics is machine - type-dependent (see below). A single level of quotes (either single or - double) is removed in each option before being passed on. Options can be - separated with spaces or newlines. +\ +mac: de:ad:be:ef:de:ad +\ - For \c{kvm} machines, if this value is present, then it replaces the - default network and disk configuration when starting the QEMU/KVM - hypervisor The options are pre-processed by replacing the question - mark in \c{ifname=?} and \c{mac=?} strings with the network interface - and MAC address, respectively.|| +If it is not specified, then a random address is generated on the first +machine bootstrap which is then reused for each build/re-bootstrap. Note that +if you specify a fixed address, then the machine can only be used by a single +\c{bbot} agent. -\h#arch-task-manifest|Task Manifest| +\h2#arch-machine-options|\c{options}| \ -SYNOPSIS +[options]: <machine-options> +\ -name: <package-name> -version: <package-version> -#location: <package-url> -repository: <repository-url> -trust: <repository-fp> +The list of machine options. The exact semantics is machine type-dependent +(see below). A single level of quotes (either single or double) is removed in +each option before being passed on. Options can be separated with spaces or +newlines. -machine: <machine-name> -target: <target-triplet> -config: <config-vars> -warning-regex: <warning-regexes> -\ +For \c{kvm} machines, if this value is present, then it replaces the default +network and disk configuration when starting the QEMU/KVM hypervisor. The +options are pre-processed by replacing the question mark in \c{ifname=?} and +\c{mac=?} strings with the network interface and MAC address, respectively. + + +\h#arch-task|Task Manifest| The task manifest describes a build task. It consists of two groups of values. The first group defines the package to build. The second group defines the -build configuration to use for building the package. +build configuration to use for building the package. The manifest synopsis is +presented next followed by the detailed description of each value in +subsequent sections. -\dl| +\ +name: <package-name> +version: <package-version> +#location: <package-url> +repository: <repository-url> +[trust]: <repository-fp> -\li|\n\c{name: <package-name>}\n +machine: <machine-name> +target: <target-triplet> +[config]: <config-vars> +[warning-regex]: <warning-regex> +\ - Package name to test.| -\li|\n\c{version: <package-version>}\n +\h2#arch-task-name|\c{name}| - Package version to test.| +\ +name: <package-name> +\ -\li|\n\c{repository: <repository-url>}\n +The package name to build. - The \c{bpkg} repository that contains the package and its dependencies.| -\li|\n\c{trust: <repository-fp>}\n +\h2#arch-task-version|\c{version}| - The SHA256 repository certificate fingerprint to trust (see the \c{bpkg} - \c{--trust} option for details). This value may be specified multiple times - to establish the authenticity of multiple certificates. If the special - \c{yes} value is specified, then all repositories will be trusted without - authentication (see the \c{bpkg} \c{--trust-yes} option). +\ +version: <package-version> +\ - Note that while the controller may return a task with \c{trust} values, - whether they will be used is up to the agent's configuration. For example, - some agents may only trust their internally-specified fingerprints to - prevent the \"man in the middle\" type of attacks.| +The package version to build. -\li|\n\c{machine: <machine-name>}\n - The name of the build machine to use.| +\h2#arch-task-repository|\c{repository}| -\li|\n\c{target: <target-triplet>}\n +\ +repository: <repository-url> +\ - The target triplet to build for. +The \c{bpkg} repository that contains the package and its dependencies. - Compared to the autotools terminology, the \c{machine} value corresponds - to \c{--build} (the machine we are building on) and \c{target} \- to - \c{--host} (the machine we are building for). While we use essentially - the same \i{target triplet} format as autotools for \c{target}, it is - not flexible enough for \c{machine}.| -\li|\n\c{config: <config-vars>}\n +\h2#arch-task-trust|\c{trust}| + +\ +[trust]: <repository-fp> +\ + +The SHA256 repository certificate fingerprint to trust (see the \c{bpkg} +\c{--trust} option for details). This value may be specified multiple times to +establish the authenticity of multiple certificates. If the special \c{yes} +value is specified, then all repositories will be trusted without +authentication (see the \c{bpkg} \c{--trust-yes} option). + +Note that while the controller may return a task with \c{trust} values, +whether they will be used is up to the agent's configuration. For example, +some agents may only trust their internally-specified fingerprints to prevent +the \"man in the middle\" attacks. - Additional build system configuration variables. - A single level of quotes (either single or double) is removed in each - variable before being passed to \c{bpkg}. For example, the following value: +\h2#arch-task-machine|\c{machine}| - \ - config: config.cc.coptions=\"-O3 -stdlib='libc++'\" - \ +\ +machine: <machine-name> +\ - Will be passed to \c{bpkg} as the following (single) argument: +The name of the build machine to use. - \ - config.cc.coptions=-O3 -stdlib='libc++' - \ - Variables can be separated with spaces or newlines.| +\h2#arch-task-target|\c{target}| -\li|\n\c{warning-regex: <warning-regexes>}\n +\ +target: <target-triplet> +\ - Additional regular expressions that should be used to detect warnings in - the logs. Note that only the first 512 bytes of each log line is considered. +The target to build for. - A single level of quotes (either single or double) is removed in each - expression before being used for search. For example, the following value: +Compared to the autotools terminology, the \c{machine} value corresponds to +\c{--build} (the machine we are building on) and \c{target} \- to \c{--host} +(the machine we are building for). While we use essentially the same \i{target +triplet} format as autotools for \c{target}, it is not flexible enough for +\c{machine}. - \ - warning-regex: \"warning C4\d{3}: \" - \ - Will be treated as the following (single) regular expression (with a - trailing space): +\h2#arch-task-config|\c{config}| + +\ +[config]: <config-vars> +\ + +The additional build system configuration variables. A single level of quotes +(either single or double) is removed in each variable before being passed to +\c{bpkg}. For example, the following value: + +\ +config: config.cc.coptions=\"-O3 -stdlib='libc++'\" +\ + +Will be passed to \c{bpkg} as the following (single) argument: + +\ +config.cc.coptions=-O3 -stdlib='libc++' +\ + +Variables can be separated with spaces or newlines. + +\h2#arch-task-warning-regex|\c{warning-regex}| + +\ +[warning-regex]: <warning-regex> +\ - \ - warning C4\d{3}: - \ +Additional regular expressions that should be used to detect warnings in the +build logs. Note that only the first 512 bytes of each log line is considered. - Expressions can be separated with spaces or newlines. They will be added to - the following default list of regular expressions that detect the \c{build2} - toolchain warnings: +A single level of quotes (either single or double) is removed in each +expression before being used for search. For example, the following value: - \ - ^warning: - ^.+: warning: - \ +\ +warning-regex: \"warning C4\d{3}: \" +\ - Note that this built-in list also covers GCC and Clang warnings (for the - English locale).|| +Will be treated as the following (single) regular expression (with a trailing +space): +\ +warning C4\d{3}: +\ -\h#arch-result-manifest|Result Manifest| +Expressions can be separated with spaces or newlines. They will be added to +the following default list of regular expressions that detect the \c{build2} +toolchain warnings: \ -SYNOPSIS +^warning: +^.+: warning: +\ -name: <package-name> -version: <package-version> +Note that this built-in list also covers GCC and Clang warnings (for the +English locale). -status: <status> -configure-status: <status> -update-status: <status> -test-status: <status> -install-status: <status> -test-installed-status: <status> -uninstall-status: <status> -configure-log: <text> -update-log: <text> -test-log: <text> -install-log: <text> -test-installed-log: <text> -uninstall-log: <text> +\h#arch-result|Result Manifest| + +The result manifest describes a build result. The manifest synopsis is +presented next followed by the detailed description of each value in +subsequent sections. + \ +name: <package-name> +version: <package-version> -The result manifest describes a build result. +status: <status> +[configure-status]: <status> +[update-status]: <status> +[test-status]: <status> +[install-status]: <status> +[test-installed-status]: <status> +[uninstall-status]: <status> -\dl| +[configure-log]: <text> +[update-log]: <text> +[test-log]: <text> +[install-log]: <text> +[test-installed-log]: <text> +[uninstall-log]: <text> +\ -\li|\n\c{name: <package-name>}\n - Package name from the task manifest.| +\h2#arch-result-name|\c{name}| -\li|\n\c{version: <package-version>}\n +\ +name: <package-name> +\ + +The package name from the task manifest. + + +\h2#arch-result-version|\c{version}| - Package version from the task manifest.| +\ +version: <package-version> +\ -\li|\n\c{status: <status>}\n +The package version from the task manifest. - An overall (cumulative) build result status. Valid values are: - \ - success # All operations completed successfully. - warning # One or more operations completed with warnings. - error # One or more operations completed with errors. - abort # One or more operations were aborted. - abnormal # One or more operations terminated abnormally. - \ +\h2#arch-result-status|\c{status}| + +\ +status: <status> +\ - The \c{abort} status indicates that the operation has been aborted by - \c{bbot}, for example, because it was consuming too many resources and/or - was taking too long. Note that a task can be aborted both by the \c{bbot} - worker as well as the agent. In the later case the whole machine is shut - down and no operation-specific status or logs will be included (@@ Maybe - we should just include 'log:' with commands that start VM, for - completeness?). +The overall (cumulative) build result status. Valid values are: - The \c{abnormal} status indicates that the operation has terminated - abnormally, for example, due to the package manager or build system crash. +\ +success # All operations completed successfully. +warning # One or more operations completed with warnings. +error # One or more operations completed with errors. +abort # One or more operations were aborted. +abnormal # One or more operations terminated abnormally. +\ - Note that the overall \c{status} value should appear before any - per-operation \c{*-status} values.| +The \c{abort} status indicates that the operation has been aborted by +\c{bbot}, for example, because it was consuming too many resources and/or was +taking too long. Note that a task can be aborted both by the \c{bbot} worker +as well as the agent. In the later case the whole machine is shut down and no +operation-specific status or logs will be included (@@ Maybe we should just +include 'log:' with commands that start VM, for completeness?). -\li|\n\c{*-status: <status>}\n +The \c{abnormal} status indicates that the operation has terminated +abnormally, for example, due to the package manager or build system crash. - A per-operation result status. Note that the \c{*-status} values should - appear in the same order as the corresponding operations were performed - and for each \c{*-status} there should be a corresponding \c{*-log}.| +Note that the overall \c{status} value should appear before any per-operation +\c{*-status} values. -\li|\n\c{*-log: <text>}\n - A per-operation result log. Note that the \c{*-log} values should appear - last and in the same order as the corresponding \c{*-status} values.|| +\h2#arch-result-x-status|\c{*-status}| +\ +[*-status]: <status> +\ -\h#arch-task-req-manifest|Task Request Manifest| +The per-operation result status. Note that the \c{*-status} values should +appear in the same order as the corresponding operations were performed +and for each \c{*-status} there should be the corresponding \c{*-log} +value. Currently supported operation names: \ -SYNOPSIS +configure +update +test +install +test-installed +uninstall +\ + + +\h2#arch-result-x-log|\c{*-log}| -agent: <agent-name> -toolchain-name: <name> -toolchain-version: <standard-version> -fingerprint: <agent-fingerprint> \ +[*-log]: <text> +\ + +The per-operation result log. Note that the \c{*-log} values should appear +last and in the same order as the corresponding \c{*-status} values. For +the list of supported operation names refer to the \c{*-status} value +description. + + +\h#arch-task-req|Task Request Manifest| An agent (or controller acting as an agent) sends a task request to its controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request starts with the task request manifest followed by a list of machine manifests. +The task request manifest synopsis is presented next followed by the detailed +description of each value in subsequent sections. -\dl| +\ +agent: <name> +toolchain-name: <name> +toolchain-version: <standard-version> +[fingerprint]: <agent-fingerprint> +\ -\li|\n\c{agent: <agent-name>}\n - The name of the agent host (\c{hostname}). This should be unique in a - particular \c{bbot} deployment.| +\h2#arch-task-req-agent|\c{agent}| -\li|\n\c{toolchain-name: <name>}\n +\ +agent: <name> +\ - The \c{build2} toolchain name being used by the agent.| +The name of the agent host (\c{hostname}). The name should be unique in a +particular \c{bbot} deployment. -\li|\n\c{toolchain-version: <standard-version>}\n - The \c{build2} toolchain version being used by the agent.| +\h2#arch-task-req-toolchain-name|\c{toolchain-name}| -\li|\n\c{fingerprint: <agent-fingerprint>}\n +\ +toolchain-name: <name> +\ - The SHA256 fingerprint of the agent's public key. An agent may be configured - not to use the public key-based authentication in which case it does not - include this value. However, the controller may be configured to require - the authentication in which case it will respond with the - 401 (unauthorized) HTTP status code.|| +The \c{build2} toolchain name being used by the agent. -\h#arch-task-res-manifest|Task Response Manifest| + +\h2#arch-task-req-toolchain-version|\c{toolchain-version}| \ -SYNOPSIS +toolchain-version: <standard-version> +\ + +The \c{build2} toolchain version being used by the agent. + + +\h2#arch-task-req-fingerprint|\c{fingerprint}| -session: <session-id> -challenge: <text> -result-url: <url> \ +[fingerprint]: <agent-fingerprint> +\ + +The SHA256 fingerprint of the agent's public key. An agent may be configured +not to use the public key-based authentication in which case it does not +include this value. However, the controller may be configured to require the +authentication in which case it should respond with the 401 (unauthorized) +HTTP status code. + + +\h#arch-task-res|Task Response Manifest| A controller sends the task response manifest in response to the task request initiated by an agent. The response is delivered as a result of the POST method. The task response starts with the task response manifest optionally -followed by a task manifest. +followed by the task manifest. The task response manifest synopsis is +presented next followed by the detailed description of each value in +subsequent sections. -\dl| +\ +session: <id> +[challenge]: <text> +[result-url]: <url> +\ -\li|\n\c{session: <session-id>}\n - An identifier assigned to this session by the controller. An empty value - indicates that the controller has no tasks at this time in which case all - the following values as well as the task manifest are absent.| +\h2#arch-task-res-session|\c{session}| -\li|\n\c{challenge: <string>}\n +\ +session: <id> +\ - Random 64-character string (nonce) used to challenge the agent's private - key. If present, then the agent must sign this string and include the - signature in the result request. +The identifier assigned to this session by the controller. An empty value +indicates that the controller has no tasks at this time in which case all the +following values as well as the task manifest are absent. - The signature should be calculated by encrypting the string with the agent's - private key and then base64-encoding the result.| -\li|\n\c{result-url: <url>}\n +\h2#arch-task-res-challenge|\c{challenge}| - The URL to post the result (upload) request to.|| +\ +[challenge]: <text> +\ + +The random, 64 characters long string (nonce) used to challenge the agent's +private key. If present, then the agent must sign this string and include the +signature in the result request (see below). + +The signature should be calculated by encrypting the string with the agent's +private key and then \c{base64}-encoding the result. -\h#arch-result-req-manifest|Result Request Manifest| +\h2#arch-task-res-result-url|\c{result-url}| \ -SYNOPSIS +[result-url]: <url> +\ + +The URL to POST (upload) the result request to. + + +\h#arch-result-req|Result Request Manifest| -session: <session-id> -challenge: <text> +On completion of a task an agent (or controller acting as an agent) sends the +result (upload) request to the controller via the POST method using the URL +returned in the task response (see above). The result request starts with the +result request manifest followed by the result manifest. Note that there is no +result response and only a successful but empty POST result is returned. The +result request manifest synopsis is presented next followed by the detailed +description of each value in subsequent sections. + +\ +session: <id> +[challenge]: <text> \ -On completion of a task an agent (or controller acting as an agent) sends a -result (upload) request to the controller via HTTP/HTTPS POST method using the -URL returned in the task response. The result request starts with the -result request manifest followed by a result manifest. Note that there is no -result response and only a successful but empty POST result is returned. -\dl| +\h2#arch-result-req-session|\c{session}| + +\ +session: <session-id> +\ + +The session id as returned by the controller in the task response. -\li|\n\c{session: <session-id>}\n - The session id as returned by the controller in the task response.| +\h2#arch-result-req-challenge|\c{challenge}| -\li|\n\c{challenge: <text>}\n +\ +[challenge]: <text> +\ - The answer to the private key challenge as posed by the controller in the - task response. Must be present only if the challenge value was present in - the task response.|| +The answer to the private key challenge as posed by the controller in the task +response. It must be present only if the challenge value was present in the +task response. \h#arch-worker|Worker Logic| |