From 28b3d34d48146c47af3d29775832b094fb197acf Mon Sep 17 00:00:00 2001 From: Boris Kolpackov Date: Tue, 28 Mar 2017 11:01:13 +0200 Subject: Update manual with worker/controller logic, task manifest completion --- doc/manual.cli | 222 +++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 177 insertions(+), 45 deletions(-) diff --git a/doc/manual.cli b/doc/manual.cli index 49cf8d9..9be688b 100644 --- a/doc/manual.cli +++ b/doc/manual.cli @@ -78,13 +78,14 @@ owner if the build failed. Similarly, a \c{bbot} controller that monitors a build failure. The email would include a link (normally HTTP/HTTPS) to the build logs hosted by the controller. -\h2#arch-machine-config|Machine Configuration| +\h2#arch-machine-config|Configurations| -A build machine has a notion of a \i{machine configuration} that captures the -operating system, installed compiler toolchain, and so on. Note that the same -build machine may be used for multiple \i{build configurations}. For example, -the same machine can normally be used to produce 32/64-bit and debug/release -builds. +The \c{bbot} architecture distinguishes between a \i{machine configuration} +and a \i{build configuration}. The machine configuration captures the +operating system, installed compiler toolchain, and so on. The same build +machine may be used to \"generate\" multiple \i{build configurations}. For +example, the same machine can normally be used to produce 32/64-bit and +debug/release builds. The machine configuration is \i{approximately} encoded in its \i{machine name}. The machine name is a list of components separated with \c{-}. Each @@ -98,7 +99,7 @@ The first component normally identifies the operating system and has the following recommended form: \ -[_][_] +[_][_][_] \ For example: @@ -108,9 +109,9 @@ windows windows_10 windows_10.1607 i686_windows_xp -freebsd_10 -centos_6.2 -ubuntu_16.04 +bsd_freebsd_10 +linux_centos_6.2 +linux_ubuntu_16.04 macos_10.12 \ @@ -139,14 +140,22 @@ Some examples of complete machine names: \ windows_10-msvc_14u3 macos_10.12-clang -ubuntu_16.04-gcc_6.3 +linux_ubuntu_16.04-gcc_6.3 \ -\h2#arch-machine-manifest|Machine Manifest| +Similarly, the build configuration is encoded in a \i{configuration name} +using the same format. As described in \l{#arch-controller Controller Logic}, +build configurations are generated from machine configurations. As a result, +it usually makes sense to have the first component identify the operating +systems and the second component \- the toolchain with the rest identifying +a particular build configuration. For example: -The build machine manifest describes the build machine on the build host (see -the Build OS documentation for their origin and location). A list of machine -manifests is also sent by \c{bbot} agents to controllers. +\ +windows-vc_14-32-debug +linux-gcc-6-cross-arm-eabi +\ + +\h2#arch-machine-manifest|Machine Manifest| \ SYNOPSIS @@ -157,6 +166,10 @@ type: summary: \ +The build machine manifest describes the build machine on the build host (see +the Build OS documentation for their origin and location). A list of machine +manifests is also sent by \c{bbot} agents to controllers. + \dl| \li|\n\c{id: }\n @@ -189,8 +202,6 @@ summary: \h2#arch-task-manifest|Task Manifest| -The task manifest describes a build task. - \ SYNOPSIS @@ -198,8 +209,16 @@ name: version: repository: #location: + +machine: +target: +config: \ +The task manifest describes a build task. It consists of two groups of values. +The first group defines the package to build. The second group defines the +build configuration to use for building the package. + \dl| \li|\n\c{name: }\n @@ -212,23 +231,29 @@ repository: \li|\n\c{repository: }\n - The \c{bpkg} repository that contains the package and its dependencies.|| + The \c{bpkg} repository that contains the package and its dependencies.| -To test a package from a repository the \c{bbot} worker would execute -the following commands: +\li|\n\c{machine: }\n -\ -bpkg -v create [cxx config.cxx=g++-6] -bpkg -v add -bpkg -v fetch -bpkg -v build --yes --configure-only / -bpkg -v update -bpkg -v test -\ + The name of the build machine to use.| -\h2#arch-result-manifest|Result Manifest| +\li|\n\c{target: }\n -The result manifest describes a build result. + The target triplet to build for. If not specified, then the default target + for this machine is used (which is usually the machine itself). + + Compared to the autotools terminology, the \c{machine} value corresponds + to \c{--build} (the machine we are building on) and \c{target} \- to + \c{--host} (the machine we are building for). While we use essentially + the same \i{target triplet} format as autotools for \c{target}, it is + not flexible enough for \c{machine}.| + +\li|\n\c{config: }\n + + Additional build system configuration variables.|| + + +\h2#arch-result-manifest|Result Manifest| \ SYNOPSIS @@ -246,6 +271,8 @@ update-log: test-log: \ +The result manifest describes a build result. + \dl| \li|\n\c{name: }\n @@ -296,10 +323,6 @@ test-log: \h2#arch-task-req-manifest|Task Request Manifest| -An agent (or controller acting as an agent) sends a task request to its -controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request -starts with the task request manifest followed by a list of machine manifests. - \ SYNOPSIS @@ -307,6 +330,10 @@ agent: fingerprint: \ +An agent (or controller acting as an agent) sends a task request to its +controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request +starts with the task request manifest followed by a list of machine manifests. + \dl| \li|\n\c{agent: }\n @@ -321,11 +348,6 @@ fingerprint: \h2#arch-task-res-manifest|Task Response Manifest| -A controller sends the task response manifest in response to the task request -initiated by an agent. The response is delivered as a result of the POST -method. The task response starts with the task response manifest optionally -followed by a task manifest. - \ SYNOPSIS @@ -333,6 +355,10 @@ session: challenge: \ +An agent (or controller acting as an agent) sends a task request to its +controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request +starts with the task request manifest followed by a list of machine manifests. + \dl| \li|\n\c{session: }\n @@ -354,12 +380,6 @@ challenge: \h2#arch-result-req-manifest|Result Request Manifest| -On completion of a task an agent (or controller acting as an agent) sends a -result (upload) request to its controller via HTTP/HTTPS POST method (@@ -URL/API endpoint). The result request starts with the result request manifest -followed by a result manifest. Note that there is no result response and -only a successful but empty POST result is returned. - \ SYNOPSIS @@ -367,6 +387,12 @@ session: challenge: \ +On completion of a task an agent (or controller acting as an agent) sends a +result (upload) request to its controller via HTTP/HTTPS POST method (@@ +URL/API endpoint). The result request starts with the result request manifest +followed by a result manifest. Note that there is no result response and +only a successful but empty POST result is returned. + \dl| \li|\n\c{session: }\n @@ -377,4 +403,110 @@ challenge: The answer to the private key challenge as posed by the controller in the task response.|| + + +\h2#arch-worker|Worker Logic| + +The \c{bbot} worker builds each package in a \i{build environment} that is +established for a particular build target. The environment has three +components: the execution environment (environment variables, etc), build +system modules, and configuration variables. + +Setting up of the environment is performed by an executable (script, batch +file, etc). Specifically, upon receiving a build task, the worker obtains its +target and looks for the environment setup executable with this name in a +specific directory. If not found or if the target is unspecified, then the +worker looks for the executable called \c{default}. Not being able to locate +the environment executable is an error. + +Once the environment setup executable is determined, the worker re-executes +itself as that executable passing to it as command line arguments the target +name (or empty value if not specified), the path to the \c{bbot} worker to be +executed once the environment is setup, and the path to the build task +manifest. + +The environment setup executable sets up the necessary execution environment +for example by adjusting \c{PATH} or running a suitable \c{vcvars} batch file. +It then re-executes itself as the \c{bbot} worker passing to it as command +line arguments the path to the build task manifest followed by the list of +build system modules (\c{}) and the list of configuration +variables (\c{}). + +The re-executed \c{bbot} worker then proceeds to test the package from the +repository by executing the following commands (\c{<>}-values are from the +task manifest and environment): + +\ +bpkg -v create +bpkg -v add +bpkg -v fetch +bpkg -v build --yes --configure-only / +bpkg -v update +bpkg -v test +\ + +As an example, the following bash script can be used to setup the environment +for building C and C++ packages with GCC 6 on most Linux distributions. + +\ +#! /usr/bin/env bash + +# $1 - target +# $2 - bbot executable +# $3 - task manifest + +trap \"exit 1\" ERR + +if [ -n \"$1\" ]; then + echo \"unknown target $1\" 1>&2 + exit 1 +fi + +exec \"$2\" --build \"$3\" cc config.c=gcc-6 +\ + +\h2#arch-controller|Controller Logic| + +A \c{bbot} controller that issues own build tasks maps available build +machines (as reported by agents) to \i{build configurations} according to the +\c{conftab} configuration file. Blank lines and lines that start with \c{#} +are ignored. All other lines in this file have the following format: + +\ + [] [] +\ + +Where \c{} is filesystem wildcard pattern that is +matched against available machine names, \c{} is the +configuration name, optional \c{} is the build target, and +optional \c{} is a list of additional build system configuration +variables. The matched machine name, the target, and configuration variables +are included into the build task manifest. + +Note that each machine name is matched against every pattern and all the +patterns that match produce a configuration. If a machine does not match any +pattern, then it is ignored (meaning that this controller is not interested in +testing its packages with this machine). If multiple machines match the same +pattern, then only a single configuration using any of the machines is +produced (meaning that this controller considers these machines equivalent). + +As an example, let's say we have a machine named \c{windows_10-vc_14u3}. If +we wanted to test both 32 and 64-bit builds as well as debug and release, then +we could have generated the following configurations: + +\ +windows*-vc_14* windows-vc_14-32-debug i686-microsoft-win32-msvc14.0 config.cc.coptions=/Z7 config.cc.loptions=/DEBUG +windows*-vc_14* windows-vc_14-32-release i686-microsoft-win32-msvc14.0 config.cc.coptions=\"/O2 /Oi\" +windows*-vc_14* windows-vc_14-64-debug x86_64-microsoft-win32-msvc14.0 config.cc.coptions=/Z7 config.cc.loptions=/DEBUG +windows*-vc_14* windows-vc_14-64-release x86_64-microsoft-win32-msvc14.0 config.cc.coptions=\"/O2 /Oi\" +\ + +As another example, let's say we have \c{linux_fedora_25-gcc_6} and +\c{linux_ubuntu_16.04-gcc_6}. If all we cared about it testing GCC 6 on Linux, +then our configurations could look like this (note the missing target): + +\ +linux*-gcc-6 linux-gcc_6-debug config.cc.coptions=-g +linux*-gcc-6 linux-gcc_6-release config.cc.coptions=-O3 +\ " -- cgit v1.1