Update manual with worker/controller logic, task manifest completion

1 files changed, 177 insertions, 45 deletions

diff --git a/doc/manual.cli b/doc/manual.cli
index 49cf8d9..9be688b 100644
--- a/doc/manual.cli
+++ b/doc/manual.cli
@@ -78,13 +78,14 @@ owner if the build failed. Similarly, a \c{bbot} controller that monitors a
 build failure. The email would include a link (normally HTTP/HTTPS) to the
 build logs hosted by the controller.
 
-\h2#arch-machine-config|Machine Configuration|
+\h2#arch-machine-config|Configurations|
 
-A build machine has a notion of a \i{machine configuration} that captures the
-operating system, installed compiler toolchain, and so on. Note that the same
-build machine may be used for multiple \i{build configurations}. For example,
-the same machine can normally be used to produce 32/64-bit and debug/release
-builds.
+The \c{bbot} architecture distinguishes between a \i{machine configuration}
+and a \i{build configuration}. The machine configuration captures the
+operating system, installed compiler toolchain, and so on. The same build
+machine may be used to \"generate\" multiple \i{build configurations}. For
+example, the same machine can normally be used to produce 32/64-bit and
+debug/release builds.
 
 The machine configuration is \i{approximately} encoded in its \i{machine
 name}. The machine name is a list of components separated with \c{-}. Each
@@ -98,7 +99,7 @@ The first component normally identifies the operating system and has the
 following recommended form:
 
 \
-[<arch>_]<os>[_<version>]
+[<arch>_][<class>_]<os>[_<version>]
 \
 
 For example:
@@ -108,9 +109,9 @@ windows
 windows_10
 windows_10.1607
 i686_windows_xp
-freebsd_10
-centos_6.2
-ubuntu_16.04
+bsd_freebsd_10
+linux_centos_6.2
+linux_ubuntu_16.04
 macos_10.12
 \
 
@@ -139,14 +140,22 @@ Some examples of complete machine names:
 \
 windows_10-msvc_14u3
 macos_10.12-clang
-ubuntu_16.04-gcc_6.3
+linux_ubuntu_16.04-gcc_6.3
 \
 
-\h2#arch-machine-manifest|Machine Manifest|
+Similarly, the build configuration is encoded in a \i{configuration name}
+using the same format. As described in \l{#arch-controller Controller Logic},
+build configurations are generated from machine configurations. As a result,
+it usually makes sense to have the first component identify the operating
+systems and the second component \- the toolchain with the rest identifying
+a particular build configuration. For example:
 
-The build machine manifest describes the build machine on the build host (see
-the Build OS documentation for their origin and location). A list of machine
-manifests is also sent by \c{bbot} agents to controllers.
+\
+windows-vc_14-32-debug
+linux-gcc-6-cross-arm-eabi
+\
+
+\h2#arch-machine-manifest|Machine Manifest|
 
 \
 SYNOPSIS
@@ -157,6 +166,10 @@ type:    <machine-type>
 summary: <string>
 \
 
+The build machine manifest describes the build machine on the build host (see
+the Build OS documentation for their origin and location). A list of machine
+manifests is also sent by \c{bbot} agents to controllers.
+
 \dl|
 
 \li|\n\c{id: <machine-id>}\n
@@ -189,8 +202,6 @@ summary: <string>
 
 \h2#arch-task-manifest|Task Manifest|
 
-The task manifest describes a build task.
-
 \
 SYNOPSIS
 
@@ -198,8 +209,16 @@ name:       <package-name>
 version:    <package-version>
 repository: <repository-url>
 #location:  <package-url>
+
+machine: <machine-name>
+target:  <target-triplet>
+config:  <config-vars>
 \
 
+The task manifest describes a build task. It consists of two groups of values.
+The first group defines the package to build. The second group defines the
+build configuration to use for building the package.
+
 \dl|
 
 \li|\n\c{name: <package-name>}\n
@@ -212,23 +231,29 @@ repository: <repository-url>
 
 \li|\n\c{repository: <repository-url>}\n
 
-  The \c{bpkg} repository that contains the package and its dependencies.||
+  The \c{bpkg} repository that contains the package and its dependencies.|
 
-To test a package from a repository the \c{bbot} worker would execute
-the following commands:
+\li|\n\c{machine: <machine-name>}\n
 
-\
-bpkg -v create [cxx config.cxx=g++-6]
-bpkg -v add <repository-url>
-bpkg -v fetch
-bpkg -v build --yes --configure-only <package-name>/<package-version>
-bpkg -v update <package-name>
-bpkg -v test <package-name>
-\
+  The name of the build machine to use.|
 
-\h2#arch-result-manifest|Result Manifest|
+\li|\n\c{target: <target-triplet>}\n
 
-The result manifest describes a build result.
+  The target triplet to build for. If not specified, then the default target
+  for this machine is used (which is usually the machine itself).
+
+  Compared to the autotools terminology, the \c{machine} value corresponds
+  to \c{--build} (the machine we are building on) and \c{target} \- to
+  \c{--host} (the machine we are building for). While we use essentially
+  the same \i{target triplet} format as autotools for \c{target}, it is
+  not flexible enough for \c{machine}.|
+
+\li|\n\c{config: <config-vars>}\n
+
+  Additional build system configuration variables.||
+
+
+\h2#arch-result-manifest|Result Manifest|
 
 \
 SYNOPSIS
@@ -246,6 +271,8 @@ update-log:    <text>
 test-log:      <text>
 \
 
+The result manifest describes a build result.
+
 \dl|
 
 \li|\n\c{name: <package-name>}\n
@@ -296,10 +323,6 @@ test-log:      <text>
 
 \h2#arch-task-req-manifest|Task Request Manifest|
 
-An agent (or controller acting as an agent) sends a task request to its
-controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request
-starts with the task request manifest followed by a list of machine manifests.
-
 \
 SYNOPSIS
 
@@ -307,6 +330,10 @@ agent:       <agent-name>
 fingerprint: <agent-fingerprint>
 \
 
+An agent (or controller acting as an agent) sends a task request to its
+controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request
+starts with the task request manifest followed by a list of machine manifests.
+
 \dl|
 
 \li|\n\c{agent: <agent-name>}\n
@@ -321,11 +348,6 @@ fingerprint: <agent-fingerprint>
 
 \h2#arch-task-res-manifest|Task Response Manifest|
 
-A controller sends the task response manifest in response to the task request
-initiated by an agent. The response is delivered as a result of the POST
-method. The task response starts with the task response manifest optionally
-followed by a task manifest.
-
 \
 SYNOPSIS
 
@@ -333,6 +355,10 @@ session:   <session-id>
 challenge: <text>
 \
 
+An agent (or controller acting as an agent) sends a task request to its
+controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request
+starts with the task request manifest followed by a list of machine manifests.
+
 \dl|
 
 \li|\n\c{session: <session-id>}\n
@@ -354,12 +380,6 @@ challenge: <text>
 
 \h2#arch-result-req-manifest|Result Request Manifest|
 
-On completion of a task an agent (or controller acting as an agent) sends a
-result (upload) request to its controller via HTTP/HTTPS POST method (@@
-URL/API endpoint). The result request starts with the result request manifest
-followed by a result manifest. Note that there is no result response and
-only a successful but empty POST result is returned.
-
 \
 SYNOPSIS
 
@@ -367,6 +387,12 @@ session:   <session-id>
 challenge: <text>
 \
 
+On completion of a task an agent (or controller acting as an agent) sends a
+result (upload) request to its controller via HTTP/HTTPS POST method (@@
+URL/API endpoint). The result request starts with the result request manifest
+followed by a result manifest. Note that there is no result response and
+only a successful but empty POST result is returned.
+
 \dl|
 
 \li|\n\c{session: <session-id>}\n
@@ -377,4 +403,110 @@ challenge: <text>
 
   The answer to the private key challenge as posed by the controller in the
   task response.||
+
+
+\h2#arch-worker|Worker Logic|
+
+The \c{bbot} worker builds each package in a \i{build environment} that is
+established for a particular build target. The environment has three
+components: the execution environment (environment variables, etc), build
+system modules, and configuration variables.
+
+Setting up of the environment is performed by an executable (script, batch
+file, etc). Specifically, upon receiving a build task, the worker obtains its
+target and looks for the environment setup executable with this name in a
+specific directory. If not found or if the target is unspecified, then the
+worker looks for the executable called \c{default}. Not being able to locate
+the environment executable is an error.
+
+Once the environment setup executable is determined, the worker re-executes
+itself as that executable passing to it as command line arguments the target
+name (or empty value if not specified), the path to the \c{bbot} worker to be
+executed once the environment is setup, and the path to the build task
+manifest.
+
+The environment setup executable sets up the necessary execution environment
+for example by adjusting \c{PATH} or running a suitable \c{vcvars} batch file.
+It then re-executes itself as the \c{bbot} worker passing to it as command
+line arguments the path to the build task manifest followed by the list of
+build system modules (\c{<env-modules>}) and the list of configuration
+variables (\c{<env-config-vars>}).
+
+The re-executed \c{bbot} worker then proceeds to test the package from the
+repository by executing the following commands (\c{<>}-values are from the
+task manifest and environment):
+
+\
+bpkg -v create <env-module> <config-vars> <env-config-vars>
+bpkg -v add <repository-url>
+bpkg -v fetch
+bpkg -v build --yes --configure-only <package-name>/<package-version>
+bpkg -v update <package-name>
+bpkg -v test <package-name>
+\
+
+As an example, the following bash script can be used to setup the environment
+for building C and C++ packages with GCC 6 on most Linux distributions.
+
+\
+#! /usr/bin/env bash
+
+# $1 - target
+# $2 - bbot executable
+# $3 - task manifest
+
+trap \"exit 1\" ERR
+
+if [ -n \"$1\" ]; then
+  echo \"unknown target $1\" 1>&2
+  exit 1
+fi
+
+exec \"$2\" --build \"$3\" cc config.c=gcc-6
+\
+
+\h2#arch-controller|Controller Logic|
+
+A \c{bbot} controller that issues own build tasks maps available build
+machines (as reported by agents) to \i{build configurations} according to the
+\c{conftab} configuration file. Blank lines and lines that start with \c{#}
+are ignored. All other lines in this file have the following format:
+
+\
+<machine-name-pattern> <config-name> [<target>] [<config-vars>]
+\
+
+Where \c{<machine-name-pattern>} is filesystem wildcard pattern that is
+matched against available machine names, \c{<config-name>} is the
+configuration name, optional \c{<target>} is the build target, and
+optional \c{<config-vars>} is a list of additional build system configuration
+variables. The matched machine name, the target, and configuration variables
+are included into the build task manifest.
+
+Note that each machine name is matched against every pattern and all the
+patterns that match produce a configuration. If a machine does not match any
+pattern, then it is ignored (meaning that this controller is not interested in
+testing its packages with this machine). If multiple machines match the same
+pattern, then only a single configuration using any of the machines is
+produced (meaning that this controller considers these machines equivalent).
+
+As an example, let's say we have a machine named \c{windows_10-vc_14u3}.  If
+we wanted to test both 32 and 64-bit builds as well as debug and release, then
+we could have generated the following configurations:
+
+\
+windows*-vc_14* windows-vc_14-32-debug   i686-microsoft-win32-msvc14.0   config.cc.coptions=/Z7 config.cc.loptions=/DEBUG
+windows*-vc_14* windows-vc_14-32-release i686-microsoft-win32-msvc14.0   config.cc.coptions=\"/O2 /Oi\"
+windows*-vc_14* windows-vc_14-64-debug   x86_64-microsoft-win32-msvc14.0 config.cc.coptions=/Z7 config.cc.loptions=/DEBUG
+windows*-vc_14* windows-vc_14-64-release x86_64-microsoft-win32-msvc14.0 config.cc.coptions=\"/O2 /Oi\"
+\
+
+As another example, let's say we have \c{linux_fedora_25-gcc_6} and
+\c{linux_ubuntu_16.04-gcc_6}. If all we cared about it testing GCC 6 on Linux,
+then our configurations could look like this (note the missing target):
+
+\
+linux*-gcc-6 linux-gcc_6-debug   config.cc.coptions=-g
+linux*-gcc-6 linux-gcc_6-release config.cc.coptions=-O3
+\
 "