path: root/doc/manual.cli

// file      : doc/manual.cli
// license   : MIT; see accompanying LICENSE file

"\name=build2-repository-interface-manual"
"\subject=repository interface"
"\title=Repository Interface"

// NOTES
//
// - Maximum <pre> line is 70 characters.
//

"
\h0#preface|Preface|

This document describes \c{brep}, the \c{build2} package repository web
interface. For the command line interface of \c{brep} utilities refer to the
\l{brep-load(1)}, \l{brep-clean(1)}, \l{brep-migrate(1)}, and
\l{brep-monitor(1)} man pages.

\h1#submit|Package Submission|

The package submission functionality allows uploading of package archives as
well as additional, repository-specific information via the HTTP \c{POST}
method using the \c{multipart/form-data} content type. The implementation in
\c{brep} only handles uploading as well as basic verification (checksum,
duplicates) expecting the rest of the submission and publishing logic to be
handled by a separate entity according to the repository policy. Such an
entity can be notified by \c{brep} about a new submission as an invocation of
the \i{handler program} (as part of the HTTP request) and/or via email. It
could also be a separate process that monitors the upload data directory.

The submission request without any parameters is treated as the submission
form request. If \c{submit-form} is configured, then such a form is generated
and returned. Otherwise, such a request is treated as an invalid submission
(missing parameters).

For each submission request \c{brep} performs the following steps.

\ol|

\li|Verify submission size limit.

The submission form-data payload size must not exceed \c{submit-max-size}.|

\li|Verify the required \c{archive} and \c{sha256sum} parameters are present.

The \c{archive} parameter must be the package archive upload while
\c{sha256sum} must be its 64 characters SHA256 checksum calculated in the
binary mode.|

\li|Verify other parameters are valid manifest name/value pairs.

The value can only contain UTF-8 encoded Unicode graphic characters as well as
tab (\c{\\t}), carriage return (\c{\\r}), and line feed (\c{\\n}).|

\li|Check for a duplicate submission.

Each submission is saved as a subdirectory in the \c{submit-data} directory
with a 12-character abbreviated checksum as its name.|

\li|Save the package archive into a temporary directory and verify its
checksum.

A temporary subdirectory is created in the \c{submit-temp} directory, the
package archive is saved into it using the submitted name, and its checksum
is calculated and compared to the submitted checksum.|

\li|Save the submission request manifest into the temporary directory.

The submission request manifest is saved as \c{request.manifest} into the
temporary subdirectory next to the archive.|

\li|Make the temporary submission directory permanent.

Move/rename the temporary submission subdirectory to \c{submit-data} as an
atomic operation using the 12-character abbreviated checksum as its new
name. If such a directory already exist, then this is a duplicate submission.|

\li|Invoke the submission handler program.

If \c{submit-handler} is configured, invoke the handler program passing to it
additional arguments specified with \c{submit-handler-argument} (if any)
followed by the absolute path to the submission directory.

The handler program is expected to write the submission result manifest to
\c{stdout} and terminate with the zero exit status. A non-zero exit status is
treated as an internal error. The handler program's \c{stderr} is logged.

Note that the handler program should report temporary server errors (service
overload, network connectivity loss, etc.) via the submission result manifest
status values in the [500-599] range (HTTP server error) rather than via a
non-zero exit status.

The handler program assumes ownership of the submission directory and can
move/remove it. If after the handler program terminates the submission
directory still exists, then it is handled by \c{brep} depending on the
handler process exit status and the submission result manifest status value.
If the process has terminated abnormally or with a non-zero exit status or the
result manifest status is in the [500-599] range (HTTP server error), then the
directory is saved for troubleshooting by appending the \c{.fail} extension
followed by a numeric extension to its name (for example,
\c{ff5a1a53d318.fail.1}). Otherwise, if the status is in the [400-499] range
(HTTP client error), then the directory is removed. If the directory is left
in place by the handler or is saved for troubleshooting, then the submission
result manifest is saved as \c{result.manifest} into this directory, next to
the request manifest and archive.

If \c{submit-handler-timeout} is configured and the handler program does not
exit in the allotted time, then it is killed and its termination is treated as
abnormal.

If the handler program is not specified, then the following submission result
manifest is implied:

\
status: 200
message: package submission is queued
reference: <abbrev-checksum>
\

|


\li|Send the submission email.

If \c{submit-email} is configured, send an email to this address containing
the submission request manifest and the submission result manifest.|

\li|Respond to the client.

Respond to the client with the submission result manifest and its \c{status}
value as the HTTP status code.|

|

Check violations (max size, duplicate submissions, etc) that are explicitly
mentioned above are always reported with the submission result manifest. Other
errors (for example, internal server errors) might be reported with
unformatted text, including HTML.

If the submission request contains the \c{simulate} parameter, then the
submission service simulates the specified outcome of the submission process
without actually performing any externally visible actions (e.g., publishing
the package, notifying the submitter, etc). Note that the package submission
email (\c{submit-email}) is not sent for simulated submissions.

Pre-defined simulation outcome values are \c{internal-error-text},
\c{internal-error-html}, \c{duplicate-archive}, and \c{success}. The
simulation outcome is included into the submission request manifest and the
handler program must at least handle \c{success} but may recognize additional
outcomes.


\h#submit-request-manifest|Submission Request Manifest|

The submission request manifest starts with the below values and in that order
optionally followed by additional values in the unspecified order
corresponding to the custom request parameters.

\
archive: <name>
sha256sum: <sum>
timestamp: <date-time>
[simulate]: <outcome>
[client-ip]: <string>
[user-agent]: <string>
\

The \c{timestamp} value is in the ISO-8601 \c{<YYYY>-<MM>-<DD>T<hh>:<mm>:<ss>Z}
form (always UTC). Note also that \c{client-ip} can be IPv4 or IPv6.


\h#submit-result-manifest|Submission Result Manifest|

The submission result manifest starts with the below values and in that order
optionally followed by additional values if returned by the handler program.
If the submission is successful, then the \c{reference} value must be present
and contain a string that can be used to identify this submission (for
example, the abbreviated checksum).

\
status: <http-code>
message: <string>
[reference]: <string>
\


\h1#ci|Package CI|

The CI functionality allows submission of package CI requests as well as
additional, repository-specific information via the HTTP \c{GET} and \c{POST}
methods using the \c{application/x-www-form-urlencoded} or
\c{multipart/form-data} parameters encoding. The implementation in \c{brep}
only handles reception as well as basic parameter verification expecting the
rest of the CI logic to be handled by a separate entity according to the
repository policy. Such an entity can be notified by \c{brep} about a new CI
request as an invocation of the \i{handler program} (as part of the HTTP
request) and/or via email. It could also be a separate process that monitors
the CI data directory.

The CI request without any parameters is treated as the CI form request. If
\c{ci-form} is configured, then such a form is generated and returned.
Otherwise, such a request is treated as an invalid CI request (missing
parameters).

For each CI request \c{brep} performs the following steps.

\ol|

\li|Verify the required \c{repository} and optional \c{package} parameters.

The \c{repository} parameter is the remote \c{bpkg} repository location that
contains the packages to be tested. If one or more \c{package} parameters are
present, then only the specified packages are tested. If no \c{package}
parameters are specified, then all the packages present in the repository (but
excluding complement repositories) are tested.

Each \c{package} parameter can specify either just the package name, in which
case all the versions of this package present in the repository will be
tested, or both the name and version in the \c{<name>/<version>} form (for
example, \c{libhello/1.2.3}.|

\li|Verify the optional \c{overrides} parameter.

The overrides parameter, if specified, must be the CI overrides manifest
upload.|

\li|Verify other parameters are valid manifest name/value pairs.

The value can only contain UTF-8 encoded Unicode graphic characters as well as
tab (\c{\\t}), carriage return (\c{\\r}), and line feed (\c{\\n}).|

\li|Generate CI request id and create request directory.

For each CI request a unique id (UUID) is generated and a request subdirectory
is created in the \c{ci-data} directory with this id as its name.|

\li|Save the CI request manifest into the request directory.

The CI request manifest is saved as \c{request.manifest} into the request
subdirectory created on the previous step.|

\li|Save the CI overrides manifest into the request directory.

If the CI overrides manifest is uploaded, then it is saved as
\c{overrides.manifest} into the request subdirectory.|

\li|Invoke the CI handler program.

If \c{ci-handler} is configured, invoke the handler program passing to it
additional arguments specified with \c{ci-handler-argument} (if any) followed
by the absolute path to the CI request directory.

The handler program is expected to write the CI result manifest to \c{stdout}
and terminate with the zero exit status. A non-zero exit status is treated as
an internal error. The handler program's \c{stderr} is logged.

Note that the handler program should report temporary server errors (service
overload, network connectivity loss, etc.) via the CI result manifest status
values in the [500-599] range (HTTP server error) rather than via a non-zero
exit status.

The handler program assumes ownership of the CI request directory and can
move/remove it. If after the handler program terminates the request directory
still exists, then it is handled by \c{brep} depending on the handler process
exit status and the CI result manifest status value. If the process has
terminated abnormally or with a non-zero exit status or the result manifest
status is in the [500-599] range (HTTP server error), then the directory is
saved for troubleshooting by appending the \c{.fail} extension to its
name. Otherwise, if the status is in the [400-499] range (HTTP client error),
then the directory is removed. If the directory is left in place by the
handler or is saved for troubleshooting, then the CI result manifest
is saved as \c{result.manifest} into this directory, next to the request
manifest.

If \c{ci-handler-timeout} is configured and the handler program does not
exit in the allotted time, then it is killed and its termination is treated as
abnormal.

If the handler program is not specified, then the following CI result
manifest is implied:

\
status: 200
message: CI request is queued
reference: <request-id>
\

|

\li|Send the CI request email.

If \c{ci-email} is configured, send an email to this address containing the CI
request manifest, the potentially empty CI overrides manifest, and the CI
result manifest.|

\li|Respond to the client.

Respond to the client with the CI result manifest and its \c{status} value as
the HTTP status code.|

|

Check violations that are explicitly mentioned above are always reported with
the CI result manifest. Other errors (for example, internal server errors)
might be reported with unformatted text, including HTML.

If the CI request contains the \c{simulate} parameter, then the CI service
simulates the specified outcome of the CI process without actually performing
any externally visible actions (e.g., testing the package, publishing the
result, etc). Note that the CI request email (\c{ci-email}) is not sent for
simulated requests.

Pre-defined simulation outcome values are \c{internal-error-text},
\c{internal-error-html}, and \c{success}. The simulation outcome is included
into the CI request manifest and the handler program must at least handle
\c{success} but may recognize additional outcomes.


\h#ci-request-manifest|CI Request Manifest|

The CI request manifest starts with the below values and in that order
optionally followed by additional values in the unspecified order
corresponding to the custom request parameters.

\
id: <request-id>
repository: <url>
[package]: <name>[/<version>]
timestamp: <date-time>
[simulate]: <outcome>
[client-ip]: <string>
[user-agent]: <string>
\

The \c{package} value can be repeated multiple times. The \c{timestamp} value
is in the ISO-8601 \c{<YYYY>-<MM>-<DD>T<hh>:<mm>:<ss>Z} form (always
UTC). Note also that \c{client-ip} can be IPv4 or IPv6.


\h#ci-overrides-manifest|CI Overrides Manifest|

The CI overrides manifest is a package manifest fragment that should be
applied to all the packages being tested. The contained values override the
whole value groups they belong to, resetting all the group values prior to
being applied. Currently, only the following value groups can be overridden:

\
build-email build-{warning,error}-email
builds build-{include,exclude}
\

See \l{bpkg#manifest-package Package Manifest} for details on these values.


\h#ci-result-manifest|CI Result Manifest|

The CI result manifest starts with the below values and in that order
optionally followed by additional values if returned by the handler program.
If the CI request is successful, then the \c{reference} value must be present
and contain a string that can be used to identify this request (for example,
the CI request id).

\
status: <http-code>
message: <string>
[reference]: <string>
\

"