aboutsummaryrefslogtreecommitdiff
path: root/INSTALL
diff options
context:
space:
mode:
Diffstat (limited to 'INSTALL')
-rw-r--r--INSTALL229
1 files changed, 114 insertions, 115 deletions
diff --git a/INSTALL b/INSTALL
index 59ec07e..79d698f 100644
--- a/INSTALL
+++ b/INSTALL
@@ -11,8 +11,9 @@ can be omitted.
1. Create 'brep' User
This user will be used to run the brep package database loader, build database
-cleaner, and the database schemes migration utility. We will also use its home
-directory to build and install the brep module, store its configuration, etc.
+cleaner, monitor, and database schemas migration utility. We will also use its
+home directory to build and install the brep module, store its configuration,
+etc.
Note: if the deployment machine employs SELinux, then this approach may
require additional configuration steps (not shown) in order to allow Apache2
@@ -43,7 +44,7 @@ b) Install PostgreSQL 9.3 or above (including the contrib package containing
the postgres_fdw extension) as well as Apache2 using your distribution's
packages. Below are the names of these packages for some distributions:
- Debian/Ubuntu: postgresql-server postgresql-contrib apache2
+ Debian/Ubuntu: postgresql postgresql-contrib apache2
Fedora/RHEL: postgresql-server postgresql-contrib httpd
FreeBSD: postgresqlXY-server postgresqlXY-contrib apache24
@@ -65,7 +66,7 @@ c) Install PostgreSQL and Apache2 development files. Specifically, we need
files. Below are the names of their packages for some distributions:
Debian/Ubuntu: libpq-dev libapr1-dev libapreq2-dev apache2-dev
- Fedora/RHEL: posqtgresql-devel apr-devel libapreq2-devel httpd-devel
+ Fedora/RHEL: postgresql-devel apr-devel libapreq2-devel httpd-devel
FreeBSD: postgresqlXY-client apr libapreq2 apache24
d) Unless you already have the build2 toolchain, install it by following
@@ -76,24 +77,13 @@ d) Unless you already have the build2 toolchain, install it by following
3. Build and Install brep
-Normally the only extra information that you need to provide on this step is
-the location of the Apache2 headers (httpd.h, etc). Below are their locations
-for some distributions:
-
-Debian/Ubuntu: /usr/include/apache2
-Fedora/RHEL: /usr/include/httpd
-FreeBSD: /usr/local/include/apache24
-
-You can also use the Apache2 apxs utility to obtain this information as shown
-below.
-
$ mkdir brep
$ cd brep
-$ bpkg create \
- cc \
- config.cc.poptions="-I$(apxs -q includedir)" \
- config.bin.rpath=$HOME/install/lib \
+$ bpkg create \
+ cc \
+ config.cc.coptions=-O3 \
+ config.bin.rpath=$HOME/install/lib \
config.install.root=$HOME/install
$ bpkg add https://pkg.cppget.org/1/alpha
@@ -103,6 +93,22 @@ $ bpkg install brep
$ cd .. # Back to brep home.
+Note that by default the location of the Apache2 headers (httpd.h, etc) is
+detected automatically, using the Apache2 apxs utility. Below are their
+locations for some distributions:
+
+Debian/Ubuntu: /usr/include/apache2
+Fedora/RHEL: /usr/include/httpd
+FreeBSD: /usr/local/include/apache24
+
+To disable this functionality and specify the location explicitly, you can
+configure brep in the development mode and specify the respective preprocessor
+option by, for example, adding the following configuration variables to the
+above bpkg-build command:
+
+config.brep.develop=true
+config.cc.poptions="-I..."
+
4. Create PostgreSQL User and Databases
@@ -157,7 +163,7 @@ CREATE EXTENSION postgres_fdw;
CREATE SERVER package_server
FOREIGN DATA WRAPPER postgres_fdw
- OPTIONS (dbname 'brep_package', updatable 'false');
+ OPTIONS (dbname 'brep_package', updatable 'true');
GRANT USAGE ON FOREIGN SERVER package_server to brep;
@@ -165,6 +171,17 @@ CREATE USER MAPPING FOR PUBLIC
SERVER package_server
OPTIONS (user 'brep-build', password '-');
+Note that starting with PostgreSQL 15 only the database owner can create the
+objects in the public schema by default. Thus, if the PostgreSQL version is 15
+or above, then all the privileges on this schema in the created databases need
+to be granted explicitly by the postgres user to the brep user:
+
+\c brep_package
+GRANT ALL PRIVILEGES ON SCHEMA public TO brep;
+
+\c brep_build
+GRANT ALL PRIVILEGES ON SCHEMA public TO brep;
+
Exit psql (^D)
The user brep-build is required (by the postgres_fdw extension) to login with
@@ -194,7 +211,7 @@ CREATE EXTENSION citext;
Exit psql (^D)
-5. Create Database Schemes and Load Repositories
+5. Create Database Schemas and Load Repositories
$ mkdir config
$ edit config/loadtab # Loader configuration, see brep-load(1).
@@ -217,7 +234,21 @@ $ cp install/share/brep/etc/brep-module.conf config/
$ edit config/brep-module.conf # Adjust default values if required.
To enable the build2 build bot controller functionality you will need to set
-the build-config option in brep-module.conf.
+the build-config option in brep-module.conf. To also enable the build
+artifacts upload functionality you will need to specify the upload-data
+directory for the desired upload types in brep-module.conf. For example, for
+generated binary distribution packages it can be as follows:
+
+upload-data bindist=/home/brep/bindist-data
+
+Note that this directory must exist and have read, write, and execute
+permissions granted to the www-data user. This, for example, can be achieved
+with the following commands:
+
+$ mkdir /home/brep/bindist-data
+$ setfacl -m g:www-data:rwx /home/brep/bindist-data
+
+For sample upload handler implementations see brep/handler/upload/.
To enable the package submission functionality you will need to specify the
submit-data and submit-temp directories in brep-module.conf. Note that these
@@ -261,75 +292,8 @@ $ edit config/ci.xhtml # Add custom form fields, adjust CSS style, etc.
For sample CI request handler implementations see brep/handler/ci/.
Here we assume you have setup an appropriate Apache2 virtual server. Open the
-corresponding Apache2 .conf file and add the following inside VirtualHost (you
-can also find this fragment in install/share/brep/etc/brep-apache2.conf):
-
- # Load the brep module.
- #
- <IfModule !brep_module>
- LoadModule brep_module /home/brep/install/libexec/brep/mod_brep.so
- </IfModule>
-
- # Repository email. This email is used for the From: header in emails
- # send by brep (for example, build failure notifications).
- #
- brep-email admin@example.org
-
- # Repository host. It specifies the scheme and the host address (but
- # not the root path; see brep-root below) that will be used whenever
- # brep needs to construct an absolute URL to one of its locations (for
- # example, a link to a build log that is being send via email).
- #
- brep-host https://example.org
-
- # Repository root. This is the part of the URL between the host name
- # and the start of the repository. For example, root value /pkg means
- # the repository URL is http://example.org/pkg/. Specify / to use the
- # web server root (e.g., http://example.org/). If using a different
- # repository root, don't forget to also change Location and Alias
- # directives below.
- #
- brep-root /pkg
-
- <Location "/pkg">
- SetHandler brep
-
- <IfModule dir_module>
- DirectoryIndex disabled
- DirectorySlash Off
- </IfModule>
- </Location>
-
- # Brep module configuration. If you prefer, you can paste the contents
- # of this file here. However, you will need to prefix every option with
- # 'brep-'.
- #
- brep-conf /home/brep/config/brep-module.conf
-
- # Static brep content (CSS files).
- #
- <IfModule !alias_module>
- Error "mod_alias is not enabled"
- </IfModule>
-
- # Note: trailing slashes are important!
- #
- Alias /pkg/@/ /home/brep/install/share/brep/www/
-
- <Directory "/home/brep/install/share/brep/www">
- Require all granted
- </Directory>
-
- # You can also serve the repository files from the repository root.
- # For example:
- #
- # http://example.org/pkg/1/... -> /path/to/repo/1/...
- #
- #AliasMatch ^/pkg/(\d+)/(.+) /path/to/repo/$1/$2
- #
- #<Directory "/path/to/repo">
- # Require all granted
- #</Directory>
+corresponding Apache2 .conf file and add the contents of
+brep/etc/brep-apache2.conf into the <VirtualHost> section.
The output content types of the brep module are application/xhtml+xml,
text/manifest and text/plain. If you would like to make sure they get
@@ -384,47 +348,71 @@ $ cd install/share/brep/www/
$ for i in *.scss; do sassc -s compressed $i `basename -s .scss $i`.css; done
-8. Setup Periodic Loader and Cleaner Execution
+8. Setup Periodic Loader, Cleaner, and Monitor Execution
Initially this guide suggested using systemd user session support to run the
-loader and the cleaner. However, the current state of user sessions has one
-major drawback: they are not started/attached-to when logging in with su -l
-(see Debian bug #813789 for details). This limitation makes them unusable in
-our setup. If you still would like to use systemd to run the loader and the
-cleaner, then you can set it up as a system-wide service which runs the
-utilities as the brep user/group. Otherwise, a cron job is a natural choice.
+loader, cleaner, and monitor. However, the current state of user sessions has
+one major drawback: they are not started/attached-to when logging in with su
+-l (see Debian bug #813789 for details). This limitation makes them unusable
+in our setup. If you still would like to use systemd to run the utilities,
+then you can set it up as a system-wide service which runs them as the brep
+user/group. Otherwise, a cron job is a natural choice.
Note that the builds cleaner execution is optional and is only required if the
build2 build bot functionality is enabled (see the build bot documentation for
-details). If it is disabled in you setup, then skip the cleaner-related
-parts in the subsequent subsections.
+details). If it is disabled in you setup, then skip the cleaner-related parts
+in the subsequent subsections.
+
+If the build artifacts upload functionality is enabled in addition to the
+build2 build bot functionality you most likely will want to additionally setup
+the cleanup of the outdated build artifacts. For example, for binary
+distribution package uploads handled by brep-upload-bindist the cleanup needs
+to be performed by periodic execution of brep-upload-bindist-clean script.
+Note that the directory where the uploads are saved to must exist and have
+read, write, and execute permissions granted to the brep user. This, for
+example, can be achieved with the following commands:
+
+# mkdir /var/bindist
+# chown www-data:www-data /var/bindist
+# setfacl -m u:brep:rwx /var/bindist
+# setfacl -dm u:brep:rwx /var/bindist
If the CI request functionality is enabled you most likely will want to
additionally setup the tenants cleanup.
+The monitor execution is also optional and currently only makes sense if the
+build2 build bot functionality is enabled. Note that you may need to replace
+the public toolchain name argument in the monitor utility command with a real
+list of toolchain names (and optionally versions) used in the brep build
+infrastructure.
+
-8.a Setup Periodic Loader and Cleaner Execution with cron
+8.a Setup Periodic Loader, Cleaner, and Monitor Execution with cron
-The following crontab entries will execute the loader every five minutes
-and the tenants and builds cleaners once a day at midnight:
+The following crontab entries will execute the loader every five minutes, the
+tenants, builds, and binary distribution cleaners once a day at midnight, and
+the monitor every hour (all shifted by a few minutes in order not to clash
+with other jobs):
$ crontab -l
MAILTO=<brep-admin-email>
PATH=/usr/local/bin:/bin:/usr/bin
*/5 * * * * $HOME/install/bin/brep-load $HOME/config/loadtab
-0 0 * * * $HOME/install/bin/brep-clean tenants 240
-0 0 * * * $HOME/install/bin/brep-clean builds $HOME/config/buildtab
+1 0 * * * $HOME/install/bin/brep-clean tenants 240
+2 0 * * * $HOME/install/bin/brep-clean builds $HOME/config/buildtab
+3 0 * * * $HOME/install/bin/brep-upload-bindist-clean /var/bindist 2880
+4 * * * * $HOME/install/bin/brep-monitor --report-timeout 86400 --clean $HOME/config/brep-module.conf public
^D
Note that here we assume that bpkg (which is executed by brep-load) is in one
of the PATH's directories (usually /usr/local/bin).
-8.b Setup Periodic Loader and Cleaner Execution with systemd
+8.b Setup Periodic Loader, Cleaner, and Monitor Execution with systemd
In this version we will use the systemd user session to periodically run the
-loader and the cleaner as the brep user. If your installation doesn't use
-systemd, then a cron job would be a natural alternative (see above).
+loader, cleaner, and monitor as the brep user. If your installation doesn't
+use systemd, then a cron job would be a natural alternative (see above).
As the first step, make sure systemd user sessions support is working for the
brep user:
@@ -443,6 +431,7 @@ $ sudo loginctl enable-linger brep
$ mkdir -p .config/systemd/user
$ cp install/share/brep/etc/systemd/brep-load.* .config/systemd/user/
$ cp install/share/brep/etc/systemd/brep-clean.* .config/systemd/user/
+$ cp install/share/brep/etc/systemd/brep-monitor.* .config/systemd/user/
Start the service to make sure there are no issues:
@@ -452,16 +441,21 @@ $ journalctl
$ systemctl --user start brep-clean.service
$ journalctl
+$ systemctl --user start brep-monitor.service
+$ journalctl
+
Start the timers and monitor them to make sure they fire:
$ systemctl --user start brep-load.timer
$ systemctl --user start brep-clean.timer
+$ systemctl --user start brep-monitor.timer
$ journalctl -f
If everything looks good, enable the timer to be started at boot time:
$ systemctl --user enable brep-load.timer
$ systemctl --user enable brep-clean.timer
+$ systemctl --user enable brep-monitor.timer
9. Upgrade Procedure
@@ -483,18 +477,20 @@ $ cd brep
$ bpkg fetch
$ bpkg build brep
-If you are using a systemd-based setup, then stop and disable the loader and
-the cleaner:
+If you are using a systemd-based setup, then stop and disable the loader,
+cleaner, and monitor:
$ systemctl --user disable --now brep-load.timer
$ systemctl --user disable --now brep-clean.timer
+$ systemctl --user disable --now brep-monitor.timer
$ systemctl --user stop brep-load.service
$ systemctl --user stop brep-clean.service
+$ systemctl --user stop brep-monitor.service
If you are using a cron-based setup, then it is not worth it commenting out the
-job entries. If the new version of the loader or the cleaner gets executed
-before or during the migration, then it will fail and you will get an email
-with the diagnostics. Other than that, it should be harmless.
+job entries. If the new version of the brep utilities gets executed before or
+during the migration, then it will fail and you will get an email with the
+diagnostics. Other than that, it should be harmless.
Stop apache:
@@ -510,7 +506,7 @@ Review brep-module.conf changes that may need to be merged:
$ diff -u install/share/brep/etc/brep-module.conf config/brep-module.conf
-Migrate database schemes:
+Migrate database schemas:
$ install/bin/brep-migrate package
$ install/bin/brep-migrate build
@@ -521,17 +517,20 @@ is not possible), then one way to do it would be:
$ psql -d brep_package -c 'DROP OWNED BY brep'
$ psql -d brep_build -c 'DROP OWNED BY brep'
-If using systemd, then start and enable the loader and the cleaner:
+If using systemd, then start and enable the loader, cleaner, and monitor:
$ systemctl --user start brep-load.service
$ systemctl --user status brep-load.service
$ systemctl --user start brep-clean.service
$ systemctl --user status brep-clean.service
+$ systemctl --user start brep-monitor.service
+$ systemctl --user status brep-monitor.service
If everything looks good, enable periodic execution:
$ systemctl --user enable --now brep-load.timer
$ systemctl --user enable --now brep-clean.timer
+$ systemctl --user enable --now brep-monitor.timer
If using cron, then simply wait for the next run.