diff options
Diffstat (limited to 'INSTALL')
-rw-r--r-- | INSTALL | 229 |
1 files changed, 114 insertions, 115 deletions
@@ -11,8 +11,9 @@ can be omitted. 1. Create 'brep' User This user will be used to run the brep package database loader, build database -cleaner, and the database schemes migration utility. We will also use its home -directory to build and install the brep module, store its configuration, etc. +cleaner, monitor, and database schemas migration utility. We will also use its +home directory to build and install the brep module, store its configuration, +etc. Note: if the deployment machine employs SELinux, then this approach may require additional configuration steps (not shown) in order to allow Apache2 @@ -43,7 +44,7 @@ b) Install PostgreSQL 9.3 or above (including the contrib package containing the postgres_fdw extension) as well as Apache2 using your distribution's packages. Below are the names of these packages for some distributions: - Debian/Ubuntu: postgresql-server postgresql-contrib apache2 + Debian/Ubuntu: postgresql postgresql-contrib apache2 Fedora/RHEL: postgresql-server postgresql-contrib httpd FreeBSD: postgresqlXY-server postgresqlXY-contrib apache24 @@ -65,7 +66,7 @@ c) Install PostgreSQL and Apache2 development files. Specifically, we need files. Below are the names of their packages for some distributions: Debian/Ubuntu: libpq-dev libapr1-dev libapreq2-dev apache2-dev - Fedora/RHEL: posqtgresql-devel apr-devel libapreq2-devel httpd-devel + Fedora/RHEL: postgresql-devel apr-devel libapreq2-devel httpd-devel FreeBSD: postgresqlXY-client apr libapreq2 apache24 d) Unless you already have the build2 toolchain, install it by following @@ -76,24 +77,13 @@ d) Unless you already have the build2 toolchain, install it by following 3. Build and Install brep -Normally the only extra information that you need to provide on this step is -the location of the Apache2 headers (httpd.h, etc). Below are their locations -for some distributions: - -Debian/Ubuntu: /usr/include/apache2 -Fedora/RHEL: /usr/include/httpd -FreeBSD: /usr/local/include/apache24 - -You can also use the Apache2 apxs utility to obtain this information as shown -below. - $ mkdir brep $ cd brep -$ bpkg create \ - cc \ - config.cc.poptions="-I$(apxs -q includedir)" \ - config.bin.rpath=$HOME/install/lib \ +$ bpkg create \ + cc \ + config.cc.coptions=-O3 \ + config.bin.rpath=$HOME/install/lib \ config.install.root=$HOME/install $ bpkg add https://pkg.cppget.org/1/alpha @@ -103,6 +93,22 @@ $ bpkg install brep $ cd .. # Back to brep home. +Note that by default the location of the Apache2 headers (httpd.h, etc) is +detected automatically, using the Apache2 apxs utility. Below are their +locations for some distributions: + +Debian/Ubuntu: /usr/include/apache2 +Fedora/RHEL: /usr/include/httpd +FreeBSD: /usr/local/include/apache24 + +To disable this functionality and specify the location explicitly, you can +configure brep in the development mode and specify the respective preprocessor +option by, for example, adding the following configuration variables to the +above bpkg-build command: + +config.brep.develop=true +config.cc.poptions="-I..." + 4. Create PostgreSQL User and Databases @@ -157,7 +163,7 @@ CREATE EXTENSION postgres_fdw; CREATE SERVER package_server FOREIGN DATA WRAPPER postgres_fdw - OPTIONS (dbname 'brep_package', updatable 'false'); + OPTIONS (dbname 'brep_package', updatable 'true'); GRANT USAGE ON FOREIGN SERVER package_server to brep; @@ -165,6 +171,17 @@ CREATE USER MAPPING FOR PUBLIC SERVER package_server OPTIONS (user 'brep-build', password '-'); +Note that starting with PostgreSQL 15 only the database owner can create the +objects in the public schema by default. Thus, if the PostgreSQL version is 15 +or above, then all the privileges on this schema in the created databases need +to be granted explicitly by the postgres user to the brep user: + +\c brep_package +GRANT ALL PRIVILEGES ON SCHEMA public TO brep; + +\c brep_build +GRANT ALL PRIVILEGES ON SCHEMA public TO brep; + Exit psql (^D) The user brep-build is required (by the postgres_fdw extension) to login with @@ -194,7 +211,7 @@ CREATE EXTENSION citext; Exit psql (^D) -5. Create Database Schemes and Load Repositories +5. Create Database Schemas and Load Repositories $ mkdir config $ edit config/loadtab # Loader configuration, see brep-load(1). @@ -217,7 +234,21 @@ $ cp install/share/brep/etc/brep-module.conf config/ $ edit config/brep-module.conf # Adjust default values if required. To enable the build2 build bot controller functionality you will need to set -the build-config option in brep-module.conf. +the build-config option in brep-module.conf. To also enable the build +artifacts upload functionality you will need to specify the upload-data +directory for the desired upload types in brep-module.conf. For example, for +generated binary distribution packages it can be as follows: + +upload-data bindist=/home/brep/bindist-data + +Note that this directory must exist and have read, write, and execute +permissions granted to the www-data user. This, for example, can be achieved +with the following commands: + +$ mkdir /home/brep/bindist-data +$ setfacl -m g:www-data:rwx /home/brep/bindist-data + +For sample upload handler implementations see brep/handler/upload/. To enable the package submission functionality you will need to specify the submit-data and submit-temp directories in brep-module.conf. Note that these @@ -261,75 +292,8 @@ $ edit config/ci.xhtml # Add custom form fields, adjust CSS style, etc. For sample CI request handler implementations see brep/handler/ci/. Here we assume you have setup an appropriate Apache2 virtual server. Open the -corresponding Apache2 .conf file and add the following inside VirtualHost (you -can also find this fragment in install/share/brep/etc/brep-apache2.conf): - - # Load the brep module. - # - <IfModule !brep_module> - LoadModule brep_module /home/brep/install/libexec/brep/mod_brep.so - </IfModule> - - # Repository email. This email is used for the From: header in emails - # send by brep (for example, build failure notifications). - # - brep-email admin@example.org - - # Repository host. It specifies the scheme and the host address (but - # not the root path; see brep-root below) that will be used whenever - # brep needs to construct an absolute URL to one of its locations (for - # example, a link to a build log that is being send via email). - # - brep-host https://example.org - - # Repository root. This is the part of the URL between the host name - # and the start of the repository. For example, root value /pkg means - # the repository URL is http://example.org/pkg/. Specify / to use the - # web server root (e.g., http://example.org/). If using a different - # repository root, don't forget to also change Location and Alias - # directives below. - # - brep-root /pkg - - <Location "/pkg"> - SetHandler brep - - <IfModule dir_module> - DirectoryIndex disabled - DirectorySlash Off - </IfModule> - </Location> - - # Brep module configuration. If you prefer, you can paste the contents - # of this file here. However, you will need to prefix every option with - # 'brep-'. - # - brep-conf /home/brep/config/brep-module.conf - - # Static brep content (CSS files). - # - <IfModule !alias_module> - Error "mod_alias is not enabled" - </IfModule> - - # Note: trailing slashes are important! - # - Alias /pkg/@/ /home/brep/install/share/brep/www/ - - <Directory "/home/brep/install/share/brep/www"> - Require all granted - </Directory> - - # You can also serve the repository files from the repository root. - # For example: - # - # http://example.org/pkg/1/... -> /path/to/repo/1/... - # - #AliasMatch ^/pkg/(\d+)/(.+) /path/to/repo/$1/$2 - # - #<Directory "/path/to/repo"> - # Require all granted - #</Directory> +corresponding Apache2 .conf file and add the contents of +brep/etc/brep-apache2.conf into the <VirtualHost> section. The output content types of the brep module are application/xhtml+xml, text/manifest and text/plain. If you would like to make sure they get @@ -384,47 +348,71 @@ $ cd install/share/brep/www/ $ for i in *.scss; do sassc -s compressed $i `basename -s .scss $i`.css; done -8. Setup Periodic Loader and Cleaner Execution +8. Setup Periodic Loader, Cleaner, and Monitor Execution Initially this guide suggested using systemd user session support to run the -loader and the cleaner. However, the current state of user sessions has one -major drawback: they are not started/attached-to when logging in with su -l -(see Debian bug #813789 for details). This limitation makes them unusable in -our setup. If you still would like to use systemd to run the loader and the -cleaner, then you can set it up as a system-wide service which runs the -utilities as the brep user/group. Otherwise, a cron job is a natural choice. +loader, cleaner, and monitor. However, the current state of user sessions has +one major drawback: they are not started/attached-to when logging in with su +-l (see Debian bug #813789 for details). This limitation makes them unusable +in our setup. If you still would like to use systemd to run the utilities, +then you can set it up as a system-wide service which runs them as the brep +user/group. Otherwise, a cron job is a natural choice. Note that the builds cleaner execution is optional and is only required if the build2 build bot functionality is enabled (see the build bot documentation for -details). If it is disabled in you setup, then skip the cleaner-related -parts in the subsequent subsections. +details). If it is disabled in you setup, then skip the cleaner-related parts +in the subsequent subsections. + +If the build artifacts upload functionality is enabled in addition to the +build2 build bot functionality you most likely will want to additionally setup +the cleanup of the outdated build artifacts. For example, for binary +distribution package uploads handled by brep-upload-bindist the cleanup needs +to be performed by periodic execution of brep-upload-bindist-clean script. +Note that the directory where the uploads are saved to must exist and have +read, write, and execute permissions granted to the brep user. This, for +example, can be achieved with the following commands: + +# mkdir /var/bindist +# chown www-data:www-data /var/bindist +# setfacl -m u:brep:rwx /var/bindist +# setfacl -dm u:brep:rwx /var/bindist If the CI request functionality is enabled you most likely will want to additionally setup the tenants cleanup. +The monitor execution is also optional and currently only makes sense if the +build2 build bot functionality is enabled. Note that you may need to replace +the public toolchain name argument in the monitor utility command with a real +list of toolchain names (and optionally versions) used in the brep build +infrastructure. + -8.a Setup Periodic Loader and Cleaner Execution with cron +8.a Setup Periodic Loader, Cleaner, and Monitor Execution with cron -The following crontab entries will execute the loader every five minutes -and the tenants and builds cleaners once a day at midnight: +The following crontab entries will execute the loader every five minutes, the +tenants, builds, and binary distribution cleaners once a day at midnight, and +the monitor every hour (all shifted by a few minutes in order not to clash +with other jobs): $ crontab -l MAILTO=<brep-admin-email> PATH=/usr/local/bin:/bin:/usr/bin */5 * * * * $HOME/install/bin/brep-load $HOME/config/loadtab -0 0 * * * $HOME/install/bin/brep-clean tenants 240 -0 0 * * * $HOME/install/bin/brep-clean builds $HOME/config/buildtab +1 0 * * * $HOME/install/bin/brep-clean tenants 240 +2 0 * * * $HOME/install/bin/brep-clean builds $HOME/config/buildtab +3 0 * * * $HOME/install/bin/brep-upload-bindist-clean /var/bindist 2880 +4 * * * * $HOME/install/bin/brep-monitor --report-timeout 86400 --clean $HOME/config/brep-module.conf public ^D Note that here we assume that bpkg (which is executed by brep-load) is in one of the PATH's directories (usually /usr/local/bin). -8.b Setup Periodic Loader and Cleaner Execution with systemd +8.b Setup Periodic Loader, Cleaner, and Monitor Execution with systemd In this version we will use the systemd user session to periodically run the -loader and the cleaner as the brep user. If your installation doesn't use -systemd, then a cron job would be a natural alternative (see above). +loader, cleaner, and monitor as the brep user. If your installation doesn't +use systemd, then a cron job would be a natural alternative (see above). As the first step, make sure systemd user sessions support is working for the brep user: @@ -443,6 +431,7 @@ $ sudo loginctl enable-linger brep $ mkdir -p .config/systemd/user $ cp install/share/brep/etc/systemd/brep-load.* .config/systemd/user/ $ cp install/share/brep/etc/systemd/brep-clean.* .config/systemd/user/ +$ cp install/share/brep/etc/systemd/brep-monitor.* .config/systemd/user/ Start the service to make sure there are no issues: @@ -452,16 +441,21 @@ $ journalctl $ systemctl --user start brep-clean.service $ journalctl +$ systemctl --user start brep-monitor.service +$ journalctl + Start the timers and monitor them to make sure they fire: $ systemctl --user start brep-load.timer $ systemctl --user start brep-clean.timer +$ systemctl --user start brep-monitor.timer $ journalctl -f If everything looks good, enable the timer to be started at boot time: $ systemctl --user enable brep-load.timer $ systemctl --user enable brep-clean.timer +$ systemctl --user enable brep-monitor.timer 9. Upgrade Procedure @@ -483,18 +477,20 @@ $ cd brep $ bpkg fetch $ bpkg build brep -If you are using a systemd-based setup, then stop and disable the loader and -the cleaner: +If you are using a systemd-based setup, then stop and disable the loader, +cleaner, and monitor: $ systemctl --user disable --now brep-load.timer $ systemctl --user disable --now brep-clean.timer +$ systemctl --user disable --now brep-monitor.timer $ systemctl --user stop brep-load.service $ systemctl --user stop brep-clean.service +$ systemctl --user stop brep-monitor.service If you are using a cron-based setup, then it is not worth it commenting out the -job entries. If the new version of the loader or the cleaner gets executed -before or during the migration, then it will fail and you will get an email -with the diagnostics. Other than that, it should be harmless. +job entries. If the new version of the brep utilities gets executed before or +during the migration, then it will fail and you will get an email with the +diagnostics. Other than that, it should be harmless. Stop apache: @@ -510,7 +506,7 @@ Review brep-module.conf changes that may need to be merged: $ diff -u install/share/brep/etc/brep-module.conf config/brep-module.conf -Migrate database schemes: +Migrate database schemas: $ install/bin/brep-migrate package $ install/bin/brep-migrate build @@ -521,17 +517,20 @@ is not possible), then one way to do it would be: $ psql -d brep_package -c 'DROP OWNED BY brep' $ psql -d brep_build -c 'DROP OWNED BY brep' -If using systemd, then start and enable the loader and the cleaner: +If using systemd, then start and enable the loader, cleaner, and monitor: $ systemctl --user start brep-load.service $ systemctl --user status brep-load.service $ systemctl --user start brep-clean.service $ systemctl --user status brep-clean.service +$ systemctl --user start brep-monitor.service +$ systemctl --user status brep-monitor.service If everything looks good, enable periodic execution: $ systemctl --user enable --now brep-load.timer $ systemctl --user enable --now brep-clean.timer +$ systemctl --user enable --now brep-monitor.timer If using cron, then simply wait for the next run. |