path: root/etc/private/README

This directory contains a virtual machine (VM) with a build2 repository web
interface (brep) installation for a private package repository plus a number
of scripts and configuration files for running this machine as a systemd
service.

A brep installation consists of a web server (Apache2), a database server
(PostgreSQL), and a number of auxiliary processes (repository loader,
submission handler, etc). While all this can be installed and configured
manually (as described in brep/INSTALL), this VM makes it possible to quickly
setup a private repository.

Note that the configuration offered by this VM is only suitable for a
private/trusted environment, normally either for personal use (host-local)
or for use within an organization's private network. Specifically:

  - The repository is accessed via HTTP.

  - The repository is not signed.

  - Submitted packages are published directly and without ownership
    authentication.

  - The VM does not perform auto-update (since it does not assume the presence
    of an Internet connection) and therefore may not have the latest security
    updates.

The below setup instructions are for host machines that run systemd-based
Linux distributions. Note, however, that it should be possible to use other
distributions or operating systems provided they are capable of running
QEMU/KVM virtual machines. The following utilities are expected to be
available on the host machine:

  - systemd >= 229    (systemd --version)
  - bash    >= 4.3    (base --version)
  - qemu    >= 2.5.0  (qemu-system-x86_64 --version)
  - screen, socat     (screen --version, socat -V)

Consult your distribution's package manager if any of these utilities are
missing.

The host machine is also expected to have KVM virtualization support as
well as at least 1G or RAM (2G recommended) and at least 5G of disk space
(4G for VM image and the rest for package storage) that can be dedicated
to the VM.

Note: commands that start with the `#` prompt must be executed as root.


1. Create the brep user and group.

In this setup, the VM image, scripts, etc., as well as the repository packages
are all kept in the home directory of the special brep user. In particular,
the packages are stored on the host machine (as opposed to inside the VM
image) and are shared with the VM (using the virtio-9p filesystem). As a
result, if necessary, you can manipulate the package repository from the host
machine (but see Step 6 below for the rules). This setup also makes it easier
to upgrade VM images by simply replacing the old one with a new.

However, to make this arrangement work reliably, the brep user/group IDs on
the host machine must match those inside the VM. As a result, we create the
brep user/group with specific IDs:

# groupadd --gid 63700 brep
# useradd --uid 63700 --gid 63700 --create-home brep
# usermod --lock brep # disable password login (if desired)

Additionally, if your distribution requires users that are allowed to use KVM
to belong to a special group (normally kvm), then add the brep user to this
group:

# usermod -G kvm brep

If unsure whether this is required, skip this step and come back to it if you
get 'KVM: permission denied' error on Step 4.


2. Download and unpack the VM archive into the brep user's home directory.

# su - brep
$ pwd
/home/brep

$ curl -fO https://download.build2.org/X.Y.Z/linux-debian-N-brep-X.Y.Z.tar.xz
$ sha256sum -b linux-debian-N-brep-X.Y.Z.tar.xz

Verify the checksum matches the one from https://build2.org/download.xhtml

$ tar -xf linux-debian-N-brep-X.Y.Z.tar.xz --strip-components=1
$ ls
bin/ etc/ vm/ vm-brep@.service


3. Configure networking for the VM.

This setup expects the VM to use bridged networking with a persistent tap
interface. This allows for a wide variety of configurations ranging between
host-local (private bridge without routing), subnet (private bridge with
routing/NAT), and local network (public bridge over host's Ethernet
adapter). In particular, the last configuration would make the repository
accessible from other machines on the same local network.

The exact steps on how to setup bridged networking and create a persistent tap
interface depend the network manager used so consult your distribution's
documentation for details. The guide found in etc/systemd-networkd/README
shows how to setup the local network configuration mentioned above using the
systemd-networkd network manager available on most systemd-based
distributions.

In the rest of this guide we assume that tap interface called tap0 is
appropriately configured and is owned by user/group brep.


4. Generate a MAC address and start the VM for testing.

The recommended way to obtain a MAC address for the VM is to generate it
based on the host's Ethernet adapter address (see inside vm-gen-macaddress
for details):

$ ~/bin/vm-gen-macaddress xx:yy:yy:yy:yy:yy 0

Where xx:yy:yy:yy:yy:yy is the MAC address of the host's Ethernet adapter that
can be viewed with the following command:

# ip link show

The address printed by vm-gen-macaddress will be in the 02:yy:yy:yy:yy:yy
form.

If you are using a local network configuration, then now is a good time to
assign the VM its IP address and hostname. If you need to submit a request to
your network administrator, then the following text could serve as a template:

"I would like to run a VM on the <host> machine that needs to have its own IP
 address and domain name (configured via DHCP). It will have fixed MAC address
 <mac> (which was derived from <host>'s physical Ethernet address; but you are
 welcome to assign a different MAC address if desired). The DHCP client ID is
 the same as the MAC address. I would like this machine to have the <vm> name
 if possible.

 This will be a QEMU/KVM virtual machine running as a systemd service. It will
 use bridged networking with a tap interface."

Where:

  <host>  host machine name, for example, myserver.lan (run hostname -f)
  <mac>   the generated mac address (02:yy:yy:yy:yy:yy)
  <vm>    VM machine name, for example, mybrep.lan

Note that the VM is configured to receive its hostname from DHCP server (the
DHCP protocol option 12, "Host Name"). Failed that, the repository URL will
use the IP address.

Next, create the package repository directory and start the VM for testing
(replace 02:yy:yy:yy:yy:yy with the actual MAC address):

$ mkdir -p state/bpkg
$ ~/bin/vm-start --stdio --tap tap0 --mac 02:yy:yy:yy:yy:yy vm/brep.img

After booting, you will be presented with a login. Login as root with password
123 (VM command prompts are shown indented with two spaces). Then verify IP
address, hostname, and the network functionality:

  # ip addr show
  # hostname -f
  # ping example.org

If everything appears correct, visit the repository web page with a browser
(for example, http://mybrep.lan). Check the About page to see the repository
URL.

Try to submit a package (for example, from your development machine):

$ bdep new hello
$ cd hello
$ git add . && git commit -m test
$ bdep init -C @test cc
$ bdep publish --control=none --repository http://mybrep.lan --force=snapshot

Visit the repository web page and confirm the package is there. Then try to
build the submitted package from the repository:

$ bpkg create -d test
$ bpkg build -d test hello@http://mybrep.lan/1

If everything is working fine, shut the VM down:

  # shutdown -h now


5. Setup the VM to run as a systemd service.

To start the VM as a systemd service on host boot, perform the following
steps.

First, create the VM configuration file (replace 02:yy:yy:yy:yy:yy with the
actual MAC address)::

$ cat <<EOF >vm/brep.conf
RAM=2G
CPU=1
TAP=tap0
MAC=02:yy:yy:yy:yy:yy
EOF

Then configure the systemd service:

# cp ~brep/vm-brep@.service /etc/systemd/system/
# chmod 644 /etc/systemd/system/vm-brep@.service
# systemctl status vm-brep@brep
# systemctl start vm-brep@brep
# systemctl status vm-brep@brep

If the VM fails to start, study the log for a possible cause:

# journalctl -u vm-brep@brep

If the VM has started successfully, perform the same verifications as on Step
4 above.

To login to the VM running as a systemd service (for example, to verify IP
and hostname) use the vm-login script (which uses screen to connect to the
VM's console):

$ ~/bin/vm-login ~/brep-con.sock

To close the login, press Ctrl-a k (or Ctrl-a a k if already running inside
screen).

If everything functions correctly, verify the VM can be stopped:

# systemctl stop vm-brep@brep
# systemctl status vm-brep@brep

Finally, if desired, enable the VM service to start on boot:

# systemctl enable vm-brep@brep

After this you may also want to reboot the host machine and confirm the VM is
started on boot.


6. Manage the repository state.

While you can submit packages to the repository using bdep-publish(1), you can
also add them manually. Also, packages can only be removed manually.

The repository packages and metadata are stored in the ~brep/state/bpkg/pkg/
directory. If you need to make any modifications in this directory, there are
two rules that you must follow:

  1. You must stop the VM before making any modifications.

  2. You must make any modification only as user brep.

After performing the modifications, remove the 1/packages.manifest file to
trigger the repository metadata regeneration on the next VM startup. You can
also customize the repositories.manifest file in the same way. For example,
you could add cppget.org as a prerequisite repository for your private
repository.

Putting it all together, the steps could look like this:

# systemctl stop vm-brep@brep
# su - brep
$ cd state/bpkg/pkg
$ <make your changes here>
$ rm 1/packages.manifest
$ exit
# systemctl start vm-brep@brep

Note also that it's easy to break the repository with manual modifications.
For example, you may add a package that has an unmet dependency or remove a
package that still has some dependents. In this case, the repository web
interface will be unavailable. In this case, you can login into the VM to
investigate:

$ ~/bin/vm-login ~/brep-con.sock

  # systemctl status brep-startup
  # journalctl -u brep-startup


7. Upgrade the VM.

To upgrade to the new version of the VM, first download and unpack the new
VM archive similar to Step 2:

$ curl -fO https://download.build2.org/X.Y.Z/linux-debian-N-brep-X.Y.Z.tar.xz
$ sha256sum -b linux-debian-N-brep-X.Y.Z.tar.xz

Verify the checksum matches the one from https://build2.org/download.xhtml

$ tar -xf linux-debian-N-brep-X.Y.Z.tar.xz

Next read the linux-debian-N-brep-X.Y.Z/NEWS file for changes and potential
backwards compatibility issues. Unless instructed otherwise by the NEWS file,
the upgrade procedure is as follows:

# systemctl stop vm-brep@brep

$ cd
$ mkdir bak
$ mv -t bak/ bin etc vm/brep.img vm-brep@.service README NEWS
$ mv -t ./ linux-debian-N-brep-X.Y.Z/*
$ rm state/bpkg/pkg/1/packages.manifest

# cp ~brep/vm-brep@.service /etc/systemd/system/
# chmod 644 /etc/systemd/system/vm-brep@.service
# systemctl daemon-reload
# systemctl start vm-brep@brep
# systemctl status vm-brep@brep

If the VM has started successfully, perform the same verifications as on Step
4 above. If everything is functioning correctly, you can remove the backup
files:

$ rm -r bak

If there are any issues, investigate as on Step 6.