diff options
| -rw-r--r-- | doc/packaging.cli | 479 |
1 files changed, 450 insertions, 29 deletions
diff --git a/doc/packaging.cli b/doc/packaging.cli index 96c2861..3ccd4b6 100644 --- a/doc/packaging.cli +++ b/doc/packaging.cli @@ -29,6 +29,9 @@ repository. For additional information, including documentation for individual \h1#intro|Introduction| +@@ Assume read through toolchain introduction and build system introduction. +Also, ideally, have some experience using \c{build2} in your own projects. + The aim of this guide is to ease the convertion of third-party C/C++ projects to the \c{build2} build system and publishing them to the \l{https://cppget.org cppget.org} package repository by codifying the best @@ -245,16 +248,34 @@ for <name>} line, where \c{<name>} is the project name.| \N|Since this is your personal repository, you can do the initial work directly in \c{master}/\c{main} or in a separate branch, it's up to you.| +As a running example, let's assume we want to package a library called \c{foo} +whose upstream repository is at \c{https://github.com/<upstream>/foo.git}. We +have created its package repository at +\c{https://github.com/<personal>/foo.git} (with the \c{build2 package for foo} +description) and can now clone it: + +\ +$ git clone https://github.com/<personal>/foo.git +\ -\h2#core-repo-init|Initialize package repository with \c{bdep new -t empty}| -From the repository root directory, run: +\h2#core-repo-init|Initialize package repository with \c{bdep new --type empty}| + +Change to the root directory of the package repository that you have clonned +on the previous step and run (continuing with our \c{foo} example): \ -bdep new -t empty +$ cd foo +$ bdep new --type empty +$ tree . +./ +├── .gitattributes +├── .gitignore +├── README.md +└── repositories.manifest \ -This command will create a number of files, including: +This command creates a number of files in the root of the repository: \dl| @@ -282,9 +303,9 @@ inside for details).|| Next add and commit these files: \ -git add . -git status -git commit -m \"Initialize repository\" +$ git add . +$ git status +$ git commit -m \"Initialize repository\" \ \N|In these guidelines we will be using the package repository setup that is @@ -328,10 +349,10 @@ purpose and all the relevant commands will be provided and explained, in case you are not familiar with this \c{git} mechanism.| Given the upstream repository URL, to add it as a submodule, run the following -command from the package repository root: +command from the package repository root (continuing with our \c{foo} example): \ -git submodule add https://github.com/.../<project>.git upstream +$ git submodule add https://github.com/<upstream>/foo.git upstream \ \N|You should prefer \c{https://} over \c{git://} for the upstream repository @@ -348,17 +369,17 @@ to update the \c{upstream} submodule to point to this release commit, run the following command: \ -cd upstream -git checkout vX.Y.Z -cd .. +$ cd upstream +$ git checkout vX.Y.Z +$ cd .. \ Then add and commit these changes: \ -git add . -git status -git commit -m \"Add upstream submodule\" +$ git add . +$ git status +$ git commit -m \"Add upstream submodule\" \ Now we have all the upstream source code for the release that we are @@ -543,15 +564,405 @@ to use, \l{https://build2.org/community.xhtml#help get in touch} to discuss the alternatives. It can be quite painful to change these things after you have completed the remaining packaging steps. +Continuing with our \c{foo} example, we will follow the recommendation and +call the library package \c{libfoo}. -@@ Where do we overlay the source code? -====================================================================== +\h2#core-package-struct|Decide on the package source code layout| + +Another aspect we need to decide on is the source code layout inside the +package. Here we want to stay as close to upstream layout as possible unless +there are valid reasons to deviate. This has the best chance of giving us a +build without any compile errors since the header inclusion in the project can +be sensitive to this layout. This also makes it easier for upstream to adopt +the \c{build2} build. + +Sometimes, however, there are good reasons for deviating from upstream, +especially in cases where upstream is clearly following bad practices, for +example including generically-named public headers without the library +subdirectory. If you do decide to change the layout, it's usually less +disruptive (to the build) to rearrange things at the outer levels than at the +inner. For example, it should normally be possible to move/rename the +top-level \c{tests/} directory or to place the library source directory into a +subdirectory. + +Our overall plan for the package is to create the initial layout and +\c{buildfile} templates automatically using \l{bdep-new(1)} in the +\c{--package} mode, then tweak \c{buildfile}s if necessary, and finally +\"fill\" the package with upstream source code using symlinks. + +The main rationale for using \l{bdep-new(1)} instead of doing everything by +hand is that there are many nuances in getting the build right and +auto-generated \c{buildfile}s had years of refinement and fine-tuning. The +familiar structure also makes it easier for others to understand your build, +for example while reviewing your package submission. + +The \l{bdep-new(1)} command supports a wide variety of +\l{bdep-new.xhtml#src-layout source layouts}. While it may take a bit of time +to understand the customization points necessary to achieve the desired layout +for your first package, this will pay off in spades when you work on +converting subsequent packages. + +And so the focus of the following steps is to iteratively discover the +\l{bdep-new(1)} command line that best approximates the upstream layout. The +recommended procedure is as follows: + +\ol| + +\li|Study the upstream source layout and existing build system.| + +\li|Craft and execute the \l{bdep-new(1)} command line necessary to achieve +the upstream layout.| + +\li|Study the auto-generated \c{buildfile}s for things that don't fit and need +to change. But don't rush to start manually editing the result. First get an +overview of the required changes and then check if it's possible to achieve +these changes automatically using one of \l{bdep-new(1)} sub-options. If +that's the case, delete the package subdirectory, and restart from step #2.|| + +This and the following two sections discuss each of these steps in more detail +and also look at some examples. + +The first step above is to study the upstream project in order to understand +where the various parts are (headers, sources, etc.) and how they are built. +Things that can help here include: + +\ul| + +\li|Read through the existing build system definitions.| + +\li|Try to build the project using the existing build system.| + +\li|Try to install the project using the existing build system.| + +\li|Look into the Debian package contents to see if there are any differences + with regards to the installation locations.|| + +For libraries, the first key pieces of information we need to find is how the +public headers are included and where they are installed. The two common +\i{good} practices is to either include the public headers with a library name +as a subdirectory, for example, \c{#include\ <foo/util.h>}, or to include the +library name into each public header name, for example, \c{#include\ +<foo_util.h>} or \c{#include\ <foo.h>} (in the last example the header name is +the library name itself, which is also fairly common). Unfortunately, there is +also a fairly common \i{bad} practice: having generically named headers (such +as \c{util.h}) included without the library subdirectory. + +\N|The reason this is a bad practice is that libraries that have such headers +cannot coexist, neither in the same build nor when installed. See +\l{intro#proj-struct Canonical Project Structure} for background and details. +See \l{#howto-bad-inclusion-practice How do I deal with bad header inclusion +practice} if you encounter such a case.| + +Where should we look to get this information? While the library source files +sound like a natural place, oftentimes they include own headers with the +\c{\"\"} style inclusion, either because the headers are in the same directory +or because the library build arranges for them to be found this way with +additional header search paths. As a result, a better place to look could be +library's examples and/or tests. Some libraries also describe which headers +they provide and how to include them in their documentation. + +The way public headers are included normally determines where they are +installed. If they are included with a subdirectory, then they are normally +installed into the same subdirectory in, say, \c{/usr/include/}. Continuing +with the above example, a header that is included as \c{<foo/util.h>} would +normally be installed as \c{/usr/include/foo/util.h}. On the other hand, if +the library name is part of the header name, then the headers are usually (but +not always) installed directly into, say, \c{/usr/include/}, for example as +\c{/usr/include/foo_util.h}. + +\N|While these are the commonly used installation schemes, there are +deviations. In particular, in both cases upstream may choose to add an +additional subdirectory when installing (so the above examples we instead +end up with, say, \c{/usr/include/sub/foo/util.h} and +\c{/usr/include/sub/foo_util.h}). See \l{#howto-extra-header-install-subdir +How do I handle extra header installation subdirectory} if you encounter such +a case.| + +The inclusion scheme would normally be recreated in the upstream source code +layout. In particular, if upstream includes public headers with a +subdirectory, then this subdirectory would normally also be present in the +upstream layout so that such a header can be included form the upstream +codebase directly. As an example, let's say we determined that public headers +of \c{libfoo} are included with the \c{foo/} subdirectory, such as +\c{<foo/util.hpp>}. One of the typical upstream layouts for such a library +would look like this: + +\ +$ tree upstream/ +upstream/ +├── include/ +│ └── foo/ +│ └── util.hpp +└── src/ + ├── priv.hpp + └── util.cpp +\ + +Notice how the \c{util.hpp} header is in the \c{foo/} subdirectory rather +than in \c{include/} directly. + +The second key pieces of information we need to find is whether and, if so, +how the public headers and sources are split. For instance, in the above +example, we can see that public headers go into \c{include/} while sources and +private headers go into \c{src/}. But they could also be combined in the same +directory, for example, as in the following layout: + +\ +upstream/ +└── foo/ + ├── priv.hpp + ├── util.cpp + └── util.hpp +\ + +\N|In multi-package projects, for example, those that provide both a library and +an executable, you would also want to understand how the sources are split +between the packages.| + +If the headers and sources are split into different directories, then the +source directory may or may not have the include subdirectory, similar to the +header directory. In the above split layout the \c{src/} directory doesn't +contain the include subdirectory (\c{foo/}) while the following layout does: + +\ +upstream/ +├── include/ +│ └── foo/ +│ └── util.hpp +└── src/ + └── foo/ + ├── priv.hpp + └── util.cpp +\ + +With the understanding of these key properties of upstream layout you should +be in a good position to start crafting the \l{bdep-new(1)} command line that +recreates it. + + +\h2#core-package-craft-cmd|Craft \c{bdep new} command line to create package| + +The recommened procedure for this step is to read through the \c{bdep-new}'s +\l{bdep-new.xhtml#src-layout SOURCE LAYOUT} section (which contains a large +number of examples) while experimenting with various options in an attempt to +create the desired layout. If the layout you've got isn't quite right yet, +simply remove the package directory along with the \c{packages.manifest} file +and try again. + +Let's illustrate this approach on the original example of the split layout: + +\ +upstream/ +├── include/ +│ └── foo/ +│ └── util.hpp +└── src/ + ├── priv.hpp + └── util.cpp +\ + +We know it's split, so let's start with that and see what we get. Remember, +our \c{foo} package repository that we have clonned and initialized earlier +looks like this: +\ +$ tree foo/ +foo/ +├── .gitattributes +├── .gitignore +├── README.md +└── repositories.manifest +\ + +Now we create the \c{libfoo} package inside: + +\ +$ cd foo +$ bdep new --package --lang c++ --type lib,split libfoo +$ tree libfoo/ +libfoo/ +├── include/ +│ └── libfoo/ +│ └── foo.hxx +└── src/ + └── libfoo/ + └── foo.cxx +\ + +The outer structure looks right, but inside \c{include/} and \c{src/} things +are a bit off. Specifically, the include subdirectory should be \c{foo/}, not +\c{libfoo/}, there shouldn't be one inside \c{src/}, and the file extensions +don't match upstream. All this can be easily tweaked, however: + +\ +$ rm -r libfoo/ packages.manifest +$ bdep new --package \ + --lang c++,cpp \ + --type lib,split,subdir=foo,no-subdir-source \ + libfoo +$ tree libfoo/ +libfoo/ +├── include/ +│ └── foo/ +│ └── foo.hpp +└── src/ + └── foo.cpp +\ + +The other \c{bdep-new} sub-options (see the \l{bdep-new(1)} man page for the +complete list) that you will likely want to use when packaging a third-party +project include: + +\dl| + +\li|\n\c{no-version} + +Omit the auto-generated version header. Usually upstream will provided its own +equivalent to this functionality. + +\N|Note that even if upstream doesn't provide any version information, it's +not a good idea to try to rectify this by providing your own version header +since upstream may add it in a future version and you may end up with a +conflict. Instead, work with the project maintainer to rectify this in +upstream.|| + +\li|\n\c{no-symexport}\n\c{auto-symexport} + +The \c{no-symexport} sub-option suppresses the generation of the DLL symbol +exporting header. This is an appropriate option if upstream provides its +own symbol exporting arrangements. + +The \c{auto-symexport} sub-option enables automatic DLL symbol exporting +support (see \l{b##cc-auto-symexport Automatic DLL Symbol Exporting} for +background). This is an appropriate option if upstream relies on similar +support in the existing build system. It is also recommended that you give +this functionality a try even if upstream does not support building +shared libraries on Windows.| + +\li|\n\c{binless} + +Create a header-only library template. See \l{#dont-header-only Don't make +library header-only if it can be compiled} and +\l{https://github.com/build2/HOWTO/blob/master/entries/make-header-only-library.md +How do I make a header-only C/C++ library?}|| + +Continuing with our \c{libfoo} example, assuming upstream provides own symbol +exporting, the final \c{bdep-new} command line would be: + +\ +$ bdep new --package \ + --lang c++,cpp \ + --type lib,split,subdir=foo,no-subdir-source,no-version,no-symexport \ + libfoo +\ + +Let's also get a more complete view of what it generates: + +\ +$ tree libfoo/ +libfoo/ +├── build/ +│ └── ... +├── include/ +│ └── foo/ +│ ├── buildfile +│ └── foo.hpp +├── src/ +│ ├── buildfile +│ └── foo.cpp +├── tests/ +│ ├── build/ +│ │ └── ... +│ ├── basics/ +│ │ ├── buildfile +│ │ └── driver.cpp +│ └── buildfile +├── buildfile +├── manifest +└── README.md +\ + +\h2#core-package-review|Review and test auto-genetated \c{buildfile} templates| + +Once the overall layout looks right, the next step is to take a closer look at +the generated \c{buildfile}s to make sure that overall they match the upstrem +build. Of particular interest are the header and source directory +\c{buildfile}s (\c{libfoo/include/buildifle} and \c{libfoo/src/buildifle} in +the above listing) which define how the library is built and installed. + +Here we are focusing on the macro-level differences that are easier to change +by tweaking the \c{bdep-new} command line rather than manually. For example, +if we look at the generated source directory \c{buildfile} and realize it +builds a \"binful\" library (that is, a library that includes source files and +therefore produces library binaries) while the upsteam library is header-only, +it is much easier to fix this by re-running \c{bdep-new} with the \c{binless} +sub-option than by changing the \c{buildfile} manually. + +\N|Don't be tempted to start making manual changes at this stage even if you +cannot see anything else that can be fixed with a \c{bdep-new} re-run. This +is still a dry-run and we will recreate the package one more time in the +following section before starting manual adjustments.| + +Besides examining the generated \c{buildfile}s, it's also a good idea to +build, test, and install the generated package to make sure everything ends up +where you expected and matches upstream where necessary. In particular, make +sure public headers are installed into the same location as upstream. + +\N|The \c{bdep-new}-generated library is a simple \"Hello, World!\" example +that can nevertheless be built, tested, and installed. The idea here is to +verify it matches upstream using the generated source files before replacing +them with the upstream source file symlinks.| + +Note that at this stage its easiest to build, test, and install in source +directly sidestepping the \c{bdep} initialization of the package (which you +would have to de-initalize before you can re-run \c{bdep-new}). Continue +with the above example, the recommended sequence of commands would be: + +\ +$ cd libfoo +$ b update +$ b test +$ b install config.install.root=/tmp/install +$ b clean +\ +Let's also briefly discuss other subdirectories and files found in the +\c{bdep-new}-generated \c{libfoo} package. +The \c{build/} subdirectory is the standard \c{build2} place for project-wide +build system information (see \l{b#intro-proj-struct Project Structure} for +details). We will look close at its contents in the following steps. +In the root directory of our package we find the root \c{buildfile} and +package \c{manifest}. We will be tweaking both in the following steps. There +is also \c{README.md} which we will replace with the upstream symlink. +The \c{tests/} subdirectory is the standard \c{build2} tests subproject (see +\l{b#intro-operations-test Testing} for details). While you can suppress its +generation with the \c{no-tests} \c{bdep-new} sub-option, we recommend that +you keep it and use it as a starting point for porting upstream tests or, if +upstream doesn't provide any, for a basic \"smoke test\" (@@ ref HOWTO). + +\N|You can easily add/remove/rename this \c{tests/} subproject. The only place +where it is mentioned explicitly and where you will need to make changes is +the root \c{buildfile}. In pacticular, if upstream provides examples that you +wish to port, it is recommended that you use a copy of the generated +\c{tests/} subproject as a starting point (not forgeting to add the +corresponding entry in the root \c{buildfile}).| + +@@ We can actually do the final creation here (with symlinking, etc). + +@@ Adding additional subdirectories to avoid symlinking individual files. + Ideally would want bdep-new mode. + +@@ Pre-symlink README, LICENSE, PACKAGE-README.md? +@@ Auto-detect the license? +@@ Where do we overlay the source code? + +@@ The Don't write buildfiles by hand entry is now duplicate/redundant. + +====================================================================== \h1#dont-do|What Not to Do| @@ -617,18 +1028,8 @@ time for manual customizations. These would normally include: \h#dont-change-upstream|Don't change upstream source code layout| It's a good idea to stay as close to the upstream's source code layout as -possible. This has the best chance of giving us a build without any compile -errors since the header inclusion in the project can be sensitive to this -layout. This also makes it easier for upstream to adopt the \c{build2} -build. - -Sometimes, however, there are good reasons for deviating from upstream, -especially in cases where upstream is clearly following bad practices, for -example installing generically-named headers without a library prefix. If you -do decide to change the layout, it's usually less disruptive (to the build) to -rearrange things at the outer levels than at the inner. For example, it should -normally be possible to move/rename the top-level \c{tests/} directory or to -place the library source directory into a subdirectory. +possible. For background and rationale, see \l{#core-package-struct Decide on +the package source code layout}. \h#dont-forget-update-manifest|Don't forget to update \c{manifest} values| @@ -825,6 +1226,26 @@ $ bdep new -t lib,prefix=libigl-core,no-subdir,no-version libigl-core \h1#howto|Packaging HOWTO| +@@ howto make smoke test (and fix ref) + +\h#howto-bad-inclusion-practice|How do I deal with bad header inclusion practice| + +This sections explains how to deal with libraries that include their public, +generically-named headers without a library name as directory prefix. Such +libraries cannot coexist, neither in the same build nor when installed. For +background and details, see \l{intro#proj-struct Canonical Project Structure}. + +@@ TODO + + +\h#howto-extra-header-install-subdir|How do I handle extra header installation subdirectory| + +This sections explains how to handle an additional header installation +subdirectory. + +@@ TODO + + \h1#faq|Packaging FAQ| \h#faq-publish-stage|Where to publish if package requires staged toolchain?| |