Factor repository types documentation into bpkg-repository-types(1)

7 files changed, 226 insertions, 183 deletions

diff --git a/bpkg/.gitignore b/bpkg/.gitignore
index f72601d..007b999 100644
--- a/bpkg/.gitignore
+++ b/bpkg/.gitignore
@@ -1,6 +1,6 @@
 bpkg
 *-options.?xx
 *-odb.?xx
-repository-signing.hxx
-repository-signing.cxx
+repository-signing.?xx
+repository-types.?xx
 version.hxx
diff --git a/bpkg/bpkg.cli b/bpkg/bpkg.cli
index bf36b74..f0e0928 100644
--- a/bpkg/bpkg.cli
+++ b/bpkg/bpkg.cli
@@ -51,15 +51,17 @@ namespace bpkg
      configuration is an amalgamation that contains packages as subprojects
      (see \l{bpkg-cfg-create(1)} for details).
 
-     A \i{bpkg package} is an archive or directory that contains a \cb{build2}
-     project plus the package \cb{manifest} file. \cb{bpkg} can either use
-     package archives/directories directly from the filesystem or it can fetch
-     archives from repositories.
+     A \i{bpkg package} is an archive or directory (potentially in a version
+     control system) that contains a \cb{build2} project plus the package
+     \cb{manifest} file. \cb{bpkg} can either use package archives/directories
+     directly from the filesystem or it can fetch them from repositories.
 
      A \i{bpkg repository} is a collection of packages as well as prerequisite
      and complement repositories. \i{Archive}, \i{directory} and \i{version
      control}-based repositories are supported. A repository is identified by
-     its location which can be a local filesystem path or a URL.
+     its location which can be a local filesystem path or a URL. See
+     \l{bpkg-repository-types(1)} for details on their structure and URL
+     format.
 
      If the same version of a package is available from multiple repositories,
      then they are assumed to contain identical package content. In such cases
@@ -284,6 +286,11 @@ namespace bpkg
       "\l{bpkg-common-options(1)} \- details on common options"
     }
 
+    bool repository-types
+    {
+      "\l{bpkg-repository-types(1)} \- repository types, structure, and URLs"
+    }
+
     bool repository-signing
     {
       "\l{bpkg-repository-signing(1)} \- how to sign repository"
diff --git a/bpkg/buildfile b/bpkg/buildfile
index 43bfe53..6f24025 100644
--- a/bpkg/buildfile
+++ b/bpkg/buildfile
@@ -42,7 +42,7 @@ rep-info-options           \
 rep-list-options           \
 rep-remove-options
 
-help_topics = repository-signing
+help_topics = repository-signing repository-types
 
 exe{bpkg}:                                                                 \
   {hxx ixx txx cxx}{** -{$options_topics} -{$help_topics} -*-odb -version} \
@@ -112,6 +112,7 @@ if $cli.configured
   # Help topics.
   #
   cli.cxx{repository-signing}: cli{repository-signing}
+  cli.cxx{repository-types}: cli{repository-types}
 
   # Option length must be the same to get commands/topics/options aligned.
   #
@@ -132,7 +133,8 @@ bpkg::pkg_build_pkg_options=exclude-base --generate-modifier
 
   # Avoid generating CLI runtime and empty inline file for help topics.
   #
-  cli.cxx{repository-signing}: cli.options += --suppress-cli --suppress-inline
+  cli.cxx{repository-signing repository-types}: cli.options += --suppress-cli \
+--suppress-inline
 
   # Include the generated cli files into the distribution and don't remove
   # them when cleaning in src (so that clean results in a state identical to
diff --git a/bpkg/help.cxx b/bpkg/help.cxx
index c66fd66..4b642e8 100644
--- a/bpkg/help.cxx
+++ b/bpkg/help.cxx
@@ -12,6 +12,7 @@
 // Help topics.
 //
 #include <bpkg/repository-signing.hxx>
+#include <bpkg/repository-types.hxx>
 
 using namespace std;
 using namespace butl;
@@ -32,6 +33,8 @@ namespace bpkg
         usage = &print_bpkg_common_options_long_usage;
       else if (t == "repository-signing")
         usage = &print_bpkg_repository_signing_usage;
+      else if (t == "repository-types")
+        usage = &print_bpkg_repository_types_usage;
       else
         fail << "unknown bpkg command/help topic '" << t << "'" <<
           info << "run 'bpkg help' for more information";
diff --git a/bpkg/rep-add.cli b/bpkg/rep-add.cli
index 126d4e1..1e60054 100644
--- a/bpkg/rep-add.cli
+++ b/bpkg/rep-add.cli
@@ -30,179 +30,20 @@ namespace bpkg
      the newly added repository. For that, use the \l{bpkg-rep-fetch(1)}
      command, normally, after adding all the repositories you wish to use.
 
-     Currently three types of repositories are supported: \cb{pkg}, \cb{dir},
-     and \cb{git}. Normally the repository type can be automatically guessed
-     by examining its URL (for example, the presence of the \cb{.git}
-     extension) or, in case of a local repository, its content (for example,
-     the presence of the \cb{.git/} subdirectory). Without any identifying
-     information the \cb{pkg} type is assumed unless explicitly specified with
-     the \cb{--type} option. Note, however, that the \cb{dir} repository type
-     is never guessed since it is not easily distinguishable from local
-     \cb{pkg} and \cb{git} repositories.
-
-     A \cb{pkg} repository is \i{archive}-based. That is, it contains a
-     collection of various packages/versions as archive files. For more
-     information on the structure of \cb{pkg} repositories refer to the
-     \l{bpkg \cb{bpkg} manual}.
-
-     A \cb{dir} repository is \i{directory}-based. That is, it contains a
-     collection of various packages as directories (but only a single version
-     per package can be present is such a repository). The \cb{dir} repository
-     location can be a local directory path or a \cb{file://} URL.
-
-     A \cb{dir} repository is expected to contain either the \cb{manifest} or
-     \cb{packages.manifest} file in the root directory of the repository. If
-     it only contains \cb{manifest}, then it is assumed to be a simple,
-     single-package repository with the \cb{manifest} file being its package
-     manifest. Otherwise, the \cb{packages.manifest} file should list the
-     available packages as described in \l{bpkg#manifest-package-list-dir
-     Package List Manifest for \cb{dir} Repositories}.
-
-     A \cb{dir} repository may also contain the \cb{repositories.manifest}
-     file in the root directory of the repository. This file can be used to
-     describe the repository itself as well as specify its prerequisite and
-     complement repositories. See \l{bpkg#manifest-repository-list Repository
-     List Manifest} for details on the format and semantics of this file.
-
-     A \cb{git} repository is \i{version control}-based. That is, it normally
-     contains multiple versions of the same package (but can also contain
-     several packages in the same repository).
-
-     Theoretically, a \cb{git} repository may contain as many package versions
-     as there are commits. Practically, however, we are normally only
-     interested in a small subset of them while fetching and processing the
-     necessary information for all of them could be prohibitively expensive.
-     As a result, by default, only advertised tags in the \cb{refs/tags/v*}
-     form where the part after \cb{v} is also a valid \l{b#module-version
-     standard version} are considered to be sources of useful package
-     versions. These commits normally correspond to releases and are called
-     the default set.
-
-     Instead of the default set, it is also possible to provide a custom set
-     of available versions by specifying one or more commit ids and/or
-     references and/or reference patterns in the repository URL fragment (see
-     \cb{git-ls-remote(1)} for details on advertised references). For example:
-
-     \
-     https://example.com/foo.git#v1.2.3
-     https://example.com/foo.git#master
-     https://example.com/foo.git#af234f56
-     https://example.com/foo.git#tags/releases/*
-     https://example.com/foo.git#HEAD,tags/v1.*.*,heads/feature-*
-     \
-
-     Furthermore, it is possible to expand (or narrow down) the default set
-     using the special \cb{##} fragment notation. For example:
-
-     \
-     https://example.com/foo.git##HEAD     - default set plus HEAD
-     https://example.com/foo.git##heads/*  - default set plus branches
-     https://example.com/foo.git##-v1.*    - default set minus v1.*
-     \
-
-     A \cb{git} repository URL fragment is a comma-separated list of reference
-     filters in the following form:
-
-     \c{[\i{refname}][\b{@}\i{commit}]}
-
-     Either \ci{refname}, \ci{commit}, or both must be specified. If both are
-     specified then \ci{refname} is only used to minimize the amount of data
-     fetched and \ci{commit} must belong to its history. For example:
-
-     \
-     .../foo.git#master@48fba3625d65941bb85a39061bcf795d4949c778
-     \
-
-     The \ci{refname} part can be an abbreviated commit id or an advertised
-     reference or reference pattern under \cb{refs/}. While \ci{commit} must
-     be the complete, 40-characters SHA1 that need not be advertised. For
-     convenience, a 40-characters filter that consists of only hexadecimal
-     digits is assumed to be \ci{commit} even if not prefixed with \cb{@}. In
-     an unlikely event this produces an incorrect result, the \cb{@}-form with
-     omitted \ci{commit} can be used. For example:
-
-     \
-     .../foo.git#48fba3625d65941bb85a39061bcf795d4949c778   (commit id)
-     .../foo.git#deadbeefdeadbeefdeadbeefdeadbeefdeadbeef@  (reference)
-     \
-
-     The \ci{refname} part can use the \cb{*} and \cb{?} wildcard pattern
-     characters with the standard semantics as well as the \cb{**} character
-     sequence which matches in subdirectories, recursively. For example:
-
-     \
-     .../foo.git#tags/v*    - tags/v1.2.3 but not tags/old/v0.1.0
-     .../foo.git#tags/v**   - tags/v1.2.3 and tags/old/v0.1.0
-     \
-
-     A relative \ci{refname} is searched for in \cb{refs/}, \cb{refs/tags/},
-     and \cb{refs/heads/} as well as symbolic references like \cb{HEAD}. To
-     anchor it to \cb{refs/} make it absolute, for example:
-
-     \
-     .../foo.git#tags/v*   - refs/tags/v1.2.3 but also refs/heads/tags/voo
-     .../foo.git#/tags/v*  - refs/tags/v1.2.3 only
-     \
-
-     While a \ci{refname} pattern may not match any references, a non-pattern
-     that doesn't resolve to a reference is invalid.
-
-     If a \ci{refname} starts with \cb{-} then it is treated as an exclusion
-     filter \- any references that it matches are excluded from the set
-     included by the preceding filters (or the default set). For example:
-
-     \
-     .../foo.git#v*,-v1.*  - exclude v1.* from v*
-     .../foo.git##-v1.*    - exclude v1.* from default set
-     \
-
-     To support specifying literal leading \cb{-}, a \ci{refname} that starts
-     with \cb{+} is treated as an inclusion filter. For example:
-
-     \
-     .../foo.git#+x   - include  x
-     .../foo.git#+-x  - include -x
-     .../foo.git#++x  - include +x
-     \
-
-     A \cb{git} repository has the same structure and manifest files as the
-     \cb{dir} repository. See \l{bpkg#manifest-package-list-dir Package List
-     Manifest for \cb{dir} Repositories} and \l{bpkg#manifest-repository-list
-     Repository List Manifest} for details on the format and semantics of the
-     manifest files.
-
-     Supported git protocols are \cb{git://}, \cb{http://}, and \cb{https://}
-     for remote repositories and \cb{file://} for local repositories. While
-     \cb{bpkg} tries to minimize the amount of information (history) fetched,
-     it is not always possible for some protocols and/or server
-     configurations, as discussed next.
-
-     A \cb{git} repository accessible via \cb{http(s)://} can use either
-     \i{dumb} or \i{smart} protocol (refer to the \cb{git} documentation for
-     details). The dumb protocol provides only limited support for fetch
-     minimization and if this protocol is used, then \cb{bpkg} has no choice
-     but to download a substantial amount of history.
-
-     The smart protocol allows fetching of minimal history for tags and
-     branches. Whether this is also possible for (all) commit ids depends on
-     whether the server is configured to allow fetching unadvertised commits.
-     For details, refer to the \cb{uploadpack.allowReachableSHA1InWant} and
-     \cb{uploadpack.allowAnySHA1InWant} \cb{git} configuration values.
-
-     The \cb{git://} protocol is similar to smart \cb{http://} in that it
-     supports fetching minimal history for tags and branches and may or may
-     not support this for commit ids depending on the server configuration.
-     Note, however, that unlike for \cb{http(s)://}, for this protocol
-     \cb{bpkg} does not try to sense if fetching unadvertised commits is
-     allowed and always assumes that it is not.
-
-     Based on this information, to achieve optimal results the recommended
-     protocol for remote repositories is smart \cb{https://}. Additionally, if
-     you are planning to refer to commit ids, then also consider configuring
-     the server to allow fetching unadvertised commits.
-
-     The \cb{file://} protocol has the same fetch minimization support as
-     \cb{git://} and is therefore treated the same."
+     Currently three types of repositories are supported: archive-based
+     \cb{pkg}, directory-based \cb{dir}, and version control-based \cb{git}.
+     See \l{bpkg-repository-types(1)} for details on their structure and URL
+     format.
+
+     Normally the repository type can be automatically guessed by examining
+     its URL (for example, the presence of the \cb{.git} extension) or, in
+     case of a local repository, its content (for example, the presence of the
+     \cb{.git/} subdirectory). Without any identifying information the
+     \cb{pkg} type is assumed unless explicitly specified with the \cb{--type}
+     option. Note, however, that the \cb{dir} repository type is never guessed
+     since it is not easily distinguishable from local \cb{pkg} and \cb{git}
+     repositories.
+     "
   }
 
   class rep_add_options: configuration_options
diff --git a/bpkg/repository-types.cli b/bpkg/repository-types.cli
new file mode 100644
index 0000000..d3cdbdf
--- /dev/null
+++ b/bpkg/repository-types.cli
@@ -0,0 +1,190 @@
+// file      : bpkg/repository-types.cli
+// copyright : Copyright (c) 2014-2017 Code Synthesis Ltd
+// license   : MIT; see accompanying LICENSE file
+
+include <bpkg/common-options.hxx>;
+
+"\section=1"
+"\name=bpkg-repository-types"
+"\summary=repository types, structure, and URLs"
+
+"
+\h|DESCRIPTION|
+
+This help topic describes the repository types recognized by \cb{bpkg}, their
+structure, and the format of their URLs. Currently three types of repositories
+are supported: archive-based \cb{pkg}, directory-based \cb{dir}, and version
+control-based \cb{git}.
+
+
+\h|PKG REPOSITORIES|
+
+A \cb{pkg} repository is \i{archive}-based. That is, it contains a collection
+of various packages/versions as archive files. For more information on the
+structure of \cb{pkg} repositories refer to \l{bpkg The \cb{build2} Package
+Manager Manual}. The \cb{dir} repository location can be a local directory
+path or an \cb{http(s)://} URL.
+
+
+\h|DIR REPOSITORIES|
+
+A \cb{dir} repository is \i{directory}-based. That is, it contains a
+collection of various packages as directories (but only a single version per
+package can be present in such a repository). The \cb{dir} repository location
+can be a local directory path or a \cb{file://} URL.
+
+A \cb{dir} repository is expected to contain either the \cb{manifest} or
+\cb{packages.manifest} file in the root directory of the repository. If it
+only contains \cb{manifest}, then it is assumed to be a simple, single-package
+repository with the \cb{manifest} file being its package manifest. Otherwise,
+the \cb{packages.manifest} file should list the available packages as
+described in \l{bpkg#manifest-package-list-dir Package List Manifest for
+\cb{dir} Repositories}.
+
+A \cb{dir} repository may also contain the \cb{repositories.manifest} file in
+the root directory of the repository. This file can be used to describe the
+repository itself as well as specify its prerequisite and complement
+repositories. See \l{bpkg#manifest-repository-list Repository List Manifest}
+for details on the format and semantics of this file.
+
+
+\h|GIR REPOSITORIES|
+
+A \cb{git} repository is \i{version control}-based. That is, it normally
+contains multiple versions of the same package (but can also contain several
+packages in the same repository).
+
+A \cb{git} repository has the same structure and manifest files as the
+\cb{dir} repository. See \l{bpkg#manifest-package-list-dir Package List
+Manifest for \cb{dir} Repositories} and \l{bpkg#manifest-repository-list
+Repository List Manifest} for details on the format and semantics of the
+manifest files.
+
+Theoretically, a \cb{git} repository may contain as many package versions as
+there are commits. Practically, however, we are normally only interested in a
+small subset of them while fetching and processing the necessary information
+for all of them could be prohibitively expensive.  As a result, by default,
+only advertised tags in the \cb{refs/tags/v*} form where the part after \cb{v}
+is also a valid \l{b#module-version standard version} are considered to be
+sources of useful package versions. These commits normally correspond to
+releases and are called the default set.
+
+Instead of the default set, it is also possible to provide a custom set of
+available versions by specifying one or more commit ids and/or references
+and/or reference patterns in the repository URL fragment (see
+\cb{git-ls-remote(1)} for details on advertised references). For example:
+
+\
+https://example.com/foo.git#v1.2.3
+https://example.com/foo.git#master
+https://example.com/foo.git#af234f56
+https://example.com/foo.git#tags/releases/*
+https://example.com/foo.git#HEAD,tags/v1.*.*,heads/feature-*
+\
+
+Furthermore, it is possible to expand (or narrow down) the default set using
+the special \cb{##} fragment notation. For example:
+
+\
+https://example.com/foo.git##HEAD     - default set plus HEAD
+https://example.com/foo.git##heads/*  - default set plus branches
+https://example.com/foo.git##-v1.*    - default set minus v1.*
+\
+
+A \cb{git} repository URL fragment is a comma-separated list of reference
+filters in the following form:
+
+\c{[\i{refname}][\b{@}\i{commit}]}
+
+Either \ci{refname}, \ci{commit}, or both must be specified. If both are
+specified then \ci{refname} is only used to minimize the amount of data
+fetched and \ci{commit} must belong to its history. For example:
+
+\
+.../foo.git#master@48fba3625d65941bb85a39061bcf795d4949c778
+\
+
+The \ci{refname} part can be an abbreviated commit id or an advertised
+reference or reference pattern under \cb{refs/}. While \ci{commit} must be the
+complete, 40-characters SHA1 that need not be advertised. For convenience, a
+40-characters filter that consists of only hexadecimal digits is assumed to be
+\ci{commit} even if not prefixed with \cb{@}. In an unlikely event this
+produces an incorrect result, the \cb{@}-form with omitted \ci{commit} can be
+used. For example:
+
+\
+.../foo.git#48fba3625d65941bb85a39061bcf795d4949c778   (commit id)
+.../foo.git#deadbeefdeadbeefdeadbeefdeadbeefdeadbeef@  (reference)
+\
+
+The \ci{refname} part can use the \cb{*} and \cb{?} wildcard pattern
+characters with the standard semantics as well as the \cb{**} character
+sequence which matches in subdirectories, recursively. For example:
+
+\
+.../foo.git#tags/v*    - tags/v1.2.3 but not tags/old/v0.1.0
+.../foo.git#tags/v**   - tags/v1.2.3 and tags/old/v0.1.0
+\
+
+A relative \ci{refname} is searched for in \cb{refs/}, \cb{refs/tags/}, and
+\cb{refs/heads/} as well as symbolic references like \cb{HEAD}. To anchor it
+to \cb{refs/} make it absolute, for example:
+
+\
+.../foo.git#tags/v*   - refs/tags/v1.2.3 but also refs/heads/tags/voo
+.../foo.git#/tags/v*  - refs/tags/v1.2.3 only
+\
+
+While a \ci{refname} pattern may not match any references, a non-pattern that
+doesn't resolve to a reference is invalid.
+
+If a \ci{refname} starts with \cb{-} then it is treated as an exclusion filter
+\- any references that it matches are excluded from the set included by the
+preceding filters (or the default set). For example:
+
+\
+.../foo.git#v*,-v1.*  - exclude v1.* from v*
+.../foo.git##-v1.*    - exclude v1.* from default set
+\
+
+To support specifying literal leading \cb{-}, a \ci{refname} that starts with
+\cb{+} is treated as an inclusion filter. For example:
+
+\
+.../foo.git#+x   - include  x
+.../foo.git#+-x  - include -x
+.../foo.git#++x  - include +x
+\
+
+Supported \cb{git} protocols are \cb{git://}, \cb{http://}, and \cb{https://}
+for remote repositories and \cb{file://} for local repositories. While
+\cb{bpkg} tries to minimize the amount of information (history) fetched, it is
+not always possible for some protocols and/or server configurations, as
+discussed next.
+
+A \cb{git} repository accessible via \cb{http(s)://} can use either \i{dumb}
+or \i{smart} protocol (refer to the \cb{git} documentation for details). The
+dumb protocol provides only limited support for fetch minimization and if this
+protocol is used, then \cb{bpkg} has no choice but to download a substantial
+amount of history.
+
+The smart protocol allows fetching of minimal history for tags and
+branches. Whether this is also possible for (all) commit ids depends on
+whether the server is configured to allow fetching unadvertised commits. For
+details, refer to the \cb{uploadpack.allowReachableSHA1InWant} and
+\cb{uploadpack.allowAnySHA1InWant} \cb{git} configuration values.
+
+The \cb{git://} protocol is similar to smart \cb{http://} in that it supports
+fetching minimal history for tags and branches and may or may not support this
+for commit ids depending on the server configuration. Note, however, that
+unlike for \cb{http(s)://}, for this protocol \cb{bpkg} does not try to sense
+if fetching unadvertised commits is allowed and always assumes that it is not.
+
+Based on this information, to achieve optimal results the recommended protocol
+for remote repositories is smart \cb{https://}. Additionally, if you are
+planning to refer to commit ids, then also consider configuring the server to
+allow fetching unadvertised commits.
+
+The \cb{file://} protocol has the same fetch minimization support as
+\cb{git://} and is therefore treated the same.
+"
diff --git a/doc/cli.sh b/doc/cli.sh
index 06d981e..a9d8a6c 100755
--- a/doc/cli.sh
+++ b/doc/cli.sh
@@ -63,7 +63,7 @@ compile "pkg-build" $o --class-doc  bpkg::pkg_build_pkg_options=exclude-base
 pages="cfg-create help pkg-clean pkg-configure pkg-disfigure \
 pkg-drop pkg-fetch pkg-checkout pkg-install pkg-purge pkg-status pkg-test \
 pkg-uninstall pkg-unpack pkg-update pkg-verify rep-add rep-remove rep-list \
-rep-create rep-fetch rep-info repository-signing"
+rep-create rep-fetch rep-info repository-signing repository-types"
 
 for p in $pages; do
   compile $p $o