summaryrefslogtreecommitdiff
path: root/README-DEV
diff options
context:
space:
mode:
authorKaren Arutyunov <karen@codesynthesis.com>2019-04-05 10:30:58 +0300
committerKaren Arutyunov <karen@codesynthesis.com>2019-04-05 17:53:59 +0300
commit72e7f011b29998d8a3e15eb5b381ef962af5fe5b (patch)
tree9fd87a0c6f545552849f466b20d3969a21e21e15 /README-DEV
parent4e0bcdb33411afb02d60b72cbffa278344a572a6 (diff)
Upgrade to 8.0.15
Diffstat (limited to 'README-DEV')
-rw-r--r--README-DEV248
1 files changed, 118 insertions, 130 deletions
diff --git a/README-DEV b/README-DEV
index 1eea89f..9c05759 100644
--- a/README-DEV
+++ b/README-DEV
@@ -1,83 +1,120 @@
-This document describes how libmysqlclient was packaged for build2. In
-particular, this understanding will be useful when upgrading to a new upstream
-version.
-
-The original libmysqlclient library is packaged together with other libraries,
-the MySQL server, and client utilities. The library source files are spread
-across several top-level subdirectories. These subdirectories also contain the
-library-irrelevant files (which we skip) along with examples and tests. The
-reasonable approach for defining the required source files is to exclude
-everything you have doubts about and rely on the linker reporting unresolved
-symbols. We balance between keeping the upstream package directory structure
-and making sure that the library can be properly imported into a build2
-projects. Below are the packaging steps in more detail.
-
-1. Copy include/mysql_version.h.in to mysql/version.h.in, and create
- mysql/mysql_version.h that includes <mysql/version.h>.
-
-2. Copy config.h.cmake to mysql/config.h.cmake.orig, and use it for creating
- mysql/my_config.h manually, defining/undefining only those macros that are
- used in the library source code (see below). Also create mysql/config.h
- that just includes <mysql/my_config.h>.
-
- Note that some macro values can not be easily determined at the
- preprocessing time. We define them based on the supported platform tests
- and add mysql/libmysql/assert.c, containing compile-time assertions for
- the macros in question.
-
- Also note that some of the macro values depend on the project
- configuration. We add such macro definitions into the mysql/version.h.in
- and include <mysql/version.h> into mysql/my_config.h. We could place them
- into a separate header template file, but probably there is already enough
- config and version headers.
-
-3. Copy libbinlogevents/export/binary_log_types.h to mysql/.
-
-4. Part of the upstream package library headers are located under the include/
- subdirectory. We will copy it recursively to mysql/, but only those headers
- that are required by the library source code. We start from an empty mysql/
- and add the headers that can't be found by the preprocessor. Note that you
- need to be careful not to pickup system-installed headers (see below).
-
-5. Recursively copy source files from zlib/, strings/, mysys/, mysys_ssl/,
- extra/yassl/, vio/, sql/, sql-common/ and libmysql/ to mysql/, preserving
- the original directory structure, except files not required for the library
- compilation/linkage (see above). Rename .cc files to .cpp afterwards, using
- the following command (upstream uses a mixture of extensions which we just
- not going to bother handling):
-
- $ for f in `find . -name '*.cc'`; do mv "$f" "${f%.cc}.cpp"; done
-
- Copy the READMEs and licenses for the bundled libraries as well.
-
-6. Copy client/get_password.c to mysql/extra/yassl/src/.
-
-7. Copy upstream package compile-time auto-generated include/mysqld_error.h
- to mysql/. It is generated from sql/share/errmsg-utf8.txt by the pre-built
- comp_err utility. The utility also produces some other headers (which we
- don't use) and localized errmsg.sys files that are not installed with the
- library. Could we not copy it from a binary or distribution or from
- Debian/Fedora package?
-
-8. Create mysql/libmysql/libmysql_exports_win32.def to contain a list of the
- exported names. For that purpose grep through libmysql/CMakeLists.txt to
- see how the .def file is generated for Windows. The corresponding code
- normally looks like:
-
- MERGE_LIBRARIES(libmysql SHARED ${LIBS}
- EXPORTS ${CLIENT_API_FUNCTIONS} ${CLIENT_API_FUNCTIONS_UNDOCUMENTED}
- COMPONENT SharedLibraries)
-
- If that's the case, collect names that get appended to the
- CLIENT_API_FUNCTIONS and CLIENT_API_FUNCTIONS_UNDOCUMENTED variables.
-
-9. Copy README to COPYING, and truncate it to contain only the licensing
- information.
-
-10. Copy COPYING to GPLv2.
-
-When merging libmysqlclient build2 package with a new version of the upstream
-package make sure that all the preprocessor include directives reference the
+This document describes how MySQL was packaged for build2. In particular, this
+understanding will be useful when upgrading to a new upstream version.
+
+The upstream package contains the MySQL server, libmysqlclient library and a
+number of client/administrative utilities (mysql, mysqlcheck, etc). Currently,
+we only package libmysqlclient (see libmysqlclient/README-DEV for details).
+
+We add the upstream package as a git submodule and symlink the required files
+and subdirectories into the build2 package subdirectories. Then, when required,
+we "overlay" the upstream with our own header/source files, placing them into
+the library/program directories and their downstream/ subdirectories.
+
+Note that symlinking upstream submodule subdirectories into a build2 package
+subdirectory results in creating intermediate build files (.d, .o, etc) inside
+upstream directory while building the package in source tree. That's why we
+need to make sure that packages do not share upstream source files via
+subdirectory symlinks, not to also share the related intermediate files. If
+several packages need to compile the same upstream source file, then only one
+of them can symlink it via the parent directory while others must symlink it
+directly.
+
+The upstream package can be configured to contain a specific feature set. We
+reproduce the union of features configured for the upstream source package in
+Debian and Fedora distributions. The configuration options defining these sets
+are specified in the Debian's rules and Fedora's RPM .spec files. Note,
+however, that at the time of this writing the latest packaged versions are
+different: 5.7.25 on Debian and 8.0.15 on Fedora. These files can be obtained
+as follows:
+
+$ wget https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/source/tree/Packages/c/community-mysql-8.0.15-1.fc30.src.rpm
+$ rpm2cpio community-mysql-8.0.15-1.fc30.src.rpm | cpio -civ '*.spec'
+
+#$ wget http://deb.debian.org/debian/pool/main/m/mysql-5.7/mysql-5.7_5.7.25-1.debian.tar.xz
+#$ tar xf mysql-5.7_5.7.25-1.debian.tar.xz debian/rules
+
+Instead of using MySQL 5.7.25 Debian's rules for MySQL 8.0.15 we are
+packaging, let's use upstream/packaging/deb-in/rules.in. That's not ideal but
+is, probably, better than nothing.
+
+Here are the discovered configuration options.
+
+Debian:
+
+ -DWITH_INNODB_MEMCACHED=ON -DWITH_NUMA=ON
+
+Fedora:
+
+ -DWITH_INNODB_MEMCACHED=ON -DENABLED_LOCAL_INFILE=ON
+
+Omitting options that are not related to the client library we end up with:
+
+ -DENABLED_LOCAL_INFILE=ON
+
+See the configuration options description at
+https://dev.mysql.com/doc/refman/8.0/en/source-configuration-options.html
+
+As a side note, on Debian and Fedora libmysqlclient is packaged under the
+libmysqlclient20 and community-mysql-libs package names respectively. The
+headers-containing development packages are libmysqlclient-dev and
+community-mysql-devel. Search for the Debian and Fedora packages at
+https://packages.debian.org/search and
+https://apps.fedoraproject.org/packages/.
+
+Normally, when packaging a cmake-based project, we try to deduce the source
+file and compilation/linking option sets analyzing the root and
+feature/component/platform-specific CMakeLists.txt and .cmake files. In
+practice, however, that can be uneasy and error prone, so you may also need
+to refer to cmake-generated configuration files or, as a last resort, to see
+the actual compiler and linker command lines in the build log. If that's the
+case, you can configure/build the upstream package on the platform of interest
+running the following commands in the project root directory.
+
+On POSIX:
+
+$ mkdir build
+$ cd build
+$ cmake -DDOWNLOAD_BOOST=ON -DWITH_BOOST=../boost -DWITHOUT_SERVER=ON \
+ -DENABLED_LOCAL_INFILE=ON ..
+
+$ make VERBOSE=1 >build.log 2>&1
+
+Here we assume that OpenSSL is installed into the system.
+
+With MSVC:
+
+Run Visual Studio and change the settings:
+
+Tools > Options > Projects and Solutions > Build and Run >
+MSBuild project build output verbosity: Detailed
+
+Build libssl from cppget.org, install to C:/openssl and rename *.dll.lib to
+*.lib to match MySQL's expectations.
+
+> set OPENSSL_ROOT_DIR=C:/openssl
+> set PATH=c:\openssl\bin;%PATH%
+> mkdir build
+> cd build
+> cmake -DDOWNLOAD_BOOST=ON -DWITH_BOOST=../boost -DWITHOUT_SERVER=ON ^
+ -DENABLED_LOCAL_INFILE=ON .. -G "Visual Studio 15 2017 Win64" ^ >config.log 2>&1
+> devenv MySQL.sln /build RelWithDebInfo >build-log 2>&1
+
+With MinGW GCC:
+
+MinGW GCC is not mentioned at the "Installing MySQL from Source" page as a
+supported compiler for Windows and is, presumably, unsupported. Trying to use
+it fails at the compilation stage:
+
+> set OPENSSL_ROOT_DIR=C:/openssl
+> set PATH=c:\openssl\bin;C:\msys64\mingw64\bin;%PATH%
+> mkdir build
+> cd build
+> cmake -DDOWNLOAD_BOOST=ON -DWITH_BOOST=../boost -DWITHOUT_SERVER=ON ^
+ -DENABLED_LOCAL_INFILE=ON .. -G "MinGW Makefiles"
+> mingw32-make
+
+When the packaging is complete, build all the project packages in source tree
+and make sure that all the preprocessor include directives reference the
packaged header files, rather than MariaDB or MySQL headers that are installed
into the system. It's easy to miss some headers in the package if MariaDB or
MySQL development package is installed on the host. We also need to check if
@@ -86,55 +123,6 @@ build the merged project, concatenate the produced .d files, sort the resulting
file removing duplicates and edit the result, leaving only the system headers.
Afterwards grep through the remained headers for some patterns:
-$ cat `find . -name '*.d'` | sort -u >headers
+$ cat `find . -type f -name '*.d'` | sort -u >headers
$ emacs headers # Edit, leaving system headers only.
$ fgrep -e 'mysql' -e 'mariadb' -e 'openssl' -e 'zlib' headers
-
-Also make sure that the macros set in mysql/my_config.h is still up to date.
-For that purpose obtain the macros that are used in the new source base, then
-obtain the macros (un)defined in the current mysql/my_config.h and compare the
-sets. That can be achieved running the following commands in the build2 project
-root directory:
-
-$ for m in `cat mysql/config.h.cmake.orig | sed -n 's/.*#\s*\(define\|cmakedefine\)\s\{1,\}\([_a-zA-Z0-9]\{1,\}\)\(\s.*\)\{0,1\}$/\2/p' | sort -u`; do
- if grep -q -e "\b$m\b" `find . -name '*.h' -a ! -name 'my_config.h' -a ! -name 'config.h' -o -name '*.c'`; then
- echo "$m"
- fi
- done >used-macros
-
-$ cat mysql/my_config.h |
- sed -n 's/#\s*\(define\|undef\)\s\{1,\}\([_a-zA-Z0-9]\{1,\}\)\(\s.*\)\{0,1\}$/\2/p' |
- sort -u >defined-macros
-
-diff defined-macros used-macros
-
-To obtain the pre-defined macros for gcc and clang use following commands:
-
-$ gcc -dM -E - < /dev/null
-$ clang -dM -E - < /dev/null
-
-Note that some macro definitions are passed to the preprocessor via the -D
-command line options. Such macro sets may be specific for source file
-subdirectories. It makes sense to check that the sets used for the build2
-package still match the ones for the new upstream package. For that purpose you
-can grep the old and new upstream package CMakeList.txt files for
-ADD_DEFINITIONS() directives and review the changes. If needed, you may also
-run cmake for the upstream project and view the flags.make files created for
-the corresponding source directories. Or, as a last resort, you can see the
-actual compiler and linker command lines running make utility with VERBOSE=1
-option. For VC, you need to set output verbosity to Diagnostics level at the
-'Tools/Options/Projects and Solutions\Build and Run' dialog tab, change the
-current directory to the project build directory in CMD console, and run the
-following command:
-
-> devenv MySQL.sln /build >build.log
-
-It also makes sense to check for changes in compiler and linker flags. You may
-grep CMakeList.txt files for the appropriate directives, or you may compile the
-upstream project in the verbose mode on the platform of interest.
-
-To configure the upstream package for the build use the commands like this:
-
-$ mkdir out
-$ cd out
-$ cmake -DDOWNLOAD_BOOST=1 -DWITH_BOOST=boost -DWITHOUT_SERVER=ON ..