Windows: Major update to Windows build instructions.

7 files changed, 373 insertions, 147 deletions

diff --git a/windows/INSTALL-MSVC.txt b/windows/INSTALL-MSVC.txt
index d9574388..ba483fb6 100644
--- a/windows/INSTALL-MSVC.txt
+++ b/windows/INSTALL-MSVC.txt
@@ -5,11 +5,15 @@ Building XZ Utils with Microsoft Visual Studio
 Introduction
 ------------
 
-    liblzma got MSVC support in XZ Utils 5.2.0, and the xz and xzdec
-    command line tools in XZ Utils 5.6.0.
+    liblzma got MSVC support in XZ Utils 5.2.0, and the xz, xzdec,
+    lzmadec, and lzmainfo command line tools in XZ Utils 5.6.0.
 
-        Note: xz and xzdec depend on the the included GNU getopt
-        which is licensed under the GNU LGPLv2.1.
+        NOTE: The *.exe files are linked against GNU getopt_long from
+        the "lib" directory. That code is under the GNU LGPLv2.1
+        and thus the matching source code must be provided when
+        distributing the *.exe files. The simplest way to comply with
+        the license requirements is to distribute the matching XZ Utils
+        source package alongside the *.exe files.
 
     MSVC 2013 update 2 and later have enough C99 support to build
     liblzma from XZ Utils 5.2.0 and later without modifications.
@@ -21,12 +25,13 @@ Introduction
 Building
 --------
 
-    It is recommended to use CMake to generate build files for MSVC.
-    Visual Studio project files are no longer provided (XZ Utils 5.4.x
-    were the last versions to include the project files).
+    Use CMake to generate build files for MSVC. Visual Studio project
+    files are no longer provided (XZ Utils 5.4.x were the last versions
+    to include the project files).
 
-    CMake-based build includes tests for liblzma but not for the
-    command line tools.
+    NOTE: GCC and Clang compatible inline assembly isn't supported by
+    the MSVC compiler. Using clang-cl under MSVC should make inline
+    assembly work (untested).
 
 
 Notes
diff --git a/windows/INSTALL-MinGW-w64_with_Autotools.txt b/windows/INSTALL-MinGW-w64_with_Autotools.txt
new file mode 100644
index 00000000..89c74409
--- /dev/null
+++ b/windows/INSTALL-MinGW-w64_with_Autotools.txt
@@ -0,0 +1,49 @@
+
+Creating XZ Utils Windows package with build.bash
+=================================================
+
+Introduction
+------------
+
+    The script build.bash can be used for building XZ Utils with
+    GCC + MinGW-w64 under MSYS2, under the ancient MSYS, or
+    cross-compiling from GNU/Linux. The script will create a package
+    with binaries and documentation in a hopefully-convenient bundle.
+
+        NOTE: build.bash requires files that are only included
+        in release tarballs. If building from xz.git, a distribution
+        tarball should be created first.
+
+    For native builds on Windows, the CMake-based build described
+    in the file INSTALL-MinGW-w64_with_CMake.txt is simpler to do as
+    it has no need for MSYS2 and it works from xz.git without extra
+    steps. For cross-compilation and package creation the script can
+    be convenient though.
+
+    These instructions are for making a package with build.bash and thus
+    don't apply to normal Autotool-based builds under Cygwin or MSYS2.
+
+
+Usage
+-----
+
+    First copy the file COPYING.MinGW-w64-runtime.txt from MinGW-w64
+    to this directory. It contains copyright and license notices that
+    apply to the MinGW-w64 runtime that gets statically linked into
+    the XZ Utils binaries being built. build.bash will include the file
+    in the final package.
+
+    Put i686 and/or x86_64 GCC-based toolchain in PATH depending on
+    which builds are wanted.
+
+    Optional: Put the 7z tool from 7-Zip or p7zip in PATH. Without
+    this, .zip and .7z files won't be created from the finished "pkg"
+    directory contents.
+
+    Run build.bash:
+
+        bash windows/build.bash
+
+    Note that it does an in-tree build so the build files will be mixed
+    with the source files in the same directory tree.
+
diff --git a/windows/INSTALL-MinGW-w64_with_CMake.txt b/windows/INSTALL-MinGW-w64_with_CMake.txt
new file mode 100644
index 00000000..5d2be5ba
--- /dev/null
+++ b/windows/INSTALL-MinGW-w64_with_CMake.txt
@@ -0,0 +1,203 @@
+
+Building XZ Utils on Windows using MinGW-w64 and CMake
+======================================================
+
+    1. Introduction
+       1.1. Licensing considerations
+    2. MSVCRT or UCRT
+    3. CMake
+    4. MinGW-w64 toolchains
+       4.1. MinGW-w64 with GCC
+       4.2. MinGW-w64 with Clang/LLVM
+    5. Building XZ Utils
+       5.1. Advanced build options
+    6. Creating an import library for MSVC / Visual Studio
+
+
+1. Introduction
+---------------
+
+    This document explains how to build XZ Utils using MinGW-w64,
+    GCC or Clang/LLVM, CMake, and GNU make (mingw32-make) natively
+    on Windows. The resulting XZ Utils library and executable files
+    will only depend on DLLs that are included in Windows.
+
+    The build tools can be extracted into separate directories and used
+    directly from there and deleted when no longer needed. There are no
+    installers to run for these and no configuration needed.
+
+    These instructions don't apply to Cygwin. XZ Utils can be built
+    under Cygwin in the same way as many other packages.
+
+
+1.1. Licensing considerations
+
+    Parts of MinGW-w64 runtime are statically linked into the binaries
+    being built. The file COPYING.MinGW-w64-runtime.txt in MinGW-w64
+    contains the license notices that apply to some parts of the
+    runtime. The notices must be distributed alongside the binaries
+    that have been built with MinGW-w64.
+
+    MinGW-w64 includes getopt_long(). The GNU getopt_long() (LGPLv2.1)
+    included in XZ Utils isn't used when building with MinGW-w64.
+
+    The code from XZ Utils that ends up liblzma.dll and the *.exe files
+    is under the BSD Zero Clause License (0BSD) which doesn't require
+    any copyright or license notices to be included when distributing
+    the binaries. See the file COPYING in the parent directory.
+
+
+2. MSVCRT or UCRT
+-----------------
+
+    Both GCC and Clang/LLVM based MinGW-w64 toolchains come in MSVCRT
+    and Universal C runtime (UCRT) variants. MSVCRT is the old one.
+    32-bit builds of XZ Utils with MSVCRT should run on Windows 2000
+    and later (even Windows 95 should still be possible with trivial
+    edits to the source code).
+
+    UCRT is included in Windows 10, and it's possible to install UCRT
+    on Windows XP and later. UCRT might be the preferred choice if
+    out-of-the-box compatibility with Windows versions older than 10
+    is not required. Visual Studio 2015 and later produce binaries
+    that use UCRT.
+
+    If you want to build liblzma.dll for use with your application,
+    it's recommended to use the same CRT for all components. If this
+    isn't possible, see the file liblzma-crt-mixing.txt.
+
+    If you only need the command line tools, the choice of CRT isn't
+    important, at least for now.
+
+
+3. CMake
+--------
+
+    CMake is used for selecting build options and generating makefiles.
+    It can also be used to extract archives, including .tar.xz and .7z.
+
+    Download a CMake binary package (.zip) from its homepage:
+
+        https://cmake.org/download/
+
+    Extract it to, for example, C:\devel\cmake so that the executables
+    end up in C:\devel\cmake\bin. Avoid spaces and other special
+    characters in the path.
+
+
+4. MinGW-w64 toolchains
+-----------------------
+
+    There are a few choices of prebuilt toolchains listed on
+    the MinGW-w64 homepage:
+
+        https://www.mingw-w64.org/downloads/
+
+    These instructions list one GCC-based version and one
+    Clang/LLVM-based version. Both include mingw32-make too.
+
+
+4.1. MinGW-w64 with GCC
+
+    For GCC, download appropriate packages from Mingw-builds depending
+    on if you want to build 32-bit or 64-bit x86 version of XZ Utils
+    and if the XZ Utils binaries should link against MSVCRT or UCRT:
+
+        https://github.com/niXman/mingw-builds-binaries/releases
+
+        i686-*-release-win32-*-msvcrt-*.7z    32-bit, uses MSVCRT (old)
+        i686-*-release-win32-*-ucrt-*.7z      32-bit, uses UCRT (new)
+        x86_64-*-release-win32-*-msvcrt-*.7z  64-bit, uses MSVCRT (old)
+        x86_64-*-release-win32-*-ucrt-*.7z    64-bit, uses UCRT (new)
+
+    Extract it, for example, to C:\devel so that the executables are
+    in C:\devel\mingw32\bin or C:\devel\mingw64\bin. To extract,
+    you can install 7-Zip from <https://7-zip.org/> or use CMake
+    on the command line:
+
+        set PATH=C:\devel\cmake\bin;%PATH%
+        c:
+        cd \devel
+        cmake -E tar xf x86_64-13.1.0-release-win32-seh-ucrt-rt_v11-rev1.7z
+
+    Then skip to the section "Building XZ Utils".
+
+
+4.2. MinGW-w64 with Clang/LLVM
+
+    For Clang/LLVM, download an appropriate package from LLVM-MinGW:
+
+        https://github.com/mstorsjo/llvm-mingw/releases
+
+        llvm-mingw-*-msvcrt-i686.zip          32-bit, uses MSVCRT (old)
+        llvm-mingw-*-ucrt-i686.zip            32-bit, uses UCRT (new)
+        llvm-mingw-*-msvcrt-x86_64.zip        64-bit, uses MSVCRT (old)
+        llvm-mingw-*-ucrt-x86_64.zip          64-bit, uses UCRT (new)
+
+    Extract it, for example, to C:\devel so that the executables end up
+    in a directory like C:\devel\llvm-mingw-20230919-ucrt-x86_64\bin.
+
+
+5. Building XZ Utils
+--------------------
+
+    For a simple builds, you can use the included build-with-cmake.bat
+    which takes these arguments:
+
+    %1 = Path to CMake's bin directory. Example:
+         c:\devel\cmake\bin
+
+    %2 = Path to MinGW-w64's bin directory. Example:
+         c:\devel\mingw64\bin
+
+    %3 = ON or OFF: Set to ON to build liblzma.dll or OFF for
+         static liblzma.a. With OFF, the *.exe files won't
+         depend on liblzma.dll.
+
+    Example:
+
+        build-with-cmake C:\devel\cmake\bin C:\devel\mingw64\bin ON
+
+    If successful, the "build" directory should then contain:
+
+        liblzma.dll     liblzma compression library
+        liblzma.def     DEF file for creating an import library
+        xz.exe          xz command line tool
+        xzdec.exe       Decompression-only tool (smaller than xz.exe)
+        lzmadec.exe     Decompression-only tool for legacy .lzma files
+        lzmainfo.exe    Shows header info of legacy .lzma files
+
+    Ignore the other files. :-)
+
+
+5.1. Advanced build options
+
+    For 32-bit x86 builds, adding -msse2 to CFLAGS improves
+    compression speed a little (but not decompression speed).
+    There is no runtime detection for SSE2 support. It is
+    recommended to use 64-bit version when possible.
+
+    It's possible to omit features from the build to reduce code size.
+    There are several CMake configuration options available. One may
+    change from CMAKE_BUILD_TYPE=Release to =MinSizeRel as well but
+    it makes the code slower.
+
+    If building for multiple targets, keep only one toolchain in PATH
+    at a time.
+
+
+6. Creating an import library for MSVC / Visual Studio
+------------------------------------------------------
+
+    To link against liblzma.dll, you need to create an import library
+    first. You need the "lib" command from MSVC and liblzma.def. Here
+    is the command that works on 32-bit x86:
+
+        lib /def:liblzma.def /out:liblzma.lib /machine:ix86
+
+    On x86-64, the /machine argument has to be changed:
+
+        lib /def:liblzma.def /out:liblzma.lib /machine:x64
+
+    IMPORTANT: See also the file liblzma-crt-mixing.txt.
+
diff --git a/windows/INSTALL-MinGW.txt b/windows/INSTALL-MinGW.txt
deleted file mode 100644
index 72e2718f..00000000
--- a/windows/INSTALL-MinGW.txt
+++ /dev/null
@@ -1,138 +0,0 @@
-
-Building XZ Utils on Windows
-============================
-
-Introduction
-------------
-
-    This document explains shortly where to get and how to install the
-    build tools that are needed to build XZ Utils on Windows. The final
-    binary package will be standalone in sense that it will depend only
-    on DLLs that are included in all Windows installations.
-
-    These instructions don't apply to Cygwin. XZ Utils can be built under
-    Cygwin in the same way as many other packages.
-
-    These instructions don't apply to MinGW and MSYS developers either,
-    who may want to package XZ Utils for MinGW or MSYS distributions.
-    You know who you are, and will probably use quite different configure
-    options etc. than what is described here.
-
-
-Installing the toolchain(s)
----------------------------
-
-    Some of the following is needed:
-      - MSYS is always needed to use the GNU Autotools based build system.
-      - MinGW builds 32-bit x86 binaries.
-      - 32-bit MinGW-w64 (I call it MingW-w32 here) builds 32-bit x86
-        executables too.
-      - MinGW-w64 builds 64-bit x86-64 binaries.
-
-    So you need to pick between MinGW and MinGW-w32 when building
-    32-bit version. You don't need both.
-
-    You might find 7-Zip <https://7-zip.org/> handy when extracting
-    some files. The ready-made build script build.bash will also use
-    7-Zip to create the distributable .zip and .7z files.
-
-    I used the following directory structure but you can use whatever
-    you want. Just note that I will use these in my examples. Each of
-    these should have a subdirectory "bin":
-
-        C:\devel\tools\msys
-        C:\devel\tools\mingw
-        C:\devel\tools\mingw-w32
-        C:\devel\tools\mingw-w64
-
-
-Installing MSYS
-
-    You can download MSYS from MinGW's Sourceforge page:
-
-        https://sourceforge.net/projects/mingw/files/MSYS/Base/msys-core/
-
-    I recommend using MSYS 1.0.11 (MSYS-1.0.11.exe or
-    msysCORE-1.0.11-bin.tar.gz) because that package includes all the
-    required tools. At least some of the later versions include only
-    a subset and thus you would need to download the rest separately.
-    The old version will work fine for building XZ Utils.
-
-    You can use either the .exe or .tar.gz package. I prefer .tar.gz,
-    because it can be extracted into any directory and later removed
-    without worrying about uninstallers.
-
-
-Installing MinGW
-
-    NOTE: This section may be outdated. I haven't tried MinGW recently.
-
-    You can download the required packages from MinGW's Sourceforge page:
-
-        https://sourceforge.net/projects/mingw/files/
-
-    These version numbers were the latest when I wrote this document, but
-    you probably should pick the latest versions:
-
-        MinGW Runtime -> mingwrt-3.17-mingw32-dev.tar.gz
-        MinGW API for MS-Windows -> w32api-3.14-mingw32-dev.tar.gz
-        GNU Binutils -> binutils-2.20-1-bin.tar.gz
-        GCC Version 4 -> gcc-full-4.4.0-mingw32-bin-2.tar.lzma
-
-    The full GCC package is quite big, but if you want a smaller
-    download, you will need to download more than one file, so I'm
-    using the full package in this document for simplicity.
-
-    Extract the packages in the above order, possibly overwriting files
-    from packages that were extracted earlier.
-
-
-Installing MinGW-w32 or MinGW-w64
-
-    I used the packages from Mingw-builds project. With that it is
-    enough to pick one .7z file for 32-bit and another for 64-bit
-    toolchain. For XZ Utils 5.2.0 I used the packages from these
-    directories:
-
-        https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.9.2/threads-win32/sjlj/
-
-        https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.9.2/threads-win32/sjlj/
-
-    If you install both MinGW-w32 and MinGW-w64, remember to extract
-    them into different directories. build.bash looks at
-    C:\devel\tools\mingw-w32 and C:\devel\tools\mingw-w64 by default.
-
-
-Building XZ Utils
------------------
-
-    Start MSYS by going to the directory C:\devel\tools\msys and running
-    msys.bat there (double-click or use command prompt). It will start
-    at "home" directory, which is C:\devel\tools\msys\home\YourUserName.
-
-    If you have xz-5.x.x.tar.gz in C:\devel, you should be able to build
-    it now with the following commands:
-
-        cd /c/devel
-        tar xzf xz-5.x.x.tar.gz
-        cd xz-5.x.x
-        bash windows/build.bash
-
-    If you used some other directory than C:\devel\tools for the build
-    tools, edit the variables near the beginning of build.bash first.
-
-    If you want to build manually, read the buildit() function in
-    build.bash. Look especially at the latter configure invocation.
-
-    Be patient. Running configure and other scripts used by the build
-    system is (very) slow under Windows.
-
-
-Using a snapshot from the Git repository
-
-    To use a snapshot, the build system files need to be generated with
-    autogen.sh or "autoreconf -fi" before trying to build using the
-    above build instructions. You can install the relevant extra packages
-    from MinGW or use Cygwin or use e.g. a GNU/Linux system to create a
-    source package with the required build system files.
-
diff --git a/windows/README-Windows.txt b/windows/README-Windows.txt
index 65dc9db1..c7820f8c 100644
--- a/windows/README-Windows.txt
+++ b/windows/README-Windows.txt
@@ -92,6 +92,8 @@ Creating an import library for MSVC / Visual Studio
 
         lib /def:liblzma.def /out:liblzma.lib /machine:x64
 
+    IMPORTANT: See also the file liblzma-crt-mixing.txt.
+
 
 Reporting bugs
 --------------
diff --git a/windows/build-with-cmake.bat b/windows/build-with-cmake.bat
new file mode 100644
index 00000000..24978ba8
--- /dev/null
+++ b/windows/build-with-cmake.bat
@@ -0,0 +1,35 @@
+@rem # SPDX-License-Identifier: 0BSD

+@rem # Author: Lasse Collin

+@rem #

+@rem ########################################################################

+@rem #

+@rem # This builds XZ Utils with CMake + MinGW-w64 (GCC or Clang/LLVM).

+@rem # See INSTALL-MinGW-w64_with_CMake.txt for detailed instructions.

+@rem #

+@rem # Summary of command line arguments:

+@rem #

+@rem # %1 = Path to CMake's bin directory. Example:

+@rem #      C:\devel\cmake\bin

+@rem #

+@rem # %2 = Path to MinGW-w64's bin directory. Example:

+@rem #      C:\devel\mingw64\bin

+@rem #

+@rem # %3 = ON or OFF: Set to ON to build liblzma.dll or OFF for

+@rem #      static liblzma.a. With OFF, the *.exe files won't

+@rem #      depend on liblzma.dll.

+@rem #

+@rem ########################################################################

+

+setlocal

+set PATH=%1;%2;%PATH%

+

+md build || exit /b

+cd build || exit /b

+

+cmake -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release -DENABLE_NLS=OFF -DBUILD_SHARED_LIBS=%3 ..\.. || exit /b

+mingw32-make || exit /b

+mingw32-make test || exit /b

+

+@rem liblzma.dll might not exist so ignore errors.

+strip xz.exe xzdec.exe lzmadec.exe lzmainfo.exe liblzma.dll

+exit /b 0

diff --git a/windows/liblzma-crt-mixing.txt b/windows/liblzma-crt-mixing.txt
new file mode 100644
index 00000000..4ef2f8ab
--- /dev/null
+++ b/windows/liblzma-crt-mixing.txt
@@ -0,0 +1,70 @@
+
+liblzma.dll and mixing C runtimes (CRTs)
+----------------------------------------
+
+    If possible, liblzma.dll should be linked against the same CRT
+    (MSVCRT or UCRT) as the application calling the liblzma functions.
+    When this isn't possible, liblzma.dll will still work but there
+    are a few API functions that need extra care.
+
+    Each CRT has its own memory allocator, stdio FILE implementation,
+    mapping of file descriptors from _open() to Windows' HANDLEs, and
+    so on. Mixing CRTs is a problem if, for example, one library calls
+    fopen() and then passes the resulting FILE* to a second library and
+    these two libraries use different CRTs. liblzma doesn't expose FILE
+    pointers or file descriptors in the API but the problem can occur
+    with memory allocation with a few specific functions.
+
+    The most commonly-used API functions in liblzma are such that both
+    memory allocation and deallocation is done internally by liblzma,
+    thus most applications won't have any problems with mixing CRTs
+    with liblzma.dll. The following API functions are the exception:
+
+        lzma/block.h:
+            lzma_block_header_decode
+
+        lzma/filter.h:
+            lzma_filters_copy
+            lzma_filters_free
+            lzma_properties_decode
+            lzma_filter_flags_decode
+            lzma_str_to_filters
+            lzma_str_from_filters
+            lzma_str_list_filters
+
+    Excluding lzma_filters_free(), the above functions allocate memory
+    and leave it to the caller to free it. lzma_filters_free() frees
+    memory given to it, and that memory may have been allocated outside
+    of liblzma.
+
+    For example, if application calls lzma_str_list_filters(&ptr, ...)
+    and then uses free(ptr), something bad (memory corruption, crash)
+    will happen if the application and liblzma.dll aren't using the
+    same CRT. This can be worked around with a few lines of extra code.
+
+    All these functions (and many others too) take a pointer to
+    lzma_allocator structure as an argument. Typically it is set to
+    NULL to let liblzma use malloc() and free() (and also calloc()
+    as it is faster than malloc() + memset()). A custom lzma_allocator
+    can be used to wrap malloc() and free() from application's CRT:
+
+        static void * LZMA_API_CALL
+        my_alloc(void *opaque, size_t nmemb, size_t size)
+        {
+            // liblzma guarantees that this won't overflow.
+            return malloc(nmemb * size);
+        }
+
+        static void LZMA_API_CALL
+        my_free(void *opaque, void *ptr)
+        {
+            free(ptr);
+        }
+
+        static const lzma_allocator allocator
+                = { &my_alloc, &my_free, NULL };
+
+    By passing &allocator to the problematic functions, CRT mixing
+    should not cause any problems. There is no need to use &allocator
+    with functions other than those listed above.
+