aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2024-02-17liblzma: Wrap a line exceeding 80 chars.Lasse Collin1-1/+2
2024-02-17liblzma/rangecoder: Exclude x32 from the x86-64 optimisation.Sebastian Andrzej Siewior1-1/+1
The x32 port has a x86-64 ABI in term of all registers but uses only 32bit pointer like x86-32. The assembly optimisation fails to compile on x32. Given the state of x32 I suggest to exclude it from the optimisation rather than trying to fix it. Signed-off-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
2024-02-17Translations: Update the Spanish translation.Jia Tan1-201/+226
2024-02-17Translations: Update the Swedish translation.Jia Tan1-204/+230
2024-02-17Translations: Update the Polish translation.Jia Tan1-200/+224
2024-02-17Translations: Update the Ukrainian translation.Jia Tan1-1/+1
2024-02-16Translations: Use the same sentence in xz.pot-header that the TP uses.Lasse Collin1-1/+1
2024-02-16Fix typos discovered by codespell.Jia Tan3-4/+4
2024-02-16Translations: Update the Ukrainian man page translations.Jia Tan1-837/+873
2024-02-16Translations: Update the Ukrainian translation.Jia Tan1-241/+225
2024-02-15Translations: Omit the generic copyright line from man page headers.Lasse Collin1-0/+1
2024-02-15Update m4/.gitignore.Jia Tan1-0/+1
2024-02-14Tests: tuktest.h: Treat Clang separately from GCC.Lasse Collin1-3/+3
Don't assume that Clang defines __GNUC__ as the extensions are available in clang-cl as well (and possibly in some other Clang variants?).
2024-02-14Tests: tuktest.h: Add a missing word to a comment.Lasse Collin1-2/+2
2024-02-14Tests: tuktest.h: Fix the comment about STest.Lasse Collin1-1/+2
2024-02-15Bump version for 5.5.2beta.larhzu/v5.5.2betaJia Tan3-4/+4
2024-02-14liblzma: Fix validate_map.sh.Lasse Collin1-1/+1
Adding the SPDX license identifier changed the line numbers.
2024-02-14Build: Start the generated ChangeLog from around 5.4.0 instead of 5.2.0.Lasse Collin1-1/+1
2024-02-14Fixed NEWS for 5.5.2beta.Lasse Collin1-2/+6
2024-02-14liblzma: Silence warnings in --enable-small build.Lasse Collin2-0/+3
2024-02-14Build: Install COPYING.0BSD as part of docs.Lasse Collin1-0/+1
2024-02-14Docs: List COPYING.0BSD in README.Lasse Collin1-0/+1
2024-02-14Docs: Include doc/examples/11_file_info.c in tarballs.Lasse Collin1-0/+1
It was added in 2017 in c2e29f06a7d1e3ba242ac2fafc69f5d6e92f62cd but it never got into any release tarballs because it was forgotten to be added to Makefile.am.
2024-02-14liblzma: Silence a warning.Lasse Collin1-1/+1
2024-02-14Add NEWS for 5.5.2beta.Lasse Collin1-0/+60
2024-02-14xz: Mention lzmainfo if trying to use 'lzma --list'.Lasse Collin1-2/+14
This kind of fixes the problem reported here: https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/1291020
2024-02-14liblzma: Add comments.Lasse Collin2-2/+18
2024-02-14Scripts: Add lz4 support to xzgrep and xzdiff.Lasse Collin4-10/+19
2024-02-14liblzma: Choose the range decoder variants using a bitmask macro.Lasse Collin1-11/+53
2024-02-14xz: Fix outdated threading related info on the man page.Lasse Collin1-8/+14
2024-02-14liblzma: Range decoder: Add x86-64 inline assembly.Lasse Collin1-0/+491
It's compatible with GCC and Clang.
2024-02-14liblzma: Range decoder: Add branchless C code.Lasse Collin1-0/+76
It's used only for basic bittrees and fixed-size reverse bittree because those showed a clear benefit on x86-64 with GCC and Clang. The other methods were more mixed and thus are commented out but they should be tested on other archs.
2024-02-14liblzma: Clarify a comment.Lasse Collin1-3/+6
2024-02-14liblzma: LZMA decoder: Optimize loop comparison.Lasse Collin2-4/+11
But now it needs one more local variable.
2024-02-14liblzma: Optimize literal_subcoder() macro slightly.Lasse Collin5-22/+24
2024-02-14liblzma: LZ decoder: Add unlikely().Lasse Collin1-1/+1
2024-02-14liblzma: LZ decoder: Remove a useless unlikely().Lasse Collin1-1/+1
2024-02-14liblzma: Optimize LZ decoder slightly.Lasse Collin3-60/+88
Now extra buffer space is reserved so that repeating bytes for any single match will never need to copy from two places (both the beginning and the end of the buffer). This simplifies dict_repeat() and helps a little with speed. This seems to reduce .lzma decompression time about 2 %, so with .xz and CRC it could be slightly less. The small things add up still.
2024-02-14liblzma: LZMA decoder: Get rid of next_state[].Lasse Collin3-24/+24
It's not completely obvious if this is better in the decoder. It should be good if compiler can avoid creating a branch (like using CMOV on x86). This also makes lzma_encoder.c use the new macros.
2024-02-14liblzma: LZMA decoder improvements.Lasse Collin3-200/+210
This adds macros for bittree decoding which prepares the code for alternative C versions and inline assembly.
2024-02-14liblzma: Creates Non-resumable and Resumable modes for lzma_decoder.Jia Tan2-213/+521
The new decoder resumes the first decoder loop in the Resumable mode. Then, the code executes in Non-resumable mode until it detects that it cannot guarantee to have enough input/output to decode another symbol. The Resumable mode is how the decoder has always worked. Before decoding every input bit, it checks if there is enough space and will save its location to be resumed later. When the decoder has more input/output, it jumps back to the correct sequence in the Resumable mode code. When the input/output buffers are large, the Resumable mode is much slower than the Non-resumable because it has more branches and is harder for the compiler to optimize since it is in a large switch block. Early benchmarking shows significant time improvement (8-10% on gcc and clang x86) by using the Non-resumable code as much as possible.
2024-02-14liblzma: Creates separate "safe" range decoder mode.Jia Tan2-103/+82
The new "safe" range decoder mode is the same as old range decoder, but now the default behavior of the range decoder will not check if there is enough input or output to complete the operation. When the buffers are close to fully consumed, the "safe" operations must be used instead. This will improve speed because it will reduce the number of branches needed for most of the range decoder operations.
2024-02-14doxygen/footer.html: Add missing closing tags and don't open a new tab.Lasse Collin1-2/+4
The footer template from Doxygen has the closing </body> </html> as Doxygen doesn't add them otherwise. target="_blank" was omitted as it's not useful here but it can be slightly annoying as one cannot just go back in the browser history. Since the footer links to the license file in the same directory and not to CC website, the rel attributes can be omitted.
2024-02-14Tweak the expressions in AUTHORS.Lasse Collin1-8/+23
2024-02-14Translations: Add the man page translators into man page header comment.Lasse Collin3-7/+26
It looked odd to only have the original English authors listed in the header comments of the translated files.
2024-02-14Translations: Translate also messages of lzmainfo.Lasse Collin1-0/+2
lzmainfo has had translation support since 2009 at least but it was never added to po/POTFILES.in so the messages weren't translated. It's a very rarely needed tool so it's not too bad. This also adds src/xz/mytime.c to po/POTFILES.in although there are no translatable strings. It's simpler this way so that it won't be forgotten if strings were ever added to that file.
2024-02-14Translations: Add custom .pot header with SPDX license identifier.Lasse Collin3-0/+16
The same is used for both po/xz.pot and po4a/xz-man.pot.
2024-02-14Translations: po4a/update-po: Add copyright notice to xz-man.pot.Lasse Collin1-1/+1
All man pages are under 0BSD now so this is simple now.
2024-02-14Update COPYING about the man pages of the scripts.Lasse Collin1-3/+3
2024-02-14xzdiff, xzgrep, and xzmore: Rewrite the man pages.Lasse Collin3-116/+173
The main reason is a kind of silly one: xz-man.pot contains strings from all man pages in XZ Utils. The man pages of xzdiff, xzgrep, and xzmore were under GPLv2 and the rest under 0BSD. Thus xz-man.pot contained strings under two licences. po4a creates the translated man pages from the combined 0BSD+GPLv2 xz-man.pot. I haven't liked this mixing in xz-man.pot but the Translation Project requires that all man pages must be in the same .pot file. So a separate xz-man-gpl.pot wasn't an option. Since these man pages are short, rewriting them was quick enough. Now xz-man.pot is entirely under 0BSD and marking the per-file licenses is simpler. As a bonus, some wording hopefully is now slightly better although it's perhaps a matter of taste. NOTE: In xzgrep.1, the EXIT STATUS section was written by me in the commit d796b6d7fdb8b7238b277056cf9146cce25db604 so that's why that section could be taken as is from the old xzgrep.1.
2024-02-14xzless: Update man page slightly.Lasse Collin1-4/+4
The xz tool can decompress three file formats and xzless has always supported uncompressed files too.
2024-02-14Translations: Change po/Makevars to add a copyright notice to po/xz.pot.Lasse Collin1-2/+2
2024-02-14Translations: Update po/Makevars to use the template from gettext 0.22.4.Lasse Collin1-5/+46
Also add SPDX license identifier now that there is a known license.
2024-02-14liblzma: Include the SPDX license identifier 0BSD to generated files.Lasse Collin11-26/+50
Perhaps the generated files aren't even copyrightable but using the same license for them as for the rest of the liblzma keeps things more consistent for tools that look for license info.
2024-02-14liblzma: Fix compilation of price_tablegen.c.Lasse Collin2-1/+9
It is built and run only manually so this didn't matter unless one wanted to regenerate the price_table.c.
2024-02-14Add SPDX license identifiers to GPL, LGPL, and FSFULLR files.Lasse Collin22-6/+37
2024-02-14Add SPDX license identifier into 0BSD source code files.Lasse Collin290-58/+588
2024-02-14liblzma: Sync the AUTHORS fix about SHA-256 to lzma.h.Lasse Collin1-6/+4
2024-02-14Change most public domain parts to 0BSD.Lasse Collin288-911/+100
Translations and doc/xz-file-format.txt and doc/lzma-file-format.txt were not touched. COPYING.0BSD was added.
2024-02-14Fix SHA-256 authors.Lasse Collin2-14/+6
The initial commit 5d018dc03549c1ee4958364712fb0c94e1bf2741 in 2007 had a comment in sha256.c that the code is based on Crypto++ Library 5.5.1. In 2009 the Authors list in sha256.c and the AUTHORS file was updated with information that the code had come from Crypto++ but via 7-Zip. I know I had viewed 7-Zip's SHA-256 code but back then the C code has been identical enough with Crypto++, so I don't why I thought the author info would need that extra step via 7-Zip for this single file. Another error is that I had mixed sha.* and shacal2.* files when checking for author info in Crypto++. The shacal2.* files aren't related to liblzma's sha256.c and thus Kevin Springle's code in Crypto++ isn't either.
2024-02-14Remove macosx/build.sh.Lasse Collin2-114/+0
It was last updated in 2013.
2024-02-14Doc: Remove doc/examples_old.Lasse Collin3-255/+0
It was good to keep these around in parallel with the newer examples but I think it's OK to remove the old ones at this point.
2024-02-13Tests: Add RISC-V filter support in a few places.Jia Tan2-0/+12
2024-02-13liblzma: Fix build error if only RISC-V BCJ filter is enabled.Jia Tan1-1/+3
If any other BCJ filter was enabled for encoding or decoding, then this was not a problem.
2024-02-13Translations: Update the Korean translation.Jia Tan1-242/+284
2024-02-13Translations: Update the Korean man page translations.Jia Tan1-605/+770
2024-02-13Translations: Update the Chinese (simplified) translation.Jia Tan1-156/+268
2024-02-09xzless: Use ||- in LESSOPEN with with "less" 451 and newer.Lasse Collin1-1/+8
2024-02-09xzless: Use --show-preproc-errors with "less" 632 and newer.Lasse Collin1-2/+9
This makes "less" show a warning if a decompression error occurred.
2024-02-09liblzma: Fix typo discovered by codespell.Jia Tan1-1/+1
2024-02-09Translations: Update the Swedish translation.Jia Tan1-166/+254
2024-02-08Translations: Update the Spanish translation.Jia Tan1-11/+11
2024-02-07Translations: Update the Spanish translation.Jia Tan1-166/+253
2024-02-07Translations: Update the Polish translation.Jia Tan1-162/+249
2024-02-07Translations: Update the German translation.Jia Tan1-154/+242
2024-02-07Translations: Update the German man page translations.Jia Tan1-601/+752
2024-02-06Translations: Update the Romanian translation.Jia Tan1-164/+252
2024-02-06Translations: Update the Romanian man page translations.Jia Tan1-793/+966
2024-02-07Translations: Update the Ukrainian translation.Jia Tan1-155/+242
2024-02-06Translations: Update the Ukrainian man page translations.Jia Tan1-599/+764
2024-02-02Update AUTHORS.Jia Tan1-1/+2
2024-02-02liblzma: Update Authors list in crc32_arm64.h.Jia Tan1-0/+1
2024-02-01liblzma: Check HAVE_USABLE_CLMUL before omitting CRC32 table.Jia Tan1-2/+2
This was split from the prior commit so it could be easily applied to the 5.4 branch. Closes: https://github.com/tukaani-project/xz/pull/77
2024-02-01liblzma: Check HAVE_USABLE_CLMUL before omitting CRC64 table.Jia Tan1-2/+2
If liblzma is configured with --disable-clmul-crc CFLAGS="-msse4.1 -mpclmul", then it will fail to compile because the generic version must be used but the CRC tables were not included.
2024-02-01liblzma: Only use ifunc in crcXX_fast.c if its needed.Jia Tan2-6/+6
The code was using HAVE_FUNC_ATTRIBUTE_IFUNC instead of CRC_USE_IFUNC. With ARM64, ifunc is incompatible because it requires non-inline function calls for runtime detection.
2024-02-01Docs: Add --disable-arm64-crc32 description to INSTALL.Jia Tan1-1/+11
2024-02-01liblzma: Omit CRC tables when not needed with ARM64 optimizations.Jia Tan3-5/+25
This is similar to the existing x86-64 CLMUL conditions to omit the tables. They were slightly refactored to improve readability.
2024-02-01liblzma: Rename crc32_aarch64.h to crc32_arm64.h.Jia Tan6-116/+122
Even though the proper name for the architecture is aarch64, this project uses ARM64 throughout. So the rename is for consistency. Additionally, crc32_arm64.h was slightly refactored for the following changes: * Added MSVC, FreeBSD, and macOS support in is_arch_extension_supported(). * crc32_arch_optimized() now checks the size when aligning the buffer. * crc32_arch_optimized() loop conditions were slightly modified to avoid both decrementing the size and incrementing the buffer pointer. * Use the intrinsic wrappers defined in <arm_acle.h> because GCC and Clang name them differently. * Minor spacing and comment changes.
2024-02-01liblzma: Refactor crc_common.h.Jia Tan3-42/+82
The CRC_GENERIC is now split into CRC32_GENERIC and CRC64_GENERIC, since the ARM64 optimizations will be different between CRC32 and CRC64. For the same reason, CRC_ARCH_OPTIMIZED is split into CRC32_ARCH_OPTIMIZED and CRC64_ARCH_OPTIMIZED. ifunc will only be used with x86-64 CLMUL because the runtime detection methods needed with ARM64 are not compatible with ifunc.
2024-02-01CMake: Add support for ARM64 CRC32 instruction detection.Jia Tan1-0/+50
2024-02-01Build: Add support for ARM64 CRC32 instruction detection.Jia Tan1-0/+52
This adds --enable-arm64-crc32/--disable-arm64-crc32 (enabled by default) for using the ARM64 CRC32 instruction. This can be disabled if one knows the binary will never need to run on an ARM64 machine with this instruction extension.
2024-01-27Speed up CRC32 calculation on ARM64Chenxi Mao6-9/+130
The CRC32 instructions in ARM64 can calculate the CRC32 result for 8 bytes in a single operation, making the use of ARM64 instructions much faster compared to the general CRC32 algorithm. Optimized CRC32 will be enabled if ARM64 has CRC extension running on Linux. Signed-off-by: Chenxi Mao <chenxi.mao2013@gmail.com>
2024-01-26Bump version number for 5.5.1alpha.larhzu/v5.5.1alphaJia Tan3-3/+3
2024-01-26Add NEWS for 5.5.1alphaJia Tan1-0/+80
2024-01-26Add NEWS for 5.4.6.Jia Tan1-0/+22
2024-01-24Move doc/logo/xz-logo.png to "doc" and Doxygen footer to "doxygen".Lasse Collin4-3/+3
The footer isn't a complete HTML file so having it in the doxygen directory is a tiny bit clearer.
2024-01-25README: Add COPYING.CC-BY-SA-4.0 entry to section 1.1.Jia Tan1-18/+20
The Overall documentation section (1.1) table spacing had to be adjusted since the filename was very long.
2024-01-25Build: Add the logo and license to the release.Jia Tan1-0/+2
2024-01-25COPYING: Add the license for the XZ logo.Jia Tan2-0/+432
2024-01-25Doxygen: Added the XZ logo and copyright information.Jia Tan3-3/+14
The PROJECT_LOGO field is now used to include the XZ logo. The footer of each page now lists the copyright information instead of the default footer. The license is also copied to statisfy the copyright and so the link in the documentation can be local.
2024-01-23xz: Use threaded mode by defaut (as if --threads=0 was used).Lasse Collin3-3/+16
This hopefully does more good than bad: + It's faster by default. + Only the threaded compressor creates files that can be decompressed in threaded mode. - Compression ratio is worse, usually not too much though. When it matters, -T1 must be used. - Memory usage increases. - Scripts that assume single-threaded mode but don't use -T1 will possibly use too much resources, for example, if they run multiple xz processes in parallel to compress multiple files. - Output from single-threaded and multi-threaded compressors differ but such changes could happen for other reasons too (they just haven't happened since 5.0.0).
2024-01-23CI: Use RISC-V filter when building with BCJ support.Jia Tan1-2/+2
2024-01-23Tests: Use smaller dictionary size in RISC-V test files.Jia Tan2-0/+0
2024-01-23Tests: Skip RISC-V test files if decoder was not built.Jia Tan1-0/+5
2024-01-23xz: Man page: Add more examples of LZMA2 options with BCJ filters.Lasse Collin1-7/+31
2024-01-23liblzma: RISC-V filter: Use byte-by-byte access.Lasse Collin1-30/+84
Not all RISC-V processors support fast unaligned access so it's better to read only one byte in the main loop. This can be faster even on x86-64 when compared to reading 32 bits at a time as half the time the address is only 16-bit aligned. The downside is larger code size on archs that do support fast unaligned access.
2024-01-23xz: Update xz -lvv for RISC-V filter.Jia Tan1-0/+10
Version 5.6.0 will be shown, even though upcoming alphas and betas will be able to support this filter. 5.6.0 looks nicer in the output and people shouldn't be encouraged to use an unstable version in production in any way.
2024-01-23Tests: Add two RISC-V Filter test files.Jia Tan3-0/+8
These test files achieve 100% code coverage in src/liblzma/simple/riscv.c. They contain all of the instructions that should be filtered and a few cases that should not.
2024-01-23xz: Update message in --long-help for RISC-V Filter.Jia Tan1-0/+1
2024-01-23xz: Update the man page for the RISC-V Filter.Jia Tan1-1/+2
A special note was added to suggest using four-byte alignment when the compressed instruction extension is not present in a RISC-V binary.
2024-01-23Tests: Add RISC-V Filter test in test_compress.sh.Jia Tan1-0/+1
2024-01-23liblzma: Update string_conversion.c to support RISC-V Filter.Jia Tan1-0/+5
2024-01-23CMake: Support RISC-V BCJ Filter for encoding and decoding.Jia Tan1-0/+1
2024-01-23liblzma: Add RISC-V BCJ filter.Jia Tan9-2/+742
The new Filter ID is 0x0B. Thanks to Chien Wong <m@xv97.com> for the initial version of the Filter, the xz CLI updates, and the Autotools build system modifications. Thanks to Igor Pavlov for his many contributions to the design of the filter.
2024-01-19Docs: Update .xz file format specification to 1.2.0.Jia Tan1-12/+17
The new RISC-V filter was added to the specification, in addition to updating the specification URL.
2024-01-19xz: Update website URLs in the man pages.Jia Tan2-5/+5
2024-01-19liblzma: Update website URL.Jia Tan2-4/+4
2024-01-19Docs: Update website URLs.Jia Tan6-15/+17
2024-01-19Build: Update website URL.Jia Tan2-2/+2
2024-01-11liblzma: CRC: Add a comment to crc_x86_clmul.h about BUILDING_ macros.Lasse Collin1-0/+6
2024-01-11liblzma: CRC: Remove crc_always_inline, use lzma_always_inline instead.Lasse Collin2-21/+1
Now crc_simd_body() in crc_x86_clmul.h is only called once in a translation unit, we no longer need to be so cautious about ensuring the always-inline behavior.
2024-01-11liblzma: CRC: Update CLMUL comments to more generic wording.Lasse Collin2-13/+13
2024-01-11liblzma: Rename arch-specific CRC functions and macros.Lasse Collin4-25/+31
CRC_CLMUL was split to CRC_ARCH_OPTIMIZED and CRC_X86_CLMUL. CRC_ARCH_OPTIMIZED is defined when an arch-optimized version is used. Currently the x86 CLMUL implementations are the only arch-optimized versions, and these also use the CRC_x86_CLMUL macro to tell when crc_x86_clmul.h needs to be included. is_clmul_supported() was renamed to is_arch_extension_supported(). crc32_clmul() and crc64_clmul() were renamed to crc32_arch_optimized() and crc64_arch_optimized(). This way the names make sense with arch-specific non-CLMUL implementations as well.
2024-01-11liblzma: Fix a comment in crc_common.h.Lasse Collin1-1/+2
2024-01-11liblzma: Avoid extern lzma_crc32_clmul() and lzma_crc64_clmul().Lasse Collin7-91/+91
A CLMUL-only build will have the crcxx_clmul() inlined into lzma_crcxx(). Previously a jump to the extern lzma_crcxx_clmul() was needed. Notes about shared liblzma on ELF platforms: - On platforms that support ifunc and -fvisibility=hidden, this was silly because CLMUL-only build would have that single extra jump instruction of extra overhead. - On platforms that support neither -fvisibility=hidden nor linker version script (liblzma*.map), jumping to lzma_crcxx_clmul() would go via PLT so a few more instructions of overhead (still not a big issue but silly nevertheless). There was a downside with static liblzma too: if an application only needs lzma_crc64(), static linking would make the linker include the CLMUL code for both CRC32 and CRC64 from crc_x86_clmul.o even though the CRC32 code wouldn't be needed, thus increasing code size of the executable (assuming that -ffunction-sections isn't used). Also, now compilers are likely to inline crc_simd_body() even if they don't support the always_inline attribute (or MSVC's __forceinline). Quite possibly all compilers that build the code do support such an attribute. But now it likely isn't a problem even if the attribute wasn't supported. Now all x86-specific stuff is in crc_x86_clmul.h. If other archs The other archs can then have their own headers with their own is_clmul_supported() and crcxx_clmul(). Another bonus is that the build system doesn't need to care if crc_clmul.c is needed. is_clmul_supported() stays as inline function as it's not needed when doing a CLMUL-only build (avoids a warning about unused function).
2024-01-11liblzma: crc_clmul.c: Add crc_attr_target macro.Lasse Collin1-14/+16
This reduces the number of the complex #if directives.
2024-01-11liblzma: Simplify existing cases with lzma_attr_no_sanitize_address.Lasse Collin1-9/+3
2024-01-11liblzma: #define crc_attr_no_sanitize_address in crc_common.h.Lasse Collin1-0/+10
2024-01-10liblzma: CRC: Add empty lines.Lasse Collin3-1/+5
And remove one too.
2024-01-10liblzma: crc_clmul.c: Tidy up the location of MSVC pragma.Lasse Collin1-2/+2
It makes no difference in practice.
2023-12-28Update THANKS.Lasse Collin1-0/+1
2023-12-28liblzma: Use 8-byte method in memcmplen.h on ARM64.Lasse Collin1-8/+10
It requires fast unaligned access to 64-bit integers and a fast instruction to count leading zeros in a 64-bit integer (__builtin_ctzll()). This perhaps should be enabled on some other archs too. Thanks to Chenxi Mao for the original patch: https://github.com/tukaani-project/xz/pull/75 (the first commit) According to the numbers there, this may improve encoding speed by about 3-5 %. This enables the 8-byte method on MSVC ARM64 too which should work but wasn't tested.
2023-12-28liblzma: Check also for __clang__ in memcmplen.h.Lasse Collin1-1/+2
This change hopefully makes no practical difference as Clang likely was detected via __GNUC__ or _MSC_VER already.
2023-12-21Translations: Update the French translation.Jia Tan1-262/+370
2023-12-21xz: Add a comment to Capsicum sandbox setup.Jia Tan1-0/+1
This comment is repeated in xzdec.c to help remind us why all the capabilities are removed from stdin in certain situations.
2023-12-21Docs: Update --enable-sandbox option in INSTALL.Jia Tan1-7/+10
xzdec now also uses the sandbox when its configured.
2023-12-21CMake: Move sandbox detection outside of xz section.Jia Tan1-80/+98
The sandbox is now enabled for xzdec as well, so it no longer belongs in just the xz section. xz and xzdec are always built, except for older MSVC versions, so there isn't a need to conditionally show the sandbox configuration. CMake will do a little unecessary work on older MSVC versions that can't build xz or xzdec, but this is a very small downside.
2023-12-20Build: Allow sandbox to be configured for just xzdec.Jia Tan1-5/+5
If xz is disabled, then xzdec can still use the sandbox.
2023-12-19xzdec: Add sandbox support for Pledge, Capsicum, and Landlock.Jia Tan1-7/+139
A very strict sandbox is used when the last file is decompressed. The likely most common use case of xzdec is to decompress a single file. The Pledge sandbox is applied to the entire process with slightly more relaxed promises, until the last file is processed. Thanks to Christian Weisgerber for the initial patch adding Pledge sandboxing.
2023-12-20liblzma: Initialize lzma_lz_encoder pointers with NULL.Jia Tan1-1/+5
This fixes the recent change to lzma_lz_encoder that used memzero instead of the NULL constant. On some compilers the NULL constant (always 0) may not equal the NULL pointer (this only needs to guarentee to not point to valid memory address). Later code compares the pointers to the NULL pointer so we must initialize them with the NULL pointer instead of 0 to guarentee code correctness.
2023-12-16liblzma: Set all values in lzma_lz_encoder to NULL after allocation.Jia Tan1-3/+1
The first member of lzma_lz_encoder doesn't necessarily need to be set to NULL since it will always be set before anything tries to use it. However the function pointer members must be set to NULL since other functions rely on this NULL value to determine if this behavior is supported or not. This fixes a somewhat serious bug, where the options_update() and set_out_limit() function pointers are not set to NULL. This seems to have been forgotten since these function pointers were added many years after the original two (code() and end()). The problem is that by not setting this to NULL we are relying on the memory allocation to zero things out if lzma_filters_update() is called on a LZMA1 encoder. The function pointer for set_out_limit() is less serious because there is not an API function that could call this in an incorrect way. set_out_limit() is only called by the MicroLZMA encoder, which must use LZMA1 where set_out_limit() is always set. Its currently not possible to call set_out_limit() on an LZMA2 encoder at this time. So calling lzma_filters_update() on an LZMA1 encoder had undefined behavior since its possible that memory could be manipulated so the options_update member pointed to a different instruction sequence. This is unlikely to be a bug in an existing application since it relies on calling lzma_filters_update() on an LZMA1 encoder in the first place. For instance, it does not affect xz because lzma_filters_update() can only be used when encoding to the .xz format. This is fixed by using memzero() to set all members of lzma_lz_encoder to NULL after it is allocated. This ensures this mistake will not occur here in the future if any additional function pointers are added.
2023-12-16liblzma: Tweak a comment.Jia Tan1-1/+1
2023-12-16liblzma: Make parameter names in function definition match declaration.Jia Tan1-4/+4
lzma_raw_encoder() and lzma_raw_encoder_init() used "options" as the parameter name instead of "filters" (used by the declaration). "filters" is more clear since the parameter represents the list of filters passed to the raw encoder, each of which contains filter options.
2023-12-16liblzma: Improve lzma encoder init function consistency.Jia Tan1-0/+3
lzma_encoder_init() did not check for NULL options, but lzma2_encoder_init() did. This is more of a code style improvement than anything else to help make lzma_encoder_init() and lzma2_encoder_init() more similar.
2023-12-16Docs: Update repository URL in Changelog.Jia Tan1-1/+1
2023-12-15CI: Update Upload Artifact Action.Jia Tan2-2/+2
2023-12-07Tests: Silence -Wsign-conversion warning on GCC version < 10.Jia Tan1-1/+1
Since GCC version 10, GCC no longer complains about simple implicit integer conversions with Arithmetic operators. For instance: uint8_t a = 5; uint32_t b = a + 5; Give a warning on GCC 9 and earlier but this: uint8_t a = 5; uint32_t b = (a + 5) * 2; Gives a warning with GCC 10+.
2023-12-07Update THANKS.Jia Tan1-0/+1
2023-12-07Tests: Minor cleanups to OSS-Fuzz files.Jia Tan7-56/+66
Most of these fixes are small typos and tweaks. A few were caused by bad advice from me. Here is the summary of what is changed: - Author line edits - Small comment changes/additions - Using the return value in the error messages in the fuzz targets' coder initialization code - Removed fuzz_encode_stream.options. This set a max length, which may prevent some worthwhile code paths from being properly exercised. - Removed the max_len option from fuzz_decode_stream.options for the same reason as fuzz_encode_stream. The alone decoder fuzz target still has this restriction. - Altered the dictionary contents for fuzz_lzma.dict. Instead of keeping the properties static and varying the dictionary size, the properties are varied and the dictionary size is kept small. The dictionary size doesn't have much impact on the code paths but the properties do. Closes: https://github.com/tukaani-project/xz/pull/73
2023-12-07Tests: Add fuzz_encode_stream ossfuzz target.Maksym Vatsyk2-0/+81
This fuzz target handles .xz stream encoding. The first byte of input is used to dynamically set the preset level in order to increase the fuzz coverage of complex critical code paths.
2023-12-07Tests: Add fuzz_decode_alone OSS-Fuzz targetMaksym Vatsyk3-0/+66
This fuzz target that handles LZMA alone decoding. A new fuzz dictionary .dict was also created with common LZMA header values to help speed up the discovery of valid headers.
2023-12-07Tests: Update OSS-Fuzz Makefile.Maksym Vatsyk1-4/+9
All .c files can be built as separate fuzz targets. This simplifies the Makefile by allowing us to use wildcards instead of having a Makefile target for each fuzz target.
2023-12-07Tests: Move common OSS-Fuzz target code to .h file.Maksym Vatsyk2-44/+71
2023-12-07Tests: Rename OSS-Fuzz files.Maksym Vatsyk4-2/+3
2023-11-30Update THANKS.Jia Tan1-0/+1
2023-11-30Tests: Fix typosKian-Meng Ang2-3/+3
2023-11-30xz: Fix typoKian-Meng Ang1-1/+1
2023-11-30Update THANKS.Jia Tan1-0/+1
2023-11-30CI: Test musl libc builds on Ubuntu runner.Jia Tan1-2/+17
2023-11-30CI: Allow ci_build.sh to set a different C compiler.Jia Tan1-1/+10
2023-11-30CMake: Use consistent indentation with check_c_source_compiles().Jia Tan1-2/+2
2023-11-30CMake: Change __attribute__((__ifunc__())) detection.Jia Tan1-8/+45
This renames ALLOW_ATTR_IFUNC to USE_ATTR_IFUNC and applies the ifunc detection changes that were made to the Autotools build. Fixes: https://github.com/tukaani-project/xz/issues/70
2023-11-30Docs: Update INSTALL for --enable_ifunc change.Jia Tan1-8/+8
2023-11-30Build: Change --enable-ifunc handling.Jia Tan1-17/+44
Some compilers support __attribute__((__ifunc__())) even though the dynamic linker does not. The compiler is able to create the binary but it will fail on startup. So it is not enough to just test if the attribute is supported. The default value for enable_ifunc is now auto, which will attempt to compile a program using __attribute__((__ifunc__())). There are additional checks in this program if glibc is being used or if it is running on FreeBSD. Setting --enable-ifunc will skip this test and always enable __attribute__((__ifunc__())), even if is not supported.
2023-11-23xz: Tweak a comment.Lasse Collin1-2/+2
2023-11-23xz: Use is_tty() in message.c.Jia Tan1-6/+1
2023-11-23xz: Create separate is_tty() function.Jia Tan2-7/+37
The new is_tty() will report if a file descriptor is a terminal or not. On POSIX systems, it is a wrapper around isatty(). However, the native Windows implementation of isatty() will return true for all character devices, not just terminals. So is_tty() has a special case for Windows so it can use alternative Windows API functions to determine if a file descriptor is a terminal. This fixes a bug with MSVC and MinGW-w64 builds that refused to read from or write to non-terminal character devices because xz thought it was a terminal. For instance: xz foo -c > /dev/null would fail because /dev/null was assumed to be a terminal.
2023-11-22tuklib_integer: Fix typo discovered by codespell.Jia Tan1-1/+1
Based on internet dictionary searches, 'choise' is an outdated spelling of 'choice'.
2023-11-18xz: Move the check for --suffix with --format=raw a few lines earlier.Lasse Collin1-22/+22
Now it reads from argv[] instead of args->arg_names.
2023-11-18Tests: Create test_suffix.sh.Jia Tan2-0/+191
This tests some complicated interactions with the --suffix= option. The suffix option must be used with --format=raw, but can optionally be used to override the default .xz suffix. This test also verifies some recent bugs have been correctly solved and to hopefully avoid further regressions in the future.
2023-11-17xz: Fix a bug with --files and --files0 in raw mode without a suffix.Jia Tan1-0/+5
The following command caused a segmentation fault: xz -Fraw --lzma1 --files=foo when foo was a valid file. The usage of --files or --files0 was not being checked when compressing or decompressing in raw mode without a suffix. The suffix checking code was meant to validate that all files to be processed are "-" (if not writing to standard out), meaning the data is only coming from standard in. In this case, there were no file names to check since --files and --files0 store their file name in a different place. Later code assumed the suffix was set and caused a segmentation fault. Now, the above command results in an error.
2023-11-17Tests: Fix typo in a comment.Jia Tan1-1/+1
2023-11-15xz: Refactor suffix test with raw format.Jia Tan1-25/+13
The previous version set opt_stdout, but this caused an issue with copying an input file to standard out when decompressing an unknown file type. The following needs to result in an error: echo foo | xz -df since -c, --stdout is not used. This fixes the previous error by not setting opt_stdout.
2023-11-14xz: Move suffix check after stdout mode is detected.Jia Tan1-8/+8
This fixes a bug introduced in cc5aa9ab138beeecaee5a1e81197591893ee9ca0 when the suffix check was initially moved. This caused a situation that previously worked: echo foo | xz -Fraw --lzma1 | wc -c to fail because the old code knew that this would write to standard out so a suffix was not needed.
2023-11-14xz: Detect when all data will be written to standard out earlier.Jia Tan1-0/+21
If the -c, --stdout argument is not used, then we can still detect when the data will be written to standard out if all of the provided filenames are "-" (denoting standard in) or if no filenames are provided.
2023-11-09liblzma: Add missing comments to lz_encoder.h.Jia Tan1-1/+5
2023-11-01Add NEWS for 5.4.5.Jia Tan1-0/+74
2023-10-31liblzma: Fix compilation of fastpos_tablegen.c.Lasse Collin1-0/+2
The macro lzma_attr_visibility_hidden has to be defined to make fastpos.h usable. The visibility attribute is irrelevant to fastpos_tablegen.c so simply #define the macro to an empty value. fastpos_tablegen.c is never built by the included build systems and so the problem wasn't noticed earlier. It's just a standalone program for generating fastpos_table.c. Fixes: https://github.com/tukaani-project/xz/pull/69 Thanks to GitHub user Jamaika1.
2023-10-31Build: Fix text wrapping in an output message.Jia Tan1-4/+5
2023-10-30liblzma: Add a note why crc_always_inline exists for now.Lasse Collin1-0/+5
Solaris Studio is a possible example (not tested) which supports the always_inline attribute but might not get detected by the common.h #ifdefs.
2023-10-30liblzma: Use lzma_always_inline in memcmplen.h.Lasse Collin1-2/+1
2023-10-30liblzma: #define lzma_always_inline in common.h.Lasse Collin1-0/+17
2023-10-30liblzma: Use lzma_attr_visibility_hidden on private extern declarations.Lasse Collin5-0/+13
These variables are internal to liblzma and not exposed in the API.
2023-10-30liblzma: #define lzma_attr_visibility_hidden in common.h.Lasse Collin1-0/+11
In ELF shared libs: -fvisibility=hidden affects definitions of symbols but not declarations.[*] This doesn't affect direct calls to functions inside liblzma as a linker can replace a call to lzma_foo@plt with a call directly to lzma_foo when -fvisibility=hidden is used. [*] It has to be like this because otherwise every installed header file would need to explictly set the symbol visibility to default. When accessing extern variables that aren't defined in the same translation unit, compiler assumes that the variable has the default visibility and thus indirection is needed. Unlike function calls, linker cannot optimize this. Using __attribute__((__visibility__("hidden"))) with the extern variable declarations tells the compiler that indirection isn't needed because the definition is in the same shared library. About 15+ years ago, someone told me that it would be good if the CRC tables would be defined in the same translation unit as the C code of the CRC functions. While I understood that it could help a tiny amount, I didn't want to change the code because a separate translation unit for the CRC tables was needed for the x86 assembly code anyway. But when visibility attributes are supported, simply marking the extern declaration with the hidden attribute will get identical result. When there are only a few affected variables, this is trivial to do. I wish I had understood this back then already.
2023-10-26liblzma: Refer to MinGW-w64 instead of MinGW in the API headers.Lasse Collin2-3/+3
MinGW (formely a MinGW.org Project, later the MinGW.OSDN Project at <https://osdn.net/projects/mingw/>) has GCC 9.2.0 as the most recent GCC package (released 2021-02-02). The project might still be alive but majority of people have switched to MinGW-w64. Thus it seems clearer to refer to MinGW-w64 in our API headers too. Building with MinGW is likely to still work but I haven't tested it in the recent years.
2023-10-26CMake: Use -D_FILE_OFFSET_BITS=64 if (and only if) needed.Lasse Collin2-1/+58
A CMake option LARGE_FILE_SUPPORT is created if and only if -D_FILE_OFFSET_BITS=64 affects sizeof(off_t). This is needed on many 32-bit platforms and even with 64-bit builds with MinGW-w64 to get support for files larger than 2 GiB.
2023-10-26CMake: Generate and install liblzma.pc if not using MSVC.Lasse Collin1-0/+21
Autotools based build uses -pthread and thus adds it to Libs.private in liblzma.pc. CMake doesn't use -pthread at all if pthread functions are available in libc so Libs.private doesn't get -pthread either.
2023-10-26CMake: Rearrange the PACKAGE_ variables.Lasse Collin1-11/+15
The windres workaround now replaces spaces with \x20 so the package name isn't repeated. These changes will help with creation of liblzma.pc.
2023-10-26liblzma: Add Cflags.private to liblzma.pc.in for MSYS2.Lasse Collin1-0/+1
It properly adds -DLZMA_API_STATIC when compiling code that will be linked against static liblzma. Having it there on systems other than Windows does no harm. See: https://www.msys2.org/docs/pkgconfig/
2023-10-26CMake: Create liblzma.def when building liblzma.dll with MinGW-w64.Lasse Collin2-0/+46
2023-10-26CMake: Change one CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_LIST_DIR.Lasse Collin1-1/+1
In this case they have identical values.
2023-10-26CMake/Windows: Fix the import library filename.Lasse Collin1-0/+1
Both PREFIX and IMPORT_PERFIX have to be set to "" to get liblzma.dll and liblzma.dll.a.
2023-10-25Build: Detect -fsanitize= in CFLAGS and incompatible build options.Lasse Collin2-4/+62
Now configure will fail if -fsanitize= is found in CFLAGS and sanitizer-incompatible ifunc or Landlock sandboxing would be used. These are incompatible with one or more sanitizers. It's simpler to reject all -fsanitize= uses instead of trying to pass those that might not cause problems. CMake-based build was updated similarly. It lets the configuration finish (SEND_ERROR instead of FATAL_ERROR) so that both error messages can be seen at once.
2023-10-24CI: Disable sandboxing in fsanitize=address,undefined job.Jia Tan1-2/+6
The sandboxing on Linux now supports Landlock, which restricts all supported filesystem actions after xz opens the files it needs. The sandbox is only enabled when one file is input and we are writing to standard out. With fsanitize=address,undefined, the instrumentation needs to read additional files after the sandbox is in place. This forces all xz based test to fail, so the sandbox must instead be disabled.
2023-10-24CI: Allow disabling the sandbox in ci_build.sh.Jia Tan1-1/+4
2023-10-22CMake: Don't shadow the cache entry ENABLE_THREADS with a normal variable.Lasse Collin1-3/+7
Using set(ENABLE_THREADS "posix") is confusing because it sets a new normal variable and leaves the cache entry with the same name unchanged. The intent wasn't to change the cache entry so this switches to a different variable name.
2023-10-22Docs: Update INSTALL about sandboxing support.Lasse Collin1-1/+6
2023-10-22xz: Support basic sandboxing with Linux Landlock (ABI versions 1-3).Lasse Collin5-5/+98
It is enabled only when decompressing one file to stdout, similar to how Capsicum is used. Landlock was added in Linux 5.13.
2023-10-22CMake: Edit threading related messages.Lasse Collin1-9/+10
It's mostly to change from "thread method" to "threading method".
2023-10-22CMake: Use FATAL_ERROR if user-supplied options aren't understood.Lasse Collin1-14/+14
This way typos are caught quickly and compounding error messages are avoided (a single typo could cause more than one error). This keeps using SEND_ERROR when the system is lacking a feature (like threading library or sandboxing method). This way the whole configuration log will be generated in case someone wishes to report a problem upstream.