aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2022-03-07liblzma: Add threaded .xz decompressor.Lasse Collin5-7/+1907
I realize that this is about a decade late. Big thanks to Sebastian Andrzej Siewior for the original patch. I made a bunch of smaller changes but after a while quite a few things got rewritten. So any bugs in the commit were created by me.
2022-03-06liblzma: Fix docs: lzma_block_decoder() cannot return LZMA_UNSUPPORTED_CHECK.Lasse Collin1-3/+0
If Check is unsupported, it will be silently ignored. It's the caller's job to handle it.
2022-03-06liblzma: Add new output queue (lzma_outq) features.Lasse Collin2-8/+79
Add lzma_outq_clear_cache2() which may leave one buffer allocated in the cache. Add lzma_outq_outbuf_memusage() to get the memory needed for a single lzma_outbuf. This is now used internally in outqueue.c too. Track both the total amount of memory allocated and the amount of memory that is in active use (not in cache). In lzma_outbuf, allow storing the current input position that matches the current output position. This way the main thread can notice when no more output is possible without first providing more input. Allow specifying return code for lzma_outq_read() in a finished lzma_outbuf.
2022-03-06liblzma: Index hash: Change return value type of hash_append() to void.Lasse Collin1-6/+5
2022-02-22liblzma: Minor addition to lzma_vli_size() API doc.Lasse Collin1-0/+2
Thanks to Jia Tan.
2022-02-22liblzma: Check the return value of lzma_index_append() in threaded encoder.Lasse Collin1-2/+5
If lzma_index_append() failed (most likely memory allocation failure) it could have gone unnoticed and the resulting .xz file would have an incorrect Index. Decompressing such a file would produce the correct uncompressed data but then an error would occur when verifying the Index field.
2022-02-22Update THANKS.Lasse Collin1-0/+1
2022-02-22liblzma: Use non-executable stack on FreeBSD as on LinuxEd Maste2-4/+4
2022-02-20liblzma: Make Block decoder catch certain types of errors better.Lasse Collin1-25/+54
Now it limits the input and output buffer sizes that are passed to a raw decoder. This way there's no need to check if the sizes can grow too big or overflow when updating Compressed Size and Uncompressed Size counts. This also means that a corrupt file cannot cause the raw decoder to process useless extra input or output that would exceed the size info in Block Header (and thus cause LZMA_DATA_ERROR anyway). More importantly, now the size information is verified more carefully in case raw decoder returns LZMA_OK. This doesn't really matter with the current single-threaded .xz decoder as the errors would be detected slightly later anyway. But this helps avoiding corner cases in the upcoming threaded decompressor, and it might help other Block decoder uses outside liblzma too. The test files bad-1-lzma2-{9,10,11}.xz test these conditions. With the single-threaded .xz decoder the only difference is that LZMA_DATA_ERROR is detected in a difference place now.
2022-02-20Tests: Add bad-1-lzma2-11.xz.Lasse Collin2-0/+5
2022-02-18Translations: Fix po4a failure with the French man page translations.Lasse Collin1-2/+4
Thanks to Mario Blättermann for the patch.
2022-02-07Translations: Add French translation of man pages.Lasse Collin2-1/+3542
This matches xz-utils 5.2.5-2 in Debian. The translation was done by "bubu", proofread by the debian-l10n-french mailing list contributors, and submitted to me on the xz-devel mailing list by Jean-Pierre Giraud. Thanks to everyone!
2022-02-07liblzma: Add NULL checks to LZMA and LZMA2 properties encoders.jiat752-0/+6
Previously lzma_lzma_props_encode() and lzma_lzma2_props_encode() assumed that the options pointers must be non-NULL because the with these filters the API says it must never be NULL. It is good to do these checks anyway.
2022-02-06liblzma: Fix uint64_t vs. size_t confusion.Lasse Collin1-4/+7
This broke 32-bit builds due to a pointer type mismatch. This bug was introduced with the output-size-limited encoding in 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. Thanks to huangqinjin for the bug report.
2022-02-06CMake: Keep compatible with Windows 95 for 32-bit build.huangqinjin1-1/+11
2021-11-13xzgrep: Update man page timestamp.Lasse Collin1-1/+1
2021-11-13Update THANKS.Lasse Collin1-0/+1
2021-11-13xzgrep: use `grep -E/-F` instead of `egrep` and `fgrep`Ville Skyttä2-6/+6
`egrep` and `fgrep` have been deprecated in GNU grep since 2007, and in current post 3.7 Git they have been made to emit obsolescence warnings: https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a9515624709865d480e3142fd959bccd1c9372d1
2021-10-28Bump the version number for 5.3.2alpha.larhzu/v5.3.2alphaLasse Collin2-2/+2
2021-10-28Add NEWS for 5.3.2alpha.Lasse Collin1-0/+86
2021-10-27Update THANKS.Lasse Collin1-0/+1
2021-10-27xz: Change the coding style of the previous commit.Lasse Collin1-5/+6
It isn't any better now but it's consistent with the rest of the code base.
2021-10-27xz: Avoid fchown(2) failure.Alexander Bluhm1-1/+7
OpenBSD does not allow to change the group of a file if the user does not belong to this group. In contrast to Linux, OpenBSD also fails if the new group is the same as the old one. Do not call fchown(2) in this case, it would change nothing anyway. This fixes an issue with Perl Alien::Build module. https://github.com/PerlAlien/Alien-Build/issues/62
2021-09-17liblzma: Fix liblzma.map for the lzma_microlzma_* symbols.Lasse Collin1-2/+2
This should have been part of d267d109c370a40b502e73f8664b154b15e4f253. Thanks to Gao Xiang.
2021-09-09Update THANKS.Lasse Collin1-0/+1
2021-09-09liblzma: Use _MSVC_LANG to detect when "noexcept" can be used with MSVC.Lasse Collin1-1/+2
By default, MSVC always sets __cplusplus to 199711L. The real C++ standard version is available in _MSVC_LANG (or one could use /Zc:__cplusplus to set __cplusplus correctly). Fixes <https://sourceforge.net/p/lzmautils/discussion/708858/thread/f6bc3b108a/>. Thanks to Dan Weiss.
2021-09-05liblzma: Rename EROFS LZMA to MicroLZMA.Lasse Collin4-47/+52
It still exists primarily for EROFS but MicroLZMA is a more generic name (that hopefully doesn't clash with something that already exists).
2021-06-04xzdiff: Update the man page about the exit status.Lasse Collin1-2/+2
This was forgotten from 194029ffaf74282a81f0c299c07f73caca3232ca.
2021-06-04xzless: Fix less(1) version detection when it contains a dot.Lasse Collin1-1/+1
Sometimes the version number from "less -V" contains a dot, sometimes not. xzless failed detect the version number when it does contain a dot. This fixes it. Thanks to nick87720z for reporting this. Apparently it had been reported here <https://bugs.gentoo.org/489362> in 2013.
2021-04-11Update THANKS.Lasse Collin1-0/+2
2021-04-11Reduce maximum possible memory limit on MIPS32Ivan A. Melnikov1-0/+6
Due to architectural limitations, address space available to a single userspace process on MIPS32 is limited to 2 GiB, not 4, even on systems that have more physical RAM -- e.g. 64-bit systems with 32-bit userspace, or systems that use XPA (an extension similar to x86's PAE). So, for MIPS32, we have to impose stronger memory limits. I've chosen 2000MiB to give the process some headroom.
2021-02-13CMake: Use interface library for better FindLibLZMA compatibility.Lasse Collin1-2/+9
https://www.mail-archive.com/xz-devel@tukaani.org/msg00446.html Thanks to Markus Rickert.
2021-01-30CMake: Try to improve compatibility with the FindLibLZMA module.Lasse Collin1-8/+13
The naming conflict with FindLibLZMA module gets worse. Not avoiding it in the first place was stupid. Normally find_package(LibLZMA) will use the module and find_package(liblzma 5.2.5 REQUIRED CONFIG) will use the config file even with a case insensitive file system. However, if CMAKE_FIND_PACKAGE_PREFER_CONFIG is TRUE and the file system is case insensitive, find_package(LibLZMA) will find our liblzma config file instead of using FindLibLZMA module. One big problem with this is that FindLibLZMA uses LibLZMA::LibLZMA and we use liblzma::liblzma as the target name. With target names CMake happens to be case sensitive. To workaround this, this commit adds add_library(LibLZMA::LibLZMA ALIAS liblzma::liblzma) to the config file. Then both spellings work. To make the behavior consistent between case sensitive and insensitive file systems, the config and related files are renamed from liblzmaConfig.cmake to liblzma-config.cmake style. With this style CMake looks for lowercase version of the package name so find_package(LiBLzmA 5.2.5 REQUIRED CONFIG) will work to find our config file. There are other differences between our config file and FindLibLZMA so it's still possible that things break for reasons other than the spelling of the target name. Hopefully those situations aren't too common. When the config file is available, it should always give as good or better results as FindLibLZMA so this commit doesn't affect the recommendation to use find_package(liblzma 5.2.5 REQUIRED CONFIG) which explicitly avoids FindLibLZMA. Thanks to Markus Rickert.
2021-01-29Update THANKS.Lasse Collin1-0/+1
2021-01-29liblzma: Fix unitialized variable.Lasse Collin1-0/+1
This was introduced two weeks ago in the commit 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. Thanks to Nathan Moinvaziri.
2021-01-24Tests: Add bad-1-lzma2-10.xz and also modify -9.xz.Lasse Collin3-2/+9
2021-01-24liblzma: Fix a wrong comment in stream_encoder_mt.c.Lasse Collin1-3/+7
2021-01-24Tests: Add bad-1-lzma2-9.xz.Lasse Collin2-0/+4
2021-01-24Tests: Add bad-1-check-crc32-2.xz.Lasse Collin2-0/+7
2021-01-17liblzma: In EROFS LZMA decoder, verify that comp_size matches at the end.Lasse Collin1-1/+6
When the uncompressed size is known to be exact, after decompressing the stream exactly comp_size bytes of input must have been consumed. This is a minor improvement to error detection.
2021-01-17liblzma: Make EROFS LZMA decoder work when exact uncomp_size isn't known.Lasse Collin2-12/+91
The caller must still not specify an uncompressed size bigger than the actual uncompressed size. As a downside, this now needs the exact compressed size.
2021-01-14liblzma: Fix missing normalization in rc_encode_dummy().Lasse Collin1-1/+6
Without this fix it could attempt to create too much output.
2021-01-14liblzma: Add EROFS LZMA encoder and decoder.Lasse Collin5-0/+367
Right now this is just a planned extra-compact format for use in the EROFS file system in Linux. At this point it's possible that the format will either change or be abandoned and removed completely. The special thing about the encoder is that it uses the output-size-limited encoding added in the previous commit. EROFS uses fixed-sized blocks (e.g. 4 KiB) to hold compressed data so the compressors must be able to create valid streams that fill the given block size.
2021-01-14liblzma: Add rough support for output-size-limited encoding in LZMA1.Lasse Collin6-35/+246
With this it is possible to encode LZMA1 data without EOPM so that the encoder will encode as much input as it can without exceeding the specified output size limit. The resulting LZMA1 stream will be a normal LZMA1 stream without EOPM. The actual uncompressed size will be available to the caller via the uncomp_size pointer. One missing thing is that the LZMA layer doesn't inform the LZ layer when the encoding is finished and thus the LZ may read more input when it won't be used. However, this doesn't matter if encoding is done with a single call (which is the planned use case for now). For proper multi-call encoding this should be improved. This commit only adds the functionality for internal use. Nothing uses it yet.
2021-01-11Scripts: Add zstd support to xzdiff.Lasse Collin2-7/+15
2021-01-11Update THANKS.Lasse Collin1-0/+2
2021-01-11xz: Make --keep accept symlinks, hardlinks, and setuid/setgid/sticky.Lasse Collin2-5/+20
Previously this required using --force but that has other effects too which might be undesirable. Changing the behavior of --keep has a small risk of breaking existing scripts but since this is a fairly special corner case I expect the likehood of breakage to be low enough. I think the new behavior is more logical. The only reason for the old behavior was to be consistent with gzip and bzip2. Thanks to Vincent Lefevre and Sebastian Andrzej Siewior.
2021-01-11Scripts: Fix exit status of xzgrep.Lasse Collin1-7/+13
Omit the -q option from xz, gzip, and bzip2. With xz this shouldn't matter. With gzip it's important because -q makes gzip replace SIGPIPE with exit status 2. With bzip2 it's important because with -q bzip2 is completely silent if input is corrupt while other decompressors still give an error message. Avoiding exit status 2 from gzip is important because bzip2 uses exit status 2 to indicate corrupt input. Before this commit xzgrep didn't recognize corrupt .bz2 files because xzgrep was treating exit status 2 as SIGPIPE for gzip compatibility. zstd still needs -q because otherwise it is noisy in normal operation. The code to detect real SIGPIPE didn't check if the exit status was due to a signal (>= 128) and so could ignore some other exit status too.
2021-01-11Scripts: Fix exit status of xzdiff/xzcmp.Lasse Collin1-14/+21
This is a minor fix since this affects only the situation when the files differ and the exit status is something else than 0. In such case there could be SIGPIPE from a decompression tool and that would result in exit status of 2 from xzdiff/xzcmp while the correct behavior would be to return 1 or whatever else diff or cmp may have returned. This commit omits the -q option from xz/gzip/bzip2/lzop arguments. I'm not sure why the -q was used in the first place, perhaps it hides warnings in some situation that I cannot see at the moment. Hopefully the removal won't introduce a new bug. With gzip the -q option was harmful because it made gzip return 2 instead of >= 128 with SIGPIPE. Ignoring exit status 2 (warning from gzip) isn't practical because bzip2 uses exit status 2 to indicate corrupt input file. It's better if SIGPIPE results in exit status >= 128. With bzip2 the removal of -q seems to be good because with -q it prints nothing if input is corrupt. The other tools aren't silent in this situation even with -q. On the other hand, if zstd support is added, it will need -q since otherwise it's noisy in normal situations. Thanks to Étienne Mollier and Sebastian Andrzej Siewior.
2021-01-09liblzma: Make lzma_outq usable for threaded decompression too.Lasse Collin3-157/+301
Before this commit all output queue buffers were allocated as a single big allocation. Now each buffer is allocated separately when needed. Used buffers are cached to avoid reallocation overhead but the cache will keep only one buffer size at a time. This should make things work OK in the decompression where most of the time the buffer sizes will be the same but with some less common files the buffer sizes may vary. While this should work fine, it's still a bit preliminary and may even get reverted if it turns out to be useless for decompression.
2020-12-23Update THANKS.Lasse Collin1-0/+1
2020-12-23liblzma: Enable Intel CET in x86 CRC assembly codesH.J. Lu2-0/+18
When Intel CET is enabled, we need to include <cet.h> in assembly codes to mark Intel CET support and add _CET_ENDBR to indirect jump targets. Tested on Intel Tiger Lake under CET enabled Linux.
2020-12-16Update THANKS.Lasse Collin1-0/+1
2020-12-16Build: Don't build bundles on Apple OSes.Lasse Collin1-0/+3
Thanks to Daniel Packard.
2020-12-05Update THANKS.Lasse Collin1-0/+1
2020-12-05Scripts: Add zstd support to xzgrep.Adam Borowski2-3/+7
Thanks to Adam Borowski.
2020-11-17CMake: Fix compatibility with CMake 3.13.Lasse Collin3-5/+5
The syntax "if(DEFINED CACHE{FOO})" requires CMake 3.14. In some other places the code treats the cache variables like normal variables already (${FOO} or if(FOO) is used, not ${CACHE{FOO}). Thanks to ygrek for reporting the bug on IRC.
2020-11-01Update THANKS.Lasse Collin1-1/+3
2020-11-01xz: Avoid unneeded \f escapes on the man page.Lasse Collin1-9/+22
I don't want to use \c in macro arguments but groff_man(7) suggests that \f has better portability. \f would be needed for the .TP strings for portability reasons anyway. Thanks to Bjarni Ingi Gislason.
2020-11-01xz: Use non-breaking spaces when intentionally using more than one space.Lasse Collin1-1/+1
This silences some style checker warnings. Seems that spaces in the beginning of a line don't need this treatment. Thanks to Bjarni Ingi Gislason.
2020-11-01xz: Protect the ellipsis (...) on the man page with \&.Lasse Collin1-2/+2
This does it only when ... appears outside macro calls. Thanks to Bjarni Ingi Gislason.
2020-11-01xz: Avoid the abbreviation "e.g." on the man page.Lasse Collin1-33/+33
A few are simply omitted, most are converted to "for example" and surrounded with commas. Sounds like that this is better style, for example, man-pages(7) recommends avoiding such abbreviations except in parenthesis. Thanks to Bjarni Ingi Gislason.
2020-07-12xz man page: Change \- (minus) to \(en (en-dash) for a numeric range.Lasse Collin1-8/+8
Docs of ancient troff/nroff mention \(em (em-dash) but not \(en and \- was used for both minus and en-dash. I don't know how portable \(en is nowadays but it can be changed back if someone complains. At least GNU groff and OpenBSD's mandoc support it. Thanks to Bjarni Ingi Gislason for the patch.
2020-07-12Windows: Fix building of resource files when config.h isn't used.Lasse Collin1-1/+3
Now CMake + Visual Studio works for building liblzma.dll. Thanks to Markus Rickert.
2020-04-06src/scripts/xzgrep.1: Filenames to xzgrep are optional.Lasse Collin1-1/+1
xzgrep --help was correct already.
2020-04-06src/script/xzgrep.1: Remove superfluous '.RB'Bjarni Ingi Gislason1-6/+6
Output is from: test-groff -b -e -mandoc -T utf8 -rF0 -t -w w -z [ "test-groff" is a developmental version of "groff" ] Input file is ./src/scripts/xzgrep.1 <src/scripts/xzgrep.1>:20 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:23 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:26 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:29 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:32 (macro RB): only 1 argument, but more are expected "abc..." does not mean the same as "abc ...". The output from nroff and troff is unchanged except for the space between "file" and "...". Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2020-04-06xzgrep.1: Delete superfluous '.PP'Bjarni Ingi Gislason1-1/+0
Summary: mandoc -T lint xzgrep.1 : mandoc: xzgrep.1:79:2: WARNING: skipping paragraph macro: PP empty There is no change in the output of "nroff" and "troff". Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2020-04-06src/xz/xz.1: Correct misused two-fonts macrosBjarni Ingi Gislason1-5/+5
Output is from: test-groff -b -e -mandoc -T utf8 -rF0 -t -w w -z [ "test-groff" is a developmental version of "groff" ] Input file is ./src/xz/xz.1 <src/xz/xz.1>:408 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1009 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1743 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1920 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:2213 (macro BR): only 1 argument, but more are expected Output from nroff and troff is unchanged, except for a font change of a full stop (.). Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2020-03-23Typo fixes from fossies.org.Lasse Collin6-7/+7
https://fossies.org/linux/misc/xz-5.2.5.tar.xz/codespell.html
2020-03-17Update NEWS for 5.2.5.Lasse Collin1-0/+105
2020-03-16README: Update outdated sections.Lasse Collin1-10/+11
2020-03-16README: Mention that man pages can be translated.Lasse Collin1-3/+4
2020-03-16Update INSTALL.generic from Automake 1.16.1.Lasse Collin1-159/+162
2020-03-15Update INSTALL for Windows and DOS and add preliminary info for z/OS.Lasse Collin1-10/+41
2020-03-15Build: Update m4/ax_pthread.m4 from Autoconf Archive (again).Lasse Collin1-102/+117
2020-03-11xz: Never use thousand separators in DJGPP builds.Lasse Collin1-2/+12
DJGPP 2.05 added support for thousands separators but it's broken at least under WinXP with Finnish locale that uses a non-breaking space as the thousands separator. Workaround by disabling thousands separators for DJGPP builds.
2020-03-11DOS: Update dos/Makefile for DJGPP 2.05.Lasse Collin1-3/+1
It doesn't need -fgnu89-inline like 2.04beta did.
2020-03-11DOS: Update instructions in dos/INSTALL.txt.Lasse Collin1-30/+29
2020-03-11DOS: Update config.h.Lasse Collin1-0/+8
The added defines assume GCC >= 4.8.
2020-03-02liblzma: Fix a comment and RC_SYMBOLS_MAX.Lasse Collin1-2/+2
The comment didn't match the value of RC_SYMBOLS_MAX and the value itself was slightly larger than actually needed. The only harm about this was that memory usage was a few bytes larger.
2020-02-27Build: Make CMake build fail if tuklib_cpucores or tuklib_physmem fails.Lasse Collin1-0/+18
2020-02-27Build: Add support for --no-po4a option to autogen.sh.Lasse Collin1-1/+10
Normally, if po4a isn't available, autogen.sh will return with non-zero exit status. The option --no-po4a can be useful when one knows that po4a isn't available but wants autogen.sh to still return with zero exit status.
2020-02-25Update THANKS.Lasse Collin1-0/+1
2020-02-25Build: Fix bugs in the CMake files.Lasse Collin7-78/+85
Seems that the phrase "add more quotes" from sh/bash scripting applies to CMake as well. E.g. passing an unquoted list ${FOO} to a function that expects one argument results in only the first element of the list being passed as an argument and the rest get ignored. Adding quotes helps ("${FOO}"). list(INSERT ...) is weird. Inserting an empty string to an empty variable results in empty list, but inserting it to a non-empty variable does insert an empty element to the list. Since INSERT requires at least one element, "${CMAKE_THREAD_LIBS_INIT}" needs to be quoted in CMakeLists.txt. It might result in an empty element in the list. It seems to not matter as empty elements consistently get ignored in that variable. In fact, calling cmake_check_push_state() and cmake_check_pop_state() will strip the empty elements from CMAKE_REQUIRED_LIBRARIES! In addition to quoting fixes, this fixes checks for the cache variables in tuklib_cpucores.cmake and tuklib_physmem.cmake. Thanks to Martin Matuška for testing and reporting the problems. These fixes aren't tested yet but hopefully they soon will be.
2020-02-25Build: Add very limited experimental CMake support.Lasse Collin7-0/+1151
This does *NOT* replace the Autotools-based build system in the foreseeable future. See the comment in the beginning of CMakeLists.txt. So far this has been tested only on GNU/Linux but I commit it anyway to make it easier for others to test. Since I haven't played much with CMake before, it's likely that there are things that have been done in a silly or wrong way and need to be fixed.
2020-02-24Update m4/.gitignore.Lasse Collin1-0/+1
2020-02-24tuklib: Omit an unneeded <sys/types.h> from a tests.Lasse Collin2-2/+0
tuklib_cpucores.c and tuklib_physmem.c don't include <sys/types.h> even via other files in this package, so clearly that header isn't needed in the tests either (no one has reported build problems due to a missing header in a .c file).
2020-02-24liblzma: Remove unneeded <sys/types.h> from fastpos_tablegen.c.Lasse Collin1-1/+0
This file only generates fastpos_table.c. It isn't built as a part of liblzma.
2020-02-22Use defined(__GNUC__) before __GNUC__ in preprocessor lines.Lasse Collin2-3/+5
This should silence the equivalent of -Wundef in compilers that don't define __GNUC__.
2020-02-21liblzma: Add more uses of lzma_memcmplen() to the normal mode of LZMA.Lasse Collin1-6/+10
This gives a tiny encoder speed improvement. This could have been done in 2014 after the commit 544aaa3d13554e8640f9caf7db717a96360ec0f6 but it was forgotten.
2020-02-21Build: Add visibility.m4 from gnulib.Lasse Collin2-1/+77
Appears that this file used to get included as a side effect of gettext. After the change to gettext version requirements this file no longer got copied to the package and so the build was broken.
2020-02-21xz: Silence a warning when sig_atomic_t is long int.Lasse Collin1-1/+1
It can be true at least on z/OS.
2020-02-21xz: Avoid unneeded access of a volatile variable.Lasse Collin1-1/+1
2020-02-21tuklib_integer.m4: Optimize the check order.Lasse Collin1-27/+29
The __builtin byteswapping is the preferred one so check for it first.
2020-02-20tuklib_exit: Add missing header.Lasse Collin1-0/+1
strerror() needs <string.h> which happened to be included via tuklib_common.h -> tuklib_config.h -> sysdefs.h if HAVE_CONFIG_H was defined. This wasn't tested without config.h before so it had worked fine.
2020-02-18Revert the previous commit and add a comment.Lasse Collin1-7/+10
The previous commit broke crc32_tablegen.c. If the whole package is built without config.h (with defines set on the compiler command line) this should still work fine as long as these headers conform to C99 well enough.
2020-02-17Do not check for HAVE_CONFIG_H in tuklib_config.h.Lasse Collin1-8/+7
In XZ Utils sysdefs.h takes care of it and the required headers.
2020-02-16sysdefs.h: Omit the conditionals around string.h and limits.h.Lasse Collin1-6/+2
string.h is used unconditionally elsewhere in the project and configure has always stopped if limits.h is missing, so these headers must have been always available even on the weirdest systems.
2020-02-15Build: Bump Autoconf and Libtool version requirements.Lasse Collin1-2/+2
There is no specific reason for this other than blocking the most ancient versions. These are still old: Autoconf 2.69 (2012) Automake 1.12 (2012) gettext 0.19.6 (2015) Libtool 2.4 (2010)
2020-02-15Build: Use AM_GNU_GETTEXT_REQUIRE_VERSION and require 0.19.6.Lasse Collin1-1/+5
This bumps the version requirement from 0.19 (from 2014) to 0.19.6 (2015). Using only the old AM_GNU_GETTEXT_VERSION results in old gettext infrastructure being placed in the package. By using both macros we get the latest gettext files while the other programs in the Autotools family can still see the old macro.
2020-02-14Translations: Add German translation of the man pages.Lasse Collin2-1/+5533
Thanks to Mario Blättermann.
2020-02-07Build: Add support for translated man pages using po4a.Lasse Collin8-45/+197
The dependency on po4a is optional. It's never required to install the translated man pages when xz is built from a release tarball. If po4a is missing when building from xz.git, the translated man pages won't be generated but otherwise the build will work normally. The translations are only updated automatically by autogen.sh and by "make mydist". This makes it easy to keep po4a as an optional dependency and ensures that I won't forget to put updated translations to a release tarball. The translated man pages aren't installed if --disable-nls is used. The installation of translated man pages abuses Automake internals by calling "install-man" with redefined dist_man_MANS and man_MANS. This makes the hairy script code slightly less hairy. If it breaks some day, this code needs to be fixed; don't blame Automake developers. Also, this adds more quotes to the existing shell script code in the Makefile.am "-hook"s.
2020-02-06Update THANKS.Lasse Collin1-0/+1
2020-02-05Update tests/.gitignore.Lasse Collin1-0/+4
2020-02-05Update m4/.gitignore.Lasse Collin1-0/+1
2020-02-05Update THANKS.Lasse Collin1-0/+1
2020-02-05xz: Make it a fatal error if enabling the sandbox fails.Lasse Collin1-1/+1
Perhaps it's too drastic but on the other hand it will let me learn about possible problems if people report the errors. This won't be backported to the v5.2 branch.
2020-02-05xz: Comment out annoying sandboxing messages.Lasse Collin1-3/+7
2020-02-05Build: Workaround a POSIX shell detection problem on Solaris.Lasse Collin1-0/+10
I don't know if the problem is in gnulib's gl_POSIX_SHELL macro or if xzgrep does something that isn't in POSIX. The workaround adds a special case for Solaris: if /usr/xpg4/bin/sh exists and gl_cv_posix_shell wasn't overriden on the configure command line, use that shell for xzgrep and other scripts. That shell is known to work and exists on most Solaris systems.
2020-02-03Build: Update m4/ax_pthread.m4 from Autoconf Archive.Lasse Collin1-119/+279
2020-02-01xz: Limit --memlimit-compress to at most 4020 MiB for 32-bit xz.Lasse Collin2-2/+51
See the code comment for reasoning. It's far from perfect but hopefully good enough for certain cases while hopefully doing nothing bad in other situations. At presets -5 ... -9, 4020 MiB vs. 4096 MiB makes no difference on how xz scales down the number of threads. The limit has to be a few MiB below 4096 MiB because otherwise things like "xz --lzma2=dict=500MiB" won't scale down the dict size enough and xz cannot allocate enough memory. With "ulimit -v $((4096 * 1024))" on x86-64, the limit in xz had to be no more than 4085 MiB. Some safety margin is good though. This is hack but it should be useful when running 32-bit xz on a 64-bit kernel that gives full 4 GiB address space to xz. Hopefully this is enough to solve this: https://bugzilla.redhat.com/show_bug.cgi?id=1196786 FreeBSD has a patch that limits the result in tuklib_physmem() to SIZE_MAX on 32-bit systems. While I think it's not the way to do it, the results on --memlimit-compress have been good. This commit should achieve practically identical results for compression while leaving decompression and tuklib_physmem() and thus lzma_physmem() unaffected.
2020-01-26xz: Set the --flush-timeout deadline when the first input byte arrives.Lasse Collin3-7/+6
xz --flush-timeout=2000, old version: 1. xz is started. The next flush will happen after two seconds. 2. No input for one second. 3. A burst of a few kilobytes of input. 4. No input for one second. 5. Two seconds have passed and flushing starts. The first second counted towards the flush-timeout even though there was no pending data. This can cause flushing to occur more often than needed. xz --flush-timeout=2000, after this commit: 1. xz is started. 2. No input for one second. 3. A burst of a few kilobytes of input. The next flush will happen after two seconds counted from the time when the first bytes of the burst were read. 4. No input for one second. 5. No input for another second. 6. Two seconds have passed and flushing starts.
2020-01-26xz: Move flush_needed from mytime.h to file_pair struct in file_io.h.Lasse Collin5-9/+7
2020-01-26xz: coder.c: Make writing output a separate function.Lasse Collin1-13/+17
The same code sequence repeats so it's nicer as a separate function. Note that in one case there was no test for opt_mode != MODE_TEST, but that was only because that condition would always be true, so this commit doesn't change the behavior there.
2020-01-26xz: Fix semi-busy-waiting in xz --flush-timeout.Lasse Collin3-4/+19
When input blocked, xz --flush-timeout=1 would wake up every millisecond and initiate flushing which would have nothing to flush and thus would just waste CPU time. The fix disables the timeout when no input has been seen since the previous flush.
2020-01-26xz: Refactor io_read() a bit.Lasse Collin1-9/+8
2020-01-26xz: Update a comment in file_io.h.Lasse Collin1-1/+4
2020-01-26xz: Move the setting of flush_needed in file_io.c to a nicer location.Lasse Collin1-4/+2
2020-01-19Use $(LIB_FUZZING_ENGINE) in tests/ossfuzz/Makefile.Lasse Collin1-1/+1
https://github.com/google/oss-fuzz/pull/3219#issuecomment-573751048 Thanks to Bhargava Shastry for sending the patch.
2019-12-31Rename unaligned_read32ne to read32ne, and similarly for the others.Lasse Collin15-62/+54
2019-12-31Rename read32ne to aligned_read32ne, and similarly for the others.Lasse Collin3-32/+32
Using the aligned methods requires more care to ensure that the address really is aligned, so it's nicer if the aligned methods are prefixed. The next commit will remove the unaligned_ prefix from the unaligned methods which in liblzma are used in more places than the aligned ones.
2019-12-31Revise tuklib_integer.h and .m4.Lasse Collin2-218/+316
Add a configure option --enable-unsafe-type-punning to get the old non-conforming memory access methods. It can be useful with old compilers or in some other less typical situations but shouldn't normally be used. Omit the packed struct trick for unaligned access. While it's best in some cases, this is simpler. If the memcpy trick doesn't work, one can request unsafe type punning from configure. Because CRC32/CRC64 code needs fast aligned reads, if no very safe way to do it is found, type punning is used as a fallback. This sucks but since it currently works in practice, it seems to be the least bad option. It's never needed with GCC >= 4.7 or Clang >= 3.6 since these support __builtin_assume_aligned and thus fast aligned access can be done with the memcpy trick. Other things: - Support GCC/Clang __builtin_bswapXX - Cleaner bswap fallback macros - Minor cleanups
2019-12-29Tests: Hopefully fix test_check.c to work on EBCDIC systems.Lasse Collin1-2/+7
Thanks to Daniel Richard G.
2019-09-24Scripts: Put /usr/xpg4/bin to the beginning of PATH on Solaris.Lasse Collin6-8/+65
This adds a configure option --enable-path-for-scripts=PREFIX which defaults to empty except on Solaris it is /usr/xpg4/bin to make POSIX grep and others available. The Solaris case had been documented in INSTALL with a manual fix but it's better to do this automatically since it is needed on most Solaris systems anyway. Thanks to Daniel Richard G.
2019-07-12Fix comment typos in tuklib_mbstr* files.Lasse Collin3-3/+3
2019-07-12Add missing include to tuklib_mbstr_width.c.Lasse Collin1-0/+1
It didn't matter in XZ Utils because sysdefs.h includes string.h anyway.
2019-07-12Update tuklib base headers to include stdbool.h.Lasse Collin2-1/+2
2019-06-28xz: Automatically align the strings in --info-memory.Lasse Collin1-11/+34
This makes it easier to translate the strings. Also, the string for amount of RAM was shortened.
2019-06-25liblzma: Fix a buggy comment.Lasse Collin1-1/+1
2019-06-25configure.ac: Fix a typo in a comment.Lasse Collin1-1/+1
2019-06-25Tests: Silence warnings from clang -Wassign-enum.Lasse Collin2-2/+3
Also changed 999 to 99 so it fits even if lzma_check happened to be 8 bits wide.
2019-06-24liblzma: Add a comment.Lasse Collin1-1/+1
2019-06-24liblzma: Silence clang -Wmissing-variable-declarations.Lasse Collin2-0/+6
2019-06-24Add LZMA_RET_INTERNAL1..8 to lzma_ret and use one for LZMA_TIMED_OUT.Lasse Collin4-7/+25
LZMA_TIMED_OUT is *internally* used as a value for lzma_ret enumeration. Previously it was #defined to 32 and cast to lzma_ret. That way it wasn't visible in the public API, but this was hackish. Now the public API has eight LZMA_RET_INTERNALx members and LZMA_TIMED_OUT is #defined to LZMA_RET_INTERNAL1. This way the code is cleaner overall although the public API has a few extra mysterious enum members.
2019-06-24xz: Silence a warning from clang -Wsign-conversion in main.c.Lasse Collin1-1/+1
2019-06-24xz: Make "headings" static in list.c.Lasse Collin1-1/+1
Caught by clang -Wmissing-variable-declarations.
2019-06-24liblzma: Remove incorrect uses of lzma_attribute((__unused__)).Lasse Collin3-6/+3
Caught by clang -Wused-but-marked-unused.
2019-06-24Tests: Silence a warning from -Wsign-conversion.Lasse Collin1-4/+4
2019-06-24xz: Fix an integer overflow with 32-bit off_t.Lasse Collin1-2/+9
Or any off_t which isn't very big (like signed 64 bit integer that most system have). A small off_t could overflow if the file being decompressed had long enough run of zero bytes, which would result in corrupt output.
2019-06-24xz: Cleanup io_seek_src() a bit.Lasse Collin1-3/+1
lseek() returns -1 on error and checking for -1 is nicer.
2019-06-24Tests: Remove a duplicate branch from tests/tests.h.Lasse Collin1-7/+2
The duplication was introduced about eleven years ago and should have been cleaned up back then already. This was caught by -Wduplicated-branches.
2019-06-24xz: Change io_seek_src and io_pread arguments from off_t to uint64_t.Lasse Collin3-11/+18
This helps fixing warnings from -Wsign-conversion and makes the code look better too.
2019-06-24xz: list.c: Fix some warnings from -Wsign-conversion.Lasse Collin1-3/+4
2019-06-23tuklib_mbstr_width: Fix a warning from -Wsign-conversion.Lasse Collin1-1/+1
2019-06-23xz: Fix some of the warnings from -Wsign-conversion.Lasse Collin7-13/+14
2019-06-23tuklib_cpucores: Silence warnings from -Wsign-conversion.Lasse Collin1-5/+5
2019-06-23xzdec: Fix warnings from -Wsign-conversion.Lasse Collin1-1/+1
2019-06-23liblzma: Fix warnings from -Wsign-conversion.Lasse Collin11-29/+31
Also, more parentheses were added to the literal_subcoder macro in lzma_comon.h (better style but no functional change in the current usage).
2019-06-23tuklib_integer: Silence warnings from -Wsign-conversion.Lasse Collin1-3/+3
2019-06-20tuklib_integer: Fix usage of conv macros.Lasse Collin1-4/+8
Use a temporary variable instead of e.g. conv32le(unaligned_read32ne(buf)) because the macro can evaluate its argument multiple times.
2019-06-03Update THANKS.Lasse Collin1-0/+1
2019-06-03liblzma: Fix comments.Lasse Collin6-7/+7
Thanks to Bruce Stark.
2019-06-02liblzma: Fix one more unaligned read to use unaligned_read16ne().Lasse Collin1-1/+1
2019-06-01Update THANKS.Lasse Collin1-0/+1
2019-06-01liblzma: memcmplen: Use ctz32() from tuklib_integer.h.Lasse Collin1-9/+1
The same compiler-specific #ifdefs are already in tuklib_integer.h
2019-06-01tuklib_integer: Cleanup MSVC-specific code.Lasse Collin1-11/+9
2019-06-01liblzma: Use unaligned_readXXne functions instead of type punning.Lasse Collin2-7/+7
Now gcc -fsanitize=undefined should be clean. Thanks to Jeffrey Walton.
2019-06-01tuklib_integer: Autodetect support for unaligned access on ARM.Lasse Collin1-0/+11
The result is used as the default for --enable-unaligned-access. The test should work with GCC and Clang.
2019-06-01tuklib_integer: Improve unaligned memory access.Lasse Collin1-12/+168
Now memcpy() or GNU C packed structs for unaligned access instead of type punning. See the comment in this commit for details. Avoiding type punning with unaligned access is needed to silence gcc -fsanitize=undefined. New functions: unaliged_readXXne and unaligned_writeXXne where XX is 16, 32, or 64.
2019-05-13liblzma: Avoid memcpy(NULL, foo, 0) because it is undefined behavior.Lasse Collin3-5/+23
I should have always known this but I didn't. Here is an example as a reminder to myself: int mycopy(void *dest, void *src, size_t n) { memcpy(dest, src, n); return dest == NULL; } In the example, a compiler may assume that dest != NULL because passing NULL to memcpy() would be undefined behavior. Testing with GCC 8.2.1, mycopy(NULL, NULL, 0) returns 1 with -O0 and -O1. With -O2 the return value is 0 because the compiler infers that dest cannot be NULL because it was already used with memcpy() and thus the test for NULL gets optimized out. In liblzma, if a null-pointer was passed to memcpy(), there were no checks for NULL *after* the memcpy() call, so I cautiously suspect that it shouldn't have caused bad behavior in practice, but it's hard to be sure, and the problematic cases had to be fixed anyway. Thanks to Jeffrey Walton.
2019-05-11Update THANKS.Lasse Collin1-0/+1
2019-05-11xz: Update xz man page date.Lasse Collin1-1/+1
2019-05-11spellingAntoine Cœur22-26/+26
2019-05-01xz: In xz -lvv look at the widths of the check names too.Lasse Collin1-6/+26
Now the widths of the check names is used to adjust the width of the Check column. This way there no longer is a need to restrict the widths of the check names to be at most ten terminal-columns.
2019-05-01xz: Fix xz -lvv column alignment to look at the translated strings.Lasse Collin1-2/+2
2019-05-01Update THANKS.Lasse Collin1-0/+1
2019-05-01Windows: Update VS version in windows/vs2019/config.h.Lasse Collin1-1/+1
2019-05-01Windows: Upgrade solution itselfJulien Marrec1-2/+5
2019-05-01Windows: Upgrade solution with VS2019Julien Marrec2-14/+16
2019-05-01Windows: Duplicate windows/vs2017 before upgradingJulien Marrec4-0/+937
2019-03-04README: Update translation instructions.Lasse Collin1-19/+13
XZ Utils is now part of the Translation Project <https://translationproject.org/>.
2019-03-04xz: Automatically align column headings in xz -lvv.Lasse Collin1-51/+212
2019-03-04xz: Automatically align strings ending in a colon in --list output.Lasse Collin1-12/+102
This should avoid alignment errors in translations with these strings.
2019-01-13Windows/VS2017: Omit WindowsTargetPlatformVersion from project files.Lasse Collin3-2/+4
I understood that if a WTPV is specified, it's often wrong because different VS installations have different SDK version installed. Omitting the WTPV tag makes VS2017 default to Windows SDK 8.1 which often is also missing, so in any case people may need to specify the WTPV before building. But some day in the future a missing WTPV tag will start to default to the latest installed SDK which sounds reasonable: https://developercommunity.visualstudio.com/content/problem/140294/windowstargetplatformversion-makes-it-impossible-t.html Thanks to "dom".
2018-12-20Update THANKS.Lasse Collin1-0/+1
2018-12-20xz: Fix a crash in progress indicator when in passthru mode.Lasse Collin3-7/+25
"xz -dcfv not_an_xz_file" crashed (all four options are required to trigger it). It caused xz to call lzma_get_progress(&strm, ...) when no coder was initialized in strm. In this situation strm.internal is NULL which leads to a crash in lzma_get_progress(). The bug was introduced when xz started using lzma_get_progress() to get progress info for multi-threaded compression, so the bug is present in versions 5.1.3alpha and higher. Thanks to Filip Palian <Filip.Palian@pjwstk.edu.pl> for the bug report.
2018-11-22xz: Update man page timestamp.Lasse Collin1-1/+1
2018-11-22'have have' typosPavel Raiskup2-2/+2
2018-11-02Update THANKS.Lasse Collin1-0/+1
2018-11-02Tests: Add a fuzz test program and a config file for OSS-Fuzz.Lasse Collin5-0/+94
Thanks to Bhargava Shastry and Github user pdknsk.
2018-10-26liblzma: Don't verify header CRC32s if building for fuzz testing.Lasse Collin4-5/+20
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is #defined when liblzma is being built for fuzz testing. Most fuzzed inputs would normally get rejected because of incorrect CRC32 and the actual header decoding code wouldn't get fuzzed. Disabling CRC32 checks avoids this problem. The fuzzer program must still use LZMA_IGNORE_CHECK flag to disable verification of integrity checks of uncompressed data.
2018-07-27xzless: Rename unused variables to silence static analysers.Lasse Collin1-1/+1
In this particular case I don't see this affecting readability of the code. Thanks to Pavel Raiskup.
2018-07-27liblzma: Remove an always-true condition from lzma_index_cat().Lasse Collin1-2/+2
This should help static analysis tools to see that newg isn't leaked. Thanks to Pavel Raiskup.
2018-05-19liblzma: Improve lzma_properties_decode() API documentation.Lasse Collin1-3/+4
2018-04-29Bump the version number to 5.3.1alpha.larhzu/v5.3.1alphaLasse Collin2-2/+2
2018-04-29extra/scanlzma: Fix compiler warnings.Lasse Collin1-1/+5
2018-04-29DOS: Add file_info.c to the list of files to build.Lasse Collin1-0/+1
2018-04-29Update NEWS for 5.3.1alpha.Lasse Collin1-0/+11
2018-04-29Add NEWS for 5.2.4.Lasse Collin1-0/+27
2018-02-06Update THANKS.Lasse Collin1-0/+2
2018-02-06nothrow: use noexcept for C++11 and newerBen Boeckel1-1/+5
In C++11, the `throw()` specifier is deprecated and `noexcept` is preffered instead.
2018-02-06liblzma: Remove incorrect #ifdef from range_common.h.Lasse Collin1-3/+1
In most cases it was harmless but it could affect some custom build systems. Thanks to Pippijn van Steenhoven.
2018-01-10Update THANKS.Lasse Collin1-0/+1
2018-01-10tuklib_integer: New Intel C compiler needs immintrin.h.Lasse Collin1-0/+11
Thanks to Melanie Blower (Intel) for the patch.
2017-09-24Update THANKS.Lasse Collin1-0/+1
2017-09-16Windows: Fix paths in VS project files.Lasse Collin4-36/+36
Some paths use slashes instead of backslashes as directory separators... now it should work (I tested VS2013 version).
2017-09-16Windows: Update VS2017 project files to include file info decoder.Lasse Collin2-0/+4
2017-09-16Windows: Add project files for VS2017.Lasse Collin4-0/+935
These files match the v5.2 branch (no file info decoder).
2017-09-16Windows: Update VS2013 project files to include file info decoder.Lasse Collin2-0/+4
2017-09-16Windows: Move VS2013 files into windows/vs2013 directory.Lasse Collin4-279/+279
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-21 16:55:08 +0000