aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2022-11-13Bump version and soname for 5.2.8.larhzu/v5.2.8Lasse Collin2-2/+2
2022-11-13Add NEWS for 5.2.8.Lasse Collin1-0/+79
2022-11-11Update THANKS.Lasse Collin1-0/+1
2022-11-11liblzma: Fix building with Intel ICC (the classic compiler).Lasse Collin1-1/+1
It claims __GNUC__ >= 10 but doesn't support __symver__ attribute. Thanks to Stephen Sachs.
2022-11-11liblzma: Fix incorrect #ifdef for x86 SSE2 support.Lasse Collin1-2/+1
__SSE2__ is the correct macro for SSE2 support with GCC, Clang, and ICC. __SSE2_MATH__ means doing floating point math with SSE2 instead of 387. Often the latter macro is defined if the first one is but it was still a bug.
2022-11-11Scripts: Ignore warnings from xz.Lasse Collin4-7/+8
In practice this means making the scripts work when the input files have an unsupported check type which isn't a problem in practice unless support for some check types has been disabled at build time.
2022-11-11xz: Add comments about stdin and src_st.st_size.Lasse Collin2-0/+13
"xz -v < regular_file > out.xz" doesn't display the percentage and estimated remaining time because it doesn't even try to check the input file size when input is read from stdin. This could be improved but for now there's just a comment to remind about it.
2022-11-11xz: Fix displaying of file sizes in progress indicator in passthru mode.Lasse Collin1-1/+5
It worked for one input file since the counters are zero when xz starts but they weren't reset when starting a new file in passthru mode. For example, if files A, B, and C are one byte each, then "xz -dcvf A B C" would show file sizes as 1, 2, and 3 bytes instead of 1, 1, and 1 byte.
2022-11-11xz: Add a comment why --to-stdout is not in --help.Lasse Collin1-0/+3
It is on the man page still.
2022-11-11Docs: Update faq.txt a little.Lasse Collin1-23/+43
2022-11-11liblzma: Update API docs about decoder flags.Lasse Collin1-4/+16
2022-11-11liblzma: Fix a comment in auto_decoder.c.Lasse Collin1-2/+2
2022-11-11xz: Avoid a compiler warning in progress_speed() in message.c.Jia Tan1-6/+3
This should be smaller too since it avoids the string constants.
2022-11-11Build: Clarify comment in configure.ac about SSE2.Lasse Collin1-1/+4
2022-11-11Build: Remove obsolete commented-out lines from configure.ac.Lasse Collin1-4/+0
2022-11-11Windows: Fix mythread_once() macro with Vista threads.Lasse Collin1-3/+4
Don't call InitOnceComplete() if initialization was already done. So far mythread_once() has been needed only when building with --enable-small. windows/build.bash does this together with --disable-threads so the Vista-specific mythread_once() is never needed by those builds. VS project files or CMake-builds don't support HAVE_SMALL builds at all.
2022-11-11CMake: Sync tuklib_cpucores.cmake with tuklib_cpucores.m4.Lasse Collin1-0/+5
This was forgotten from commit 2611c4d90535652d3eb7ef4a026a6691276fab43.
2022-11-11Build: Use AC_CONFIG_HEADERS instead of the ancient AC_CONFIG_HEADER.Lasse Collin1-1/+1
We require Autoconf >= 2.69 and that has AC_CONFIG_HEADERS. There is a warning about AC_PROG_CC_C99 being obsolete but it cannot be removed because it is needed with Autoconf 2.69.
2022-11-11Build: Update m4/ax_pthread.m4 from Autoconf Archive.Lasse Collin1-7/+22
2022-11-11xz: Fix --single-stream with an empty .xz Stream.Lasse Collin1-0/+9
Example: $ xz -dc --single-stream good-0-empty.xz xz: good-0-empty.xz: Internal error (bug) The code, that is tries to catch some input file issues early, didn't anticipate LZMA_STREAM_END which is possible in that code only when --single-stream is used.
2022-11-11xz: Fix decompressor behavior if input uses an unsupported check type.Lasse Collin1-4/+15
Now files with unsupported check will make xz display a warning, set the exit status to 2 (unless --no-warn is used), and then decompress the file normally. This is how it was supposed to work since the beginning but this was broken by the commit 231c3c7098f1099a56abb8afece76fc9b8699f05, that is, a little before 5.0.0 was released. The buggy behavior displayed a message, set exit status 1 (error), and xz didn't attempt to to decompress the file. This doesn't matter today except for special builds that disable CRC64 or SHA-256 at build time (but such builds should be used in special situations only). The bug matters if new check type is added in the future and an old xz version is used to decompress such a file; however, it's likely that such files would use a new filter too and an old xz wouldn't be able to decompress the file anyway. The first hunk in the commit is the actual fix. The second hunk is a cleanup since LZMA_TELL_ANY_CHECK isn't used in xz. There is a test file for unsupported check type but it wasn't used by test_files.sh, perhaps due to different behavior between xz and the simpler xzdec.
2022-11-11xz: Clarify the man page: input file isn't removed if an error occurs.Lasse Collin1-2/+3
2022-11-11xz: If input file cannot be removed, treat it as a warning, not error.Lasse Collin1-2/+2
Treating it as a warning (message + exit status 2) matches gzip and it seems more logical as at that point the output file has already been successfully closed. When it's a warning it is possible to suppress it with --no-warn.
2022-11-11tuklib_cpucores: Use HW_NCPUONLINE on OpenBSD.Lasse Collin2-0/+14
On OpenBSD the number of cores online is often less than what HW_NCPU would return because OpenBSD disables simultaneous multi-threading (SMT) by default. Thanks to Christian Weisgerber.
2022-11-11NEWS: Omit the extra copy of 5.2.5 NEWS.Lasse Collin1-105/+0
It was a copy-paste error.
2022-11-10Translations: Rename poa4/fr_FR.po to po4a/fr.po.Lasse Collin2-1/+1
That's how it is preferred at the Translation Project. On my system /usr/share/man/fr_FR doesn't contain any other man pages than XZ Utils while /usr/share/man/fr has quite a few, so this will fix that too. Thanks to Benno Schulenberg from the Translation Project.
2022-11-08Translations: Update Turkish translation.Lasse Collin1-1/+1
2022-11-08Translations: Update Croatian translation.Lasse Collin1-94/+96
2022-09-30Bump version and soname for 5.2.7.larhzu/v5.2.7Lasse Collin2-2/+2
2022-09-30Add NEWS for 5.2.7.Lasse Collin1-0/+74
2022-09-30liblzma: Add API doc note about the .xz decoder LZMA_MEMLIMIT_ERROR bug.Lasse Collin1-0/+11
The bug was fixed in 660739f99ab211edec4071de98889fb32ed04e98.
2022-09-29liblzma: Add dest and src NULL checks to lzma_index_cat.Jia Tan1-0/+3
The documentation states LZMA_PROG_ERROR can be returned from lzma_index_cat. Previously, lzma_index_cat could not return LZMA_PROG_ERROR. Now, the validation is similar to lzma_index_append, which does a NULL check on the index parameter.
2022-09-29Tests: Create a test for the lzma_index_cat bug.Jia Tan1-1/+42
2022-09-29liblzma: Fix copying of check type statistics in lzma_index_cat().Jia Tan1-1/+6
The check type of the last Stream in dest was never copied to dest->checks (the code tried to copy it but it was done too late). This meant that the value returned by lzma_index_checks() would only include the check type of the last Stream when multiple lzma_indexes had been concatenated. In xz --list this meant that the summary would only list the check type of the last Stream, so in this sense this was only a visual bug. However, it's possible that some applications use this information for purposes other than merely showing it to the users in an informational message. I'm not aware of such applications though and it's quite possible that such applications don't exist. Regular streamed decompression in xz or any other application doesn't use lzma_index_cat() and so this bug cannot affect them.
2022-09-29tuklib_physmem: Fix Unicode builds on Windows.Lasse Collin1-1/+1
Thanks to ArSaCiA Game.
2022-09-29liblzma: Stream decoder: Fix restarting after LZMA_MEMLIMIT_ERROR.Lasse Collin1-3/+13
If lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible to use lzma_memlimit_set() to increase the limit and continue decoding. This was supposed to work from the beginning but there was a bug. With other decoders (.lzma or threaded .xz) this already worked correctly.
2022-09-29liblzma: Stream decoder: Fix comments.Lasse Collin1-7/+5
2022-09-17Update THANKS.Lasse Collin1-0/+1
2022-09-17xzgrep: Fix compatibility with old shells.Lasse Collin1-3/+3
Running the current xzgrep on Slackware 10.1 with GNU bash 3.00.15: xzgrep: line 231: syntax error near unexpected token `;;' On SCO OpenServer 5.0.7 with Korn Shell 93r: syntax error at line 231 : `;;' unexpected Turns out that some old shells don't like apostrophes (') inside command substitutions. For example, the following fails: x=$(echo foo # asdf'zxcv echo bar) printf '%s\n' "$x" The problem was introduced by commits 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 (2022-03-29), bd7b290f3fe4faeceb7d3497ed9bf2e6ed5e7dc5 (2022-07-18), and a648978b20495b7aa4a8b029c5a810b5ad9d08ff (2022-07-19). 5.2.6 is the only stable release that included this problem. Thanks to Kevin R. Bulgrien for reporting the problem on SCO OpenServer 5.0.7 and for providing the fix.
2022-09-17liblzma: lzma_filters_copy: Keep dest[] unmodified if an error occurs.Lasse Collin2-7/+15
lzma_stream_encoder() and lzma_stream_encoder_mt() always assumed this. Before this patch, failing lzma_filters_copy() could result in free(invalid_pointer) or invalid memory reads in stream_encoder.c or stream_encoder_mt.c. To trigger this, allocating memory for a filter options structure has to fail. These are tiny allocations so in practice they very rarely fail. Certain badness in the filter chain array could also make lzma_filters_copy() fail but both stream_encoder.c and stream_encoder_mt.c validate the filter chain before trying to copy it, so the crash cannot occur this way.
2022-09-17Tests: Add a test file for lzma_index_append() integer overflow bug.Lasse Collin3-0/+18
This test fails before commit 18d7facd3802b55c287581405c4d49c98708c136. test_files.sh now runs xz -l for bad-3-index-uncomp-overflow.xz because only then the previously-buggy code path gets tested. Normal decompression doesn't use lzma_index_append() at all. Instead, lzma_index_hash functions are used and those already did the overflow check.
2022-09-17liblzma: lzma_index_append: Add missing integer overflow check.Jia Tan1-0/+4
The documentation in src/liblzma/api/lzma/index.h suggests that both the unpadded (compressed) size and the uncompressed size are checked for overflow, but only the unpadded size was checked. The uncompressed check is done first since that is more likely to occur than the unpadded or index field size overflows.
2022-09-16Update THANKS.Lasse Collin1-0/+1
2022-09-16liblzma: Vaccinate against an ill patch from RHEL/CentOS 7.Lasse Collin10-15/+374
RHEL/CentOS 7 shipped with 5.1.2alpha, including the threaded encoder that is behind #ifdef LZMA_UNSTABLE in the API headers. In 5.1.2alpha these symbols are under XZ_5.1.2alpha in liblzma.map. API/ABI compatibility tracking isn't done between development releases so newer releases didn't have XZ_5.1.2alpha anymore. Later RHEL/CentOS 7 updated xz to 5.2.2 but they wanted to keep the exported symbols compatible with 5.1.2alpha. After checking the ABI changes it turned out that >= 5.2.0 ABI is backward compatible with the threaded encoder functions from 5.1.2alpha (but not vice versa as fixes and extensions to these functions were made between 5.1.2alpha and 5.2.0). In RHEL/CentOS 7, XZ Utils 5.2.2 was patched with xz-5.2.2-compat-libs.patch to modify liblzma.map: - XZ_5.1.2alpha was added with lzma_stream_encoder_mt and lzma_stream_encoder_mt_memusage. This matched XZ Utils 5.1.2alpha. - XZ_5.2 was replaced with XZ_5.2.2. It is clear that this was an error; the intention was to keep using XZ_5.2 (XZ_5.2.2 has never been used in XZ Utils). So XZ_5.2.2 lists all symbols that were listed under XZ_5.2 before the patch. lzma_stream_encoder_mt and _mt_memusage are included too so they are listed both here and under XZ_5.1.2alpha. The patch didn't add any __asm__(".symver ...") lines to the .c files. Thus the resulting liblzma.so exports the threaded encoder functions under XZ_5.1.2alpha only. Listing the two functions also under XZ_5.2.2 in liblzma.map has no effect without matching .symver lines. The lack of XZ_5.2 in RHEL/CentOS 7 means that binaries linked against unpatched XZ Utils 5.2.x won't run on RHEL/CentOS 7. This is unfortunate but this alone isn't too bad as the problem is contained within RHEL/CentOS 7 and doesn't affect users of other distributions. It could also be fixed internally in RHEL/CentOS 7. The second problem is more serious: In XZ Utils 5.2.2 the API headers don't have #ifdef LZMA_UNSTABLE for obvious reasons. This is true in RHEL/CentOS 7 version too. Thus now programs using new APIs can be compiled without an extra #define. However, the programs end up depending on symbol version XZ_5.1.2alpha (and possibly also XZ_5.2.2) instead of XZ_5.2 as they would with an unpatched XZ Utils 5.2.2. This means that such binaries won't run on other distributions shipping XZ Utils >= 5.2.0 as they don't provide XZ_5.1.2alpha or XZ_5.2.2; they only provide XZ_5.2 (and XZ_5.0). (This includes RHEL/CentOS 8 as the patch luckily isn't included there anymore with XZ Utils 5.2.4.) Binaries built by RHEL/CentOS 7 users get distributed and then people wonder why they don't run on some other distribution. Seems that people have found out about the patch and been copying it to some build scripts, seemingly curing the symptoms but actually spreading the illness further and outside RHEL/CentOS 7. The ill patch seems to be from late 2016 (RHEL 7.3) and in 2017 it had spread at least to EasyBuild. I heard about the events only recently. :-( This commit splits liblzma.map into two versions: one for GNU/Linux and another for other OSes that can use symbol versioning (FreeBSD, Solaris, maybe others). The Linux-specific file and the matching additions to .c files add full compatibility with binaries that have been built against a RHEL/CentOS-patched liblzma. Builds for OSes other than GNU/Linux won't get the vaccine as they should be immune to the problem (I really hope that no build script uses the RHEL/CentOS 7 patch outside GNU/Linux). The RHEL/CentOS compatibility symbols XZ_5.1.2alpha and XZ_5.2.2 are intentionally put *after* XZ_5.2 in liblzma_linux.map. This way if one forgets to #define HAVE_SYMBOL_VERSIONS_LINUX when building, the resulting liblzma.so.5 will have lzma_stream_encoder_mt@@XZ_5.2 since XZ_5.2 {...} is the first one that lists that function. Without HAVE_SYMBOL_VERSIONS_LINUX @XZ_5.1.2alpha and @XZ_5.2.2 will be missing but that's still a minor problem compared to only having lzma_stream_encoder_mt@@XZ_5.1.2alpha! The "local: *;" line was moved to XZ_5.0 so that it doesn't need to be moved around. It doesn't matter where it is put. Having two similar liblzma_*.map files is a bit silly as it is, at least for now, easily possible to generate the generic one from the Linux-specific file. But that adds extra steps and increases the risk of mistakes when supporting more than one build system. So I rather maintain two files in parallel and let validate_map.sh check that they are in sync when "make mydist" is run. This adds .symver lines for lzma_stream_encoder_mt@XZ_5.2.2 and lzma_stream_encoder_mt_memusage@XZ_5.2.2 even though these weren't exported by RHEL/CentOS 7 (only @@XZ_5.1.2alpha was for these two). I added these anyway because someone might misunderstand the RHEL/CentOS 7 patch and think that @XZ_5.2.2 (@@XZ_5.2.2) versions were exported too. At glance one could suggest using __typeof__ to copy the function prototypes when making aliases. However, this doesn't work trivially because __typeof__ won't copy attributes (lzma_nothrow, lzma_pure) and it won't change symbol visibility from hidden to default (done by LZMA_API()). Attributes could be copied with __copy__ attribute but that needs GCC 9 and a fallback method would be needed anyway. This uses __symver__ attribute with GCC >= 10 and __asm__(".symver ...") with everything else. The attribute method is required for LTO (-flto) support with GCC. Using -flto with GCC older than 10 is now broken on GNU/Linux and will not be fixed (can silently result in a broken liblzma build that has dangerously incorrect symbol versions). LTO builds with Clang seem to work with the traditional __asm__(".symver ...") method. Thanks to Boud Roukema for reporting the problem and discussing the details and testing the fix.
2022-09-16CMake: Clarify a comment about Windows symlinks without file extension.Jia Tan1-4/+3
2022-09-16CMake: Update for liblzma_*.map files and fix wrong common_w32res.rc dep.Lasse Collin1-5/+19
The previous commit split liblzma.map into liblzma_linux.map and liblzma_generic.map. This commit updates the CMake build for those. common_w32res.rc dependency was listed under Linux/FreeBSD while obviously it belongs to Windows when building a DLL.
2022-09-16CMake: Add xz symlinks.Lasse Collin1-1/+37
These are a minor thing especially since the xz build has some real problems still like lack of large file support on 32-bit systems but I'll commit this since the code exists. Thanks to Jia Tan.
2022-09-16CMake: Put xz man page install under if(UNIX) like is for xzdec.Lasse Collin1-3/+5
Thanks to Jia Tan.
2022-09-16Translations: Add Turkish translation.Lasse Collin2-0/+978
2022-08-18Build: Include the CMake files in the distribution.Lasse Collin1-0/+2
This was supposed to be done in 2020 with 5.2.5 release already but it was noticed only today. 5.2.5 and 5.2.6 even mention experiemental CMake support in the NEWS entries. Thanks to Olivier B. for reporting the problem.
2022-08-18Windows: Fix broken liblzma.dll build with Visual Studio project files.Lasse Collin3-0/+18
The bug was introduced in 352ba2d69af2136bc814aa1df1a132559d445616 "Windows: Fix building of resource files when config.h isn't used." That commit fixed liblzma.dll build with CMake while keeping it working with Autotools on Windows but the VS project files were forgotten. I haven't tested these changes. Thanks to Olivier B. for reporting the bug and for the initial patch.
2022-08-12Bump version and soname for 5.2.6.larhzu/v5.2.6Lasse Collin2-2/+2
2022-08-12Add NEWS for 5.2.6.Lasse Collin1-0/+226
2022-08-12Add Jia Tan to AUTHORS.Lasse Collin1-1/+1
2022-07-25Translations: Change the copyright comment string to use with po4a.Lasse Collin1-1/+1
This affects the second line in po4a/xz-man.pot. The man pages of xzdiff, xzgrep, and xzmore are from GNU gzip and under GNU GPLv2+ while the rest of the man pages are in the public domain.
2022-07-25liblzma: Refactor lzma_mf_is_supported() to use a switch-statement.Jia Tan1-18/+14
2022-07-25Build: Don't allow empty LIST in --enable-match-finders=LIST.Jia Tan1-0/+4
It's enforced only when a match finder is needed, that is, when LZMA1 or LZMA2 encoder is enabled.
2022-07-24xz: Make --keep accept symlinks, hardlinks, and setuid/setgid/sticky.Lasse Collin2-5/+20
Previously this required using --force but that has other effects too which might be undesirable. Changing the behavior of --keep has a small risk of breaking existing scripts but since this is a fairly special corner case I expect the likehood of breakage to be low enough. I think the new behavior is more logical. The only reason for the old behavior was to be consistent with gzip and bzip2. Thanks to Vincent Lefevre and Sebastian Andrzej Siewior.
2022-07-24xzgrep man page: Document exit statuses.Lasse Collin1-1/+14
2022-07-24xzgrep: Improve error handling, especially signals.Lasse Collin1-19/+53
xzgrep wouldn't exit on SIGPIPE or SIGQUIT when it clearly should have. It's quite possible that it's not perfect still but at least it's much better. If multiple exit statuses compete, now it tries to pick the largest of value. Some comments were added. The exit status handling of signals is still broken if the shell uses values larger than 255 in $? to indicate that a process died due to a signal ***and*** their "exit" command doesn't take this into account. This seems to work well with the ksh and yash versions I tried. However, there is a report in gzip/zgrep that OpenSolaris 5.11 (not 5.10) has a problem with "exit" truncating the argument to 8 bits: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22900#25 Such a bug would break xzgrep but I didn't add a workaround at least for now. 5.11 is old and I don't know if the problem exists in modern descendants, or if the problem exists in other ksh implementations in use.
2022-07-24xzgrep: Make the fix for ZDI-CAN-16587 more robust.Lasse Collin1-1/+4
I don't know if this can make a difference in the real world but it looked kind of suspicious (what happens with sed implementations that cannot process very long lines?). At least this commit shouldn't make it worse.
2022-07-24xzgrep: Use grep -H --label when available (GNU, *BSDs).Lasse Collin1-0/+21
It avoids the use of sed for prefixing filenames to output lines. Using sed for that is slower and prone to security bugs so now the sed method is only used as a fallback. This also fixes an actual bug: When grepping a binary file, GNU grep nowadays prints its diagnostics to stderr instead of stdout and thus the sed-method for prefixing the filename doesn't work. So with this commit grepping binary files gives reasonable output with GNU grep now. This was inspired by zgrep but the implementation is different.
2022-07-24xzgrep: Use -e to specify the pattern to grep.Lasse Collin1-8/+4
Now we don't need the separate test for adding the -q option as it can be added directly in the two places where it's needed.
2022-07-24Scripts: Use printf instead of echo in a few places.Lasse Collin4-11/+11
It's a good habbit as echo has some portability corner cases when the string contents can be anything.
2022-07-24xzgrep: Add more LC_ALL=C to avoid bugs with multibyte characters.Lasse Collin1-6/+8
Also replace one use of expr with printf. The rationale for LC_ALL=C was already mentioned in 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 that fixed a security issue. However, unrelated uses weren't changed in that commit yet. POSIX says that with sed and such tools one should use LC_ALL=C to ensure predictable behavior when strings contain byte sequences that aren't valid multibyte characters in the current locale. See under "Application usage" in here: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html With GNU sed invalid multibyte strings would work without this; it's documented in its Texinfo manual. Some other implementations aren't so forgiving.
2022-07-24xzgrep: Fix parsing of certain options.Lasse Collin1-2/+17
Fix handling of "xzgrep -25 foo" (in GNU grep "grep -25 foo" is an alias for "grep -C25 foo"). xzgrep would treat "foo" as filename instead of as a pattern. This bug was fixed in zgrep in gzip in 2012. Add -E, -F, -G, and -P to the "no argument required" list. Add -X to "argument required" list. It is an intentionally-undocumented GNU grep option so this isn't an important option for xzgrep but it seems that other grep implementations (well, those that I checked) don't support -X so I hope this change is an improvement still. grep -d (grep --directories=ACTION) requires an argument. In contrast to zgrep, I kept -d in the "no argument required" list because it's not supported in xzgrep (or zgrep). This way "xzgrep -d" gives an error about option being unsupported instead of telling that it requires an argument. Both zgrep and xzgrep tell that it's unsupported if an argument is specified. Add comments.
2022-07-24Tests: Add the .lzma files to test_files.sh.Lasse Collin1-0/+20
2022-07-24Tests: Add .lzma test files.Lasse Collin9-13/+50
2022-07-24liblzma: Rename a variable and improve a comment.Lasse Collin1-4/+9
2022-07-24Update THANKS.Lasse Collin1-0/+1
2022-07-24liblzma: Add optional autodetection of LZMA end marker.Lasse Collin6-38/+94
Turns out that this is needed for .lzma files as the spec in LZMA SDK says that end marker may be present even if the size is stored in the header. Such files are rare but exist in the real world. The code in liblzma is so old that the spec didn't exist in LZMA SDK back then and I had understood that such files weren't possible (the lzma tool in LZMA SDK didn't create such files). This modifies the internal API so that LZMA decoder can be told if EOPM is allowed even when the uncompressed size is known. It's allowed with .lzma and not with other uses. Thanks to Karl Beldan for reporting the problem.
2022-07-24Tests: Add test file good-1-empty-bcj-lzma2.xz.Lasse Collin2-0/+5
This is from test_bcj_exact_size.c. It's good to have it as a standalone file.
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12xzgrep: Fix escaping of malicious filenames (ZDI-CAN-16587).Lasse Collin1-8/+12
Malicious filenames can make xzgrep to write to arbitrary files or (with a GNU sed extension) lead to arbitrary code execution. xzgrep from XZ Utils versions up to and including 5.2.5 are affected. 5.3.1alpha and 5.3.2alpha are affected as well. This patch works for all of them. This bug was inherited from gzip's zgrep. gzip 1.12 includes a fix for zgrep. The issue with the old sed script is that with multiple newlines, the N-command will read the second line of input, then the s-commands will be skipped because it's not the end of the file yet, then a new sed cycle starts and the pattern space is printed and emptied. So only the last line or two get escaped. One way to fix this would be to read all lines into the pattern space first. However, the included fix is even simpler: All lines except the last line get a backslash appended at the end. To ensure that shell command substitution doesn't eat a possible trailing newline, a colon is appended to the filename before escaping. The colon is later used to separate the filename from the grep output so it is fine to add it here instead of a few lines later. The old code also wasn't POSIX compliant as it used \n in the replacement section of the s-command. Using \<newline> is the POSIX compatible method. LC_ALL=C was added to the two critical sed commands. POSIX sed manual recommends it when using sed to manipulate pathnames because in other locales invalid multibyte sequences might cause issues with some sed implementations. In case of GNU sed, these particular sed scripts wouldn't have such problems but some other scripts could have, see: info '(sed)Locale Considerations' This vulnerability was discovered by: cleemy desu wayo working with Trend Micro Zero Day Initiative Thanks to Jim Meyering and Paul Eggert discussing the different ways to fix this and for coordinating the patch release schedule with gzip.
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12liblzma: Fix docs: lzma_block_decoder() cannot return LZMA_UNSUPPORTED_CHECK.Lasse Collin1-3/+0
If Check is unsupported, it will be silently ignored. It's the caller's job to handle it.
2022-07-12liblzma: Index hash: Change return value type of hash_append() to void.Lasse Collin1-6/+5
2022-07-12liblzma: Minor addition to lzma_vli_size() API doc.Lasse Collin1-0/+2
Thanks to Jia Tan.
2022-07-12liblzma: Check the return value of lzma_index_append() in threaded encoder.Lasse Collin1-2/+5
If lzma_index_append() failed (most likely memory allocation failure) it could have gone unnoticed and the resulting .xz file would have an incorrect Index. Decompressing such a file would produce the correct uncompressed data but then an error would occur when verifying the Index field.
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12liblzma: Use non-executable stack on FreeBSD as on LinuxEd Maste2-4/+4
2022-07-12liblzma: Make Block decoder catch certain types of errors better.Lasse Collin1-25/+54
Now it limits the input and output buffer sizes that are passed to a raw decoder. This way there's no need to check if the sizes can grow too big or overflow when updating Compressed Size and Uncompressed Size counts. This also means that a corrupt file cannot cause the raw decoder to process useless extra input or output that would exceed the size info in Block Header (and thus cause LZMA_DATA_ERROR anyway). More importantly, now the size information is verified more carefully in case raw decoder returns LZMA_OK. This doesn't really matter with the current single-threaded .xz decoder as the errors would be detected slightly later anyway. But this helps avoiding corner cases in the upcoming threaded decompressor, and it might help other Block decoder uses outside liblzma too. The test files bad-1-lzma2-{9,10,11}.xz test these conditions. With the single-threaded .xz decoder the only difference is that LZMA_DATA_ERROR is detected in a difference place now.
2022-07-12Tests: Add bad-1-lzma2-11.xz.Lasse Collin2-0/+5
2022-07-12Translations: Fix po4a failure with the French man page translations.Lasse Collin1-2/+4
Thanks to Mario Blättermann for the patch.
2022-07-12Translations: Add French translation of man pages.Lasse Collin2-1/+3542
This matches xz-utils 5.2.5-2 in Debian. The translation was done by "bubu", proofread by the debian-l10n-french mailing list contributors, and submitted to me on the xz-devel mailing list by Jean-Pierre Giraud. Thanks to everyone!
2022-07-12liblzma: Add NULL checks to LZMA and LZMA2 properties encoders.jiat752-0/+6
Previously lzma_lzma_props_encode() and lzma_lzma2_props_encode() assumed that the options pointers must be non-NULL because the with these filters the API says it must never be NULL. It is good to do these checks anyway.
2022-07-12CMake: Keep compatible with Windows 95 for 32-bit build.huangqinjin1-1/+11
2022-07-12xzgrep: Update man page timestamp.Lasse Collin1-1/+1
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12xzgrep: use `grep -E/-F` instead of `egrep` and `fgrep`Ville Skyttä2-6/+6
`egrep` and `fgrep` have been deprecated in GNU grep since 2007, and in current post 3.7 Git they have been made to emit obsolescence warnings: https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a9515624709865d480e3142fd959bccd1c9372d1
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12xz: Change the coding style of the previous commit.Lasse Collin1-5/+6
It isn't any better now but it's consistent with the rest of the code base.
2022-07-12xz: Avoid fchown(2) failure.Alexander Bluhm1-1/+7
OpenBSD does not allow to change the group of a file if the user does not belong to this group. In contrast to Linux, OpenBSD also fails if the new group is the same as the old one. Do not call fchown(2) in this case, it would change nothing anyway. This fixes an issue with Perl Alien::Build module. https://github.com/PerlAlien/Alien-Build/issues/62
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12liblzma: Use _MSVC_LANG to detect when "noexcept" can be used with MSVC.Lasse Collin1-1/+2
By default, MSVC always sets __cplusplus to 199711L. The real C++ standard version is available in _MSVC_LANG (or one could use /Zc:__cplusplus to set __cplusplus correctly). Fixes <https://sourceforge.net/p/lzmautils/discussion/708858/thread/f6bc3b108a/>. Thanks to Dan Weiss.
2022-07-12xzdiff: Update the man page about the exit status.Lasse Collin1-2/+2
This was forgotten from 194029ffaf74282a81f0c299c07f73caca3232ca.
2022-07-12xzless: Fix less(1) version detection when it contains a dot.Lasse Collin1-1/+1
Sometimes the version number from "less -V" contains a dot, sometimes not. xzless failed detect the version number when it does contain a dot. This fixes it. Thanks to nick87720z for reporting this. Apparently it had been reported here <https://bugs.gentoo.org/489362> in 2013.
2022-07-12xz: Document the special memlimit case of 2000 MiB on MIPS32.Lasse Collin1-2/+6
See commit 95806a8a52ae57bddf6c77dfd19cf7938a92e040.
2022-07-12Update THANKS.Lasse Collin1-0/+2
2022-07-12Reduce maximum possible memory limit on MIPS32Ivan A. Melnikov1-0/+6
Due to architectural limitations, address space available to a single userspace process on MIPS32 is limited to 2 GiB, not 4, even on systems that have more physical RAM -- e.g. 64-bit systems with 32-bit userspace, or systems that use XPA (an extension similar to x86's PAE). So, for MIPS32, we have to impose stronger memory limits. I've chosen 2000MiB to give the process some headroom.
2022-07-12CMake: Use interface library for better FindLibLZMA compatibility.Lasse Collin1-2/+9
https://www.mail-archive.com/xz-devel@tukaani.org/msg00446.html Thanks to Markus Rickert.
2022-07-12CMake: Try to improve compatibility with the FindLibLZMA module.Lasse Collin1-8/+13
The naming conflict with FindLibLZMA module gets worse. Not avoiding it in the first place was stupid. Normally find_package(LibLZMA) will use the module and find_package(liblzma 5.2.5 REQUIRED CONFIG) will use the config file even with a case insensitive file system. However, if CMAKE_FIND_PACKAGE_PREFER_CONFIG is TRUE and the file system is case insensitive, find_package(LibLZMA) will find our liblzma config file instead of using FindLibLZMA module. One big problem with this is that FindLibLZMA uses LibLZMA::LibLZMA and we use liblzma::liblzma as the target name. With target names CMake happens to be case sensitive. To workaround this, this commit adds add_library(LibLZMA::LibLZMA ALIAS liblzma::liblzma) to the config file. Then both spellings work. To make the behavior consistent between case sensitive and insensitive file systems, the config and related files are renamed from liblzmaConfig.cmake to liblzma-config.cmake style. With this style CMake looks for lowercase version of the package name so find_package(LiBLzmA 5.2.5 REQUIRED CONFIG) will work to find our config file. There are other differences between our config file and FindLibLZMA so it's still possible that things break for reasons other than the spelling of the target name. Hopefully those situations aren't too common. When the config file is available, it should always give as good or better results as FindLibLZMA so this commit doesn't affect the recommendation to use find_package(liblzma 5.2.5 REQUIRED CONFIG) which explicitly avoids FindLibLZMA. Thanks to Markus Rickert.
2022-07-12Tests: Add bad-1-lzma2-10.xz and also modify -9.xz.Lasse Collin3-2/+9
2022-07-12Tests: Add bad-1-lzma2-9.xz.Lasse Collin2-0/+4
2022-07-12Tests: Add bad-1-check-crc32-2.xz.Lasse Collin2-0/+7
2022-07-12Scripts: Add zstd support to xzdiff.Lasse Collin2-7/+15
2022-07-12Scripts: Fix exit status of xzgrep.Lasse Collin1-7/+13
Omit the -q option from xz, gzip, and bzip2. With xz this shouldn't matter. With gzip it's important because -q makes gzip replace SIGPIPE with exit status 2. With bzip2 it's important because with -q bzip2 is completely silent if input is corrupt while other decompressors still give an error message. Avoiding exit status 2 from gzip is important because bzip2 uses exit status 2 to indicate corrupt input. Before this commit xzgrep didn't recognize corrupt .bz2 files because xzgrep was treating exit status 2 as SIGPIPE for gzip compatibility. zstd still needs -q because otherwise it is noisy in normal operation. The code to detect real SIGPIPE didn't check if the exit status was due to a signal (>= 128) and so could ignore some other exit status too.
2022-07-12Scripts: Fix exit status of xzdiff/xzcmp.Lasse Collin1-14/+21
This is a minor fix since this affects only the situation when the files differ and the exit status is something else than 0. In such case there could be SIGPIPE from a decompression tool and that would result in exit status of 2 from xzdiff/xzcmp while the correct behavior would be to return 1 or whatever else diff or cmp may have returned. This commit omits the -q option from xz/gzip/bzip2/lzop arguments. I'm not sure why the -q was used in the first place, perhaps it hides warnings in some situation that I cannot see at the moment. Hopefully the removal won't introduce a new bug. With gzip the -q option was harmful because it made gzip return 2 instead of >= 128 with SIGPIPE. Ignoring exit status 2 (warning from gzip) isn't practical because bzip2 uses exit status 2 to indicate corrupt input file. It's better if SIGPIPE results in exit status >= 128. With bzip2 the removal of -q seems to be good because with -q it prints nothing if input is corrupt. The other tools aren't silent in this situation even with -q. On the other hand, if zstd support is added, it will need -q since otherwise it's noisy in normal situations. Thanks to Étienne Mollier and Sebastian Andrzej Siewior.
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12liblzma: Enable Intel CET in x86 CRC assembly codesH.J. Lu2-0/+18
When Intel CET is enabled, we need to include <cet.h> in assembly codes to mark Intel CET support and add _CET_ENDBR to indirect jump targets. Tested on Intel Tiger Lake under CET enabled Linux.
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12Build: Don't build bundles on Apple OSes.Lasse Collin1-0/+3
Thanks to Daniel Packard.
2022-07-12Update THANKS.Lasse Collin1-0/+1
2022-07-12Scripts: Add zstd support to xzgrep.Adam Borowski2-3/+7
Thanks to Adam Borowski.
2022-07-12CMake: Fix compatibility with CMake 3.13.Lasse Collin3-5/+5
The syntax "if(DEFINED CACHE{FOO})" requires CMake 3.14. In some other places the code treats the cache variables like normal variables already (${FOO} or if(FOO) is used, not ${CACHE{FOO}). Thanks to ygrek for reporting the bug on IRC.
2022-07-12Update THANKS.Lasse Collin1-1/+3
2022-07-12xz: Avoid unneeded \f escapes on the man page.Lasse Collin1-9/+22
I don't want to use \c in macro arguments but groff_man(7) suggests that \f has better portability. \f would be needed for the .TP strings for portability reasons anyway. Thanks to Bjarni Ingi Gislason.
2022-07-12xz: Use non-breaking spaces when intentionally using more than one space.Lasse Collin1-1/+1
This silences some style checker warnings. Seems that spaces in the beginning of a line don't need this treatment. Thanks to Bjarni Ingi Gislason.
2022-07-12xz: Protect the ellipsis (...) on the man page with \&.Lasse Collin1-2/+2
This does it only when ... appears outside macro calls. Thanks to Bjarni Ingi Gislason.
2022-07-12xz: Avoid the abbreviation "e.g." on the man page.Lasse Collin1-33/+33
A few are simply omitted, most are converted to "for example" and surrounded with commas. Sounds like that this is better style, for example, man-pages(7) recommends avoiding such abbreviations except in parenthesis. Thanks to Bjarni Ingi Gislason.
2022-07-12xz man page: Change \- (minus) to \(en (en-dash) for a numeric range.Lasse Collin1-8/+8
Docs of ancient troff/nroff mention \(em (em-dash) but not \(en and \- was used for both minus and en-dash. I don't know how portable \(en is nowadays but it can be changed back if someone complains. At least GNU groff and OpenBSD's mandoc support it. Thanks to Bjarni Ingi Gislason for the patch.
2022-07-12Windows: Fix building of resource files when config.h isn't used.Lasse Collin1-1/+3
Now CMake + Visual Studio works for building liblzma.dll. Thanks to Markus Rickert.
2022-07-12src/scripts/xzgrep.1: Filenames to xzgrep are optional.Lasse Collin1-1/+1
xzgrep --help was correct already.
2022-07-12Translations: Add Portuguese translation.Lasse Collin2-0/+1002
Jia Tan made white-space changes and also changed "Language: pt_BR\n" to pt. The translator wasn't reached so I'm hoping these changes are OK and will commit it without translator's approval. Thanks to Pedro Albuquerque and Jia Tan.
2022-07-12src/script/xzgrep.1: Remove superfluous '.RB'Bjarni Ingi Gislason1-6/+6
Output is from: test-groff -b -e -mandoc -T utf8 -rF0 -t -w w -z [ "test-groff" is a developmental version of "groff" ] Input file is ./src/scripts/xzgrep.1 <src/scripts/xzgrep.1>:20 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:23 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:26 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:29 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:32 (macro RB): only 1 argument, but more are expected "abc..." does not mean the same as "abc ...". The output from nroff and troff is unchanged except for the space between "file" and "...". Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2022-07-12xzgrep.1: Delete superfluous '.PP'Bjarni Ingi Gislason1-1/+0
Summary: mandoc -T lint xzgrep.1 : mandoc: xzgrep.1:79:2: WARNING: skipping paragraph macro: PP empty There is no change in the output of "nroff" and "troff". Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2022-07-12src/xz/xz.1: Correct misused two-fonts macrosBjarni Ingi Gislason1-5/+5
Output is from: test-groff -b -e -mandoc -T utf8 -rF0 -t -w w -z [ "test-groff" is a developmental version of "groff" ] Input file is ./src/xz/xz.1 <src/xz/xz.1>:408 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1009 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1743 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1920 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:2213 (macro BR): only 1 argument, but more are expected Output from nroff and troff is unchanged, except for a font change of a full stop (.). Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2022-07-10Translations: Add Serbian translation.Lasse Collin2-0/+988
Quite a few white-space changes were made by Jia Tan to make this look good. Contacting the translator didn't succeed so I'm committing this without getting translator's approval. Thanks to Мирослав Николић (Miroslav Nikolic) and Jia Tan.
2022-07-04Translations: Add Swedish translation.Lasse Collin2-0/+984
Thanks to Sebastian Rasmussen and Jia Tan.
2022-07-04Translations: Add Esperanto translation.Lasse Collin2-0/+985
Thanks to Keith Bowes and Jia Tan.
2022-07-01Translations: Add Catalan translation.Lasse Collin2-0/+1077
Thanks to Jordi Mas and Jia Tan.
2022-06-30Translations: Add Ukrainian translation.Lasse Collin2-0/+997
Thanks to Yuri Chornoivan and Jia Tan.
2022-06-30Translators: Add Romanian translation.Lasse Collin2-0/+1017
Thanks to Remus-Gabriel Chelu and Jia Tan.
2022-06-29Translations: Update Brazilian Portuguese translation.Lasse Collin1-94/+92
One msgstr was changed. The diff is long due to changes in the source code line numbers in the comments. Thanks to Rafael Fontenelle.
2022-06-29Translations: Add Croatian translation.Lasse Collin2-0/+988
Thanks to Božidar Putanec and Jia Tan.
2022-06-29Translations: Add Spanish translation.Lasse Collin2-0/+985
Thanks to Cristian Othón Martínez Vera and Jia Tan.
2022-06-29Translations: Add Korean translation.Lasse Collin2-0/+973
Thanks to Seong-ho Cho and Jia Tan.
2020-03-23v5.2-specific typo fix from fossies.org.Lasse Collin1-1/+1
2020-03-23Typo fixes from fossies.org.Lasse Collin6-7/+7
https://fossies.org/linux/misc/xz-5.2.5.tar.xz/codespell.html
2020-03-17Bump version and soname for 5.2.5.larhzu/v5.2.5Lasse Collin2-2/+2
2020-03-17Update NEWS for 5.2.5.Lasse Collin1-0/+105
2020-03-16Translations: Rebuild cs.po to avoid incorrect fuzzy strings.Lasse Collin1-270/+322
"make dist" updates the .po files and the fuzzy strings would result in multiple very wrong translations.
2020-03-16README: Update outdated sections.Lasse Collin1-10/+11
2020-03-16README: Mention that translatable strings will change after 5.2.x.Lasse Collin1-71/+3
2020-03-16README: Mention that man pages can be translated.Lasse Collin1-3/+4
2020-03-16Translations: Add partial Danish translation.Lasse Collin2-0/+897
I made a few minor white space changes without getting them approved by the Danish translation team.
2020-03-16Update INSTALL.generic from Automake 1.16.1.Lasse Collin1-159/+162
2020-03-15Update INSTALL for Windows and DOS and add preliminary info for z/OS.Lasse Collin1-10/+41
2020-03-15Build: Update m4/ax_pthread.m4 from Autoconf Archive (again).Lasse Collin1-102/+117
2020-03-11xz: Never use thousand separators in DJGPP builds.Lasse Collin1-2/+12
DJGPP 2.05 added support for thousands separators but it's broken at least under WinXP with Finnish locale that uses a non-breaking space as the thousands separator. Workaround by disabling thousands separators for DJGPP builds.
2020-03-11DOS: Update dos/Makefile for DJGPP 2.05.Lasse Collin1-3/+1
It doesn't need -fgnu89-inline like 2.04beta did.
2020-03-11DOS: Update instructions in dos/INSTALL.txt.Lasse Collin1-30/+29
2020-03-11DOS: Update config.h.Lasse Collin1-0/+8
The added defines assume GCC >= 4.8.
2020-03-11Translations: Add hu, zh_CN, and zh_TW.Lasse Collin4-0/+2907
I made a few white space changes to these without getting them approved by the translation teams. (I tried to contact the hu and zh_TW teams but didn't succeed. I didn't contact the zh_CN team.)
2020-03-11Translations: Update vi.po to match the file from the TP.Lasse Collin1-228/+179
The translated strings haven't been updated but word wrapping is different.
2020-03-11Translations: Add fi and pt_BR, and update de, fr, it, and pl.Lasse Collin7-746/+2697
The German translation isn't identical to the file in the Translation Project but the changes (white space changes only) were approved by the translator Mario Blättermann.
2020-03-11Update THANKS.Lasse Collin1-0/+1
2020-03-11Build: Add very limited experimental CMake support.Lasse Collin7-0/+1174
This version matches CMake files in the master branch (commit 265daa873c0d871f5f23f9b56e133a6f20045a0a) except that this omits two source files that aren't in v5.2 and in the beginning of CMakeLists.txt the first paragraph in the comment is slightly different to point out possible issues in building shared liblzma.
2020-03-11Build: Add support for --no-po4a option to autogen.sh.Lasse Collin1-1/+10
Normally, if po4a isn't available, autogen.sh will return with non-zero exit status. The option --no-po4a can be useful when one knows that po4a isn't available but wants autogen.sh to still return with zero exit status.
2020-03-11Update m4/.gitignore.Lasse Collin1-0/+1
2020-03-11liblzma: Remove unneeded <sys/types.h> from fastpos_tablegen.c.Lasse Collin1-1/+0
This file only generates fastpos_table.c. It isn't built as a part of liblzma.
2020-03-11Use defined(__GNUC__) before __GNUC__ in preprocessor lines.Lasse Collin2-3/+5
This should silence the equivalent of -Wundef in compilers that don't define __GNUC__.
2020-03-11liblzma: Add more uses of lzma_memcmplen() to the normal mode of LZMA.Lasse Collin1-6/+10
This gives a tiny encoder speed improvement. This could have been done in 2014 after the commit 544aaa3d13554e8640f9caf7db717a96360ec0f6 but it was forgotten.
2020-03-11Build: Add visibility.m4 from gnulib.Lasse Collin2-1/+77
Appears that this file used to get included as a side effect of gettext. After the change to gettext version requirements this file no longer got copied to the package and so the build was broken.
2020-03-11xz: Silence a warning when sig_atomic_t is long int.Lasse Collin1-1/+1
It can be true at least on z/OS.
2020-03-11xz: Avoid unneeded access of a volatile variable.Lasse Collin1-1/+1
2020-03-11tuklib_integer.m4: Optimize the check order.Lasse Collin1-27/+29
The __builtin byteswapping is the preferred one so check for it first.
2020-03-11tuklib_exit: Add missing header.Lasse Collin1-0/+1
strerror() needs <string.h> which happened to be included via tuklib_common.h -> tuklib_config.h -> sysdefs.h if HAVE_CONFIG_H was defined. This wasn't tested without config.h before so it had worked fine.
2020-03-11sysdefs.h: Omit the conditionals around string.h and limits.h.Lasse Collin1-6/+2
string.h is used unconditionally elsewhere in the project and configure has always stopped if limits.h is missing, so these headers must have been always available even on the weirdest systems.
2020-03-11Build: Bump Autoconf and Libtool version requirements.Lasse Collin1-2/+2
There is no specific reason for this other than blocking the most ancient versions. These are still old: Autoconf 2.69 (2012) Automake 1.12 (2012) gettext 0.19.6 (2015) Libtool 2.4 (2010)
2020-03-11Build: Use AM_GNU_GETTEXT_REQUIRE_VERSION and require 0.19.6.Lasse Collin1-1/+5
This bumps the version requirement from 0.19 (from 2014) to 0.19.6 (2015). Using only the old AM_GNU_GETTEXT_VERSION results in old gettext infrastructure being placed in the package. By using both macros we get the latest gettext files while the other programs in the Autotools family can still see the old macro.
2020-03-11Translations: Add German translation of the man pages.Lasse Collin2-1/+5533
Thanks to Mario Blättermann.
2020-03-11Build: Add support for translated man pages using po4a.Lasse Collin8-45/+197
The dependency on po4a is optional. It's never required to install the translated man pages when xz is built from a release tarball. If po4a is missing when building from xz.git, the translated man pages won't be generated but otherwise the build will work normally. The translations are only updated automatically by autogen.sh and by "make mydist". This makes it easy to keep po4a as an optional dependency and ensures that I won't forget to put updated translations to a release tarball. The translated man pages aren't installed if --disable-nls is used. The installation of translated man pages abuses Automake internals by calling "install-man" with redefined dist_man_MANS and man_MANS. This makes the hairy script code slightly less hairy. If it breaks some day, this code needs to be fixed; don't blame Automake developers. Also, this adds more quotes to the existing shell script code in the Makefile.am "-hook"s.
2020-02-06Update THANKS (sync with the master branch).Lasse Collin1-0/+3
2020-02-06Update tests/.gitignore.Lasse Collin1-0/+4
2020-02-06Update m4/.gitignore.Lasse Collin1-0/+1
2020-02-06Update THANKS.Lasse Collin1-0/+1
2020-02-05xz: Comment out annoying sandboxing messages.Lasse Collin1-3/+7
2020-02-05Build: Workaround a POSIX shell detection problem on Solaris.Lasse Collin1-0/+10
I don't know if the problem is in gnulib's gl_POSIX_SHELL macro or if xzgrep does something that isn't in POSIX. The workaround adds a special case for Solaris: if /usr/xpg4/bin/sh exists and gl_cv_posix_shell wasn't overriden on the configure command line, use that shell for xzgrep and other scripts. That shell is known to work and exists on most Solaris systems.
2020-02-05Build: Update m4/ax_pthread.m4 from Autoconf Archive.Lasse Collin1-119/+279
2020-02-05xz: Limit --memlimit-compress to at most 4020 MiB for 32-bit xz.Lasse Collin2-2/+51
See the code comment for reasoning. It's far from perfect but hopefully good enough for certain cases while hopefully doing nothing bad in other situations. At presets -5 ... -9, 4020 MiB vs. 4096 MiB makes no difference on how xz scales down the number of threads. The limit has to be a few MiB below 4096 MiB because otherwise things like "xz --lzma2=dict=500MiB" won't scale down the dict size enough and xz cannot allocate enough memory. With "ulimit -v $((4096 * 1024))" on x86-64, the limit in xz had to be no more than 4085 MiB. Some safety margin is good though. This is hack but it should be useful when running 32-bit xz on a 64-bit kernel that gives full 4 GiB address space to xz. Hopefully this is enough to solve this: https://bugzilla.redhat.com/show_bug.cgi?id=1196786 FreeBSD has a patch that limits the result in tuklib_physmem() to SIZE_MAX on 32-bit systems. While I think it's not the way to do it, the results on --memlimit-compress have been good. This commit should achieve practically identical results for compression while leaving decompression and tuklib_physmem() and thus lzma_physmem() unaffected.
2020-02-05xz: Set the --flush-timeout deadline when the first input byte arrives.Lasse Collin3-7/+6
xz --flush-timeout=2000, old version: 1. xz is started. The next flush will happen after two seconds. 2. No input for one second. 3. A burst of a few kilobytes of input. 4. No input for one second. 5. Two seconds have passed and flushing starts. The first second counted towards the flush-timeout even though there was no pending data. This can cause flushing to occur more often than needed. xz --flush-timeout=2000, after this commit: 1. xz is started. 2. No input for one second. 3. A burst of a few kilobytes of input. The next flush will happen after two seconds counted from the time when the first bytes of the burst were read. 4. No input for one second. 5. No input for another second. 6. Two seconds have passed and flushing starts.
2020-02-05xz: Move flush_needed from mytime.h to file_pair struct in file_io.h.Lasse Collin5-9/+7
2020-02-05xz: coder.c: Make writing output a separate function.Lasse Collin1-13/+17
The same code sequence repeats so it's nicer as a separate function. Note that in one case there was no test for opt_mode != MODE_TEST, but that was only because that condition would always be true, so this commit doesn't change the behavior there.
2020-02-05xz: Fix semi-busy-waiting in xz --flush-timeout.Lasse Collin3-4/+19
When input blocked, xz --flush-timeout=1 would wake up every millisecond and initiate flushing which would have nothing to flush and thus would just waste CPU time. The fix disables the timeout when no input has been seen since the previous flush.
2020-02-05xz: Refactor io_read() a bit.Lasse Collin1-9/+8
2020-02-05xz: Update a comment in file_io.h.Lasse Collin1-1/+4
2020-02-05xz: Move the setting of flush_needed in file_io.c to a nicer location.Lasse Collin1-4/+2
2020-02-05xz: Enable Capsicum sandboxing by default if available.Lasse Collin2-10/+4
It has been enabled in FreeBSD for a while and reported to work fine. Thanks to Xin Li.
2019-12-31Rename unaligned_read32ne to read32ne, and similarly for the others.Lasse Collin15-62/+54
2019-12-31Rename read32ne to aligned_read32ne, and similarly for the others.Lasse Collin3-32/+32
Using the aligned methods requires more care to ensure that the address really is aligned, so it's nicer if the aligned methods are prefixed. The next commit will remove the unaligned_ prefix from the unaligned methods which in liblzma are used in more places than the aligned ones.
2019-12-31Revise tuklib_integer.h and .m4.Lasse Collin2-217/+314
Add a configure option --enable-unsafe-type-punning to get the old non-conforming memory access methods. It can be useful with old compilers or in some other less typical situations but shouldn't normally be used. Omit the packed struct trick for unaligned access. While it's best in some cases, this is simpler. If the memcpy trick doesn't work, one can request unsafe type punning from configure. Because CRC32/CRC64 code needs fast aligned reads, if no very safe way to do it is found, type punning is used as a fallback. This sucks but since it currently works in practice, it seems to be the least bad option. It's never needed with GCC >= 4.7 or Clang >= 3.6 since these support __builtin_assume_aligned and thus fast aligned access can be done with the memcpy trick. Other things: - Support GCC/Clang __builtin_bswapXX - Cleaner bswap fallback macros - Minor cleanups
2019-12-31Tests: Hopefully fix test_check.c to work on EBCDIC systems.Lasse Collin1-2/+7
Thanks to Daniel Richard G.
2019-12-31Scripts: Put /usr/xpg4/bin to the beginning of PATH on Solaris.Lasse Collin6-8/+65
This adds a configure option --enable-path-for-scripts=PREFIX which defaults to empty except on Solaris it is /usr/xpg4/bin to make POSIX grep and others available. The Solaris case had been documented in INSTALL with a manual fix but it's better to do this automatically since it is needed on most Solaris systems anyway. Thanks to Daniel Richard G.
2019-12-31Fix comment typos in tuklib_mbstr* files.Lasse Collin3-3/+3
2019-12-31Add missing include to tuklib_mbstr_width.c.Lasse Collin1-0/+1
It didn't matter in XZ Utils because sysdefs.h includes string.h anyway.
2019-12-31liblzma: Fix a buggy comment.Lasse Collin1-1/+1
2019-12-31configure.ac: Fix a typo in a comment.Lasse Collin1-1/+1
2019-12-31Tests: Silence warnings from clang -Wassign-enum.Lasse Collin2-2/+3
Also changed 999 to 99 so it fits even if lzma_check happened to be 8 bits wide.
2019-12-31liblzma: Add a comment.Lasse Collin1-1/+1