aboutsummaryrefslogtreecommitdiff
path: root/src (follow)
AgeCommit message (Collapse)AuthorFilesLines
2022-09-19liblzma: Add experimental ARM64 BCJ filter with a temporary Filter ID.Lasse Collin9-1/+308
That is, the Filter ID will be changed once the design is final. The current version will be removed. So files created with the tempoary Filter ID won't be supported in the future.
2022-09-17liblzma: Simple/BCJ filters: Allow disabling generic BCJ options.Lasse Collin8-9/+10
This will be needed for the ARM64 BCJ filter as it will use its own options struct.
2022-09-16xzgrep: Fix compatibility with old shells.Lasse Collin1-3/+3
Running the current xzgrep on Slackware 10.1 with GNU bash 3.00.15: xzgrep: line 231: syntax error near unexpected token `;;' On SCO OpenServer 5.0.7 with Korn Shell 93r: syntax error at line 231 : `;;' unexpected Turns out that some old shells don't like apostrophes (') inside command substitutions. For example, the following fails: x=$(echo foo # asdf'zxcv echo bar) printf '%s\n' "$x" The problem was introduced by commits 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 (2022-03-29), bd7b290f3fe4faeceb7d3497ed9bf2e6ed5e7dc5 (2022-07-18), and a648978b20495b7aa4a8b029c5a810b5ad9d08ff (2022-07-19). 5.2.6 is the only stable release that included this problem. Thanks to Kevin R. Bulgrien for reporting the problem on SCO OpenServer 5.0.7 and for providing the fix.
2022-09-09liblzma: lzma_filters_copy: Keep dest[] unmodified if an error occurs.Lasse Collin2-7/+15
lzma_stream_encoder() and lzma_stream_encoder_mt() always assumed this. Before this patch, failing lzma_filters_copy() could result in free(invalid_pointer) or invalid memory reads in stream_encoder.c or stream_encoder_mt.c. To trigger this, allocating memory for a filter options structure has to fail. These are tiny allocations so in practice they very rarely fail. Certain badness in the filter chain array could also make lzma_filters_copy() fail but both stream_encoder.c and stream_encoder_mt.c validate the filter chain before trying to copy it, so the crash cannot occur this way.
2022-09-08liblzma: lzma_index_append: Add missing integer overflow check.Jia Tan1-0/+4
The documentation in src/liblzma/api/lzma/index.h suggests that both the unpadded (compressed) size and the uncompressed size are checked for overflow, but only the unpadded size was checked. The uncompressed check is done first since that is more likely to occur than the unpadded or index field size overflows.
2022-09-08liblzma: Vaccinate against an ill patch from RHEL/CentOS 7.Lasse Collin9-14/+360
RHEL/CentOS 7 shipped with 5.1.2alpha, including the threaded encoder that is behind #ifdef LZMA_UNSTABLE in the API headers. In 5.1.2alpha these symbols are under XZ_5.1.2alpha in liblzma.map. API/ABI compatibility tracking isn't done between development releases so newer releases didn't have XZ_5.1.2alpha anymore. Later RHEL/CentOS 7 updated xz to 5.2.2 but they wanted to keep the exported symbols compatible with 5.1.2alpha. After checking the ABI changes it turned out that >= 5.2.0 ABI is backward compatible with the threaded encoder functions from 5.1.2alpha (but not vice versa as fixes and extensions to these functions were made between 5.1.2alpha and 5.2.0). In RHEL/CentOS 7, XZ Utils 5.2.2 was patched with xz-5.2.2-compat-libs.patch to modify liblzma.map: - XZ_5.1.2alpha was added with lzma_stream_encoder_mt and lzma_stream_encoder_mt_memusage. This matched XZ Utils 5.1.2alpha. - XZ_5.2 was replaced with XZ_5.2.2. It is clear that this was an error; the intention was to keep using XZ_5.2 (XZ_5.2.2 has never been used in XZ Utils). So XZ_5.2.2 lists all symbols that were listed under XZ_5.2 before the patch. lzma_stream_encoder_mt and _mt_memusage are included too so they are listed both here and under XZ_5.1.2alpha. The patch didn't add any __asm__(".symver ...") lines to the .c files. Thus the resulting liblzma.so exports the threaded encoder functions under XZ_5.1.2alpha only. Listing the two functions also under XZ_5.2.2 in liblzma.map has no effect without matching .symver lines. The lack of XZ_5.2 in RHEL/CentOS 7 means that binaries linked against unpatched XZ Utils 5.2.x won't run on RHEL/CentOS 7. This is unfortunate but this alone isn't too bad as the problem is contained within RHEL/CentOS 7 and doesn't affect users of other distributions. It could also be fixed internally in RHEL/CentOS 7. The second problem is more serious: In XZ Utils 5.2.2 the API headers don't have #ifdef LZMA_UNSTABLE for obvious reasons. This is true in RHEL/CentOS 7 version too. Thus now programs using new APIs can be compiled without an extra #define. However, the programs end up depending on symbol version XZ_5.1.2alpha (and possibly also XZ_5.2.2) instead of XZ_5.2 as they would with an unpatched XZ Utils 5.2.2. This means that such binaries won't run on other distributions shipping XZ Utils >= 5.2.0 as they don't provide XZ_5.1.2alpha or XZ_5.2.2; they only provide XZ_5.2 (and XZ_5.0). (This includes RHEL/CentOS 8 as the patch luckily isn't included there anymore with XZ Utils 5.2.4.) Binaries built by RHEL/CentOS 7 users get distributed and then people wonder why they don't run on some other distribution. Seems that people have found out about the patch and been copying it to some build scripts, seemingly curing the symptoms but actually spreading the illness further and outside RHEL/CentOS 7. The ill patch seems to be from late 2016 (RHEL 7.3) and in 2017 it had spread at least to EasyBuild. I heard about the events only recently. :-( This commit splits liblzma.map into two versions: one for GNU/Linux and another for other OSes that can use symbol versioning (FreeBSD, Solaris, maybe others). The Linux-specific file and the matching additions to .c files add full compatibility with binaries that have been built against a RHEL/CentOS-patched liblzma. Builds for OSes other than GNU/Linux won't get the vaccine as they should be immune to the problem (I really hope that no build script uses the RHEL/CentOS 7 patch outside GNU/Linux). The RHEL/CentOS compatibility symbols XZ_5.1.2alpha and XZ_5.2.2 are intentionally put *after* XZ_5.2 in liblzma_linux.map. This way if one forgets to #define HAVE_SYMBOL_VERSIONS_LINUX when building, the resulting liblzma.so.5 will have lzma_stream_encoder_mt@@XZ_5.2 since XZ_5.2 {...} is the first one that lists that function. Without HAVE_SYMBOL_VERSIONS_LINUX @XZ_5.1.2alpha and @XZ_5.2.2 will be missing but that's still a minor problem compared to only having lzma_stream_encoder_mt@@XZ_5.1.2alpha! The "local: *;" line was moved to XZ_5.0 so that it doesn't need to be moved around. It doesn't matter where it is put. Having two similar liblzma_*.map files is a bit silly as it is, at least for now, easily possible to generate the generic one from the Linux-specific file. But that adds extra steps and increases the risk of mistakes when supporting more than one build system. So I rather maintain two files in parallel and let validate_map.sh check that they are in sync when "make mydist" is run. This adds .symver lines for lzma_stream_encoder_mt@XZ_5.2.2 and lzma_stream_encoder_mt_memusage@XZ_5.2.2 even though these weren't exported by RHEL/CentOS 7 (only @@XZ_5.1.2alpha was for these two). I added these anyway because someone might misunderstand the RHEL/CentOS 7 patch and think that @XZ_5.2.2 (@@XZ_5.2.2) versions were exported too. At glance one could suggest using __typeof__ to copy the function prototypes when making aliases. However, this doesn't work trivially because __typeof__ won't copy attributes (lzma_nothrow, lzma_pure) and it won't change symbol visibility from hidden to default (done by LZMA_API()). Attributes could be copied with __copy__ attribute but that needs GCC 9 and a fallback method would be needed anyway. This uses __symver__ attribute with GCC >= 10 and __asm__(".symver ...") with everything else. The attribute method is required for LTO (-flto) support with GCC. Using -flto with GCC older than 10 is now broken on GNU/Linux and will not be fixed (can silently result in a broken liblzma build that has dangerously incorrect symbol versions). LTO builds with Clang seem to work with the traditional __asm__(".symver ...") method. Thanks to Boud Roukema for reporting the problem and discussing the details and testing the fix.
2022-08-22Bump version number for 5.3.3alpha.larhzu/v5.3.3alphaLasse Collin2-2/+2
2022-08-22xz: Try to clarify --memlimit-mt-decompress vs. --memlimit-compress.Lasse Collin1-12/+19
2022-08-19xz: Revise --info-memory output.Lasse Collin2-6/+27
The strings could be more descriptive but it's good to have some version of this committed now. --robot mode wasn't changed yet.
2022-08-19xz: Update the man page for threaded decompression and memlimits.Lasse Collin1-27/+121
This documents the changes made in commits 6c6da57ae2aa962aabde6892442227063d87e88c, cad299008cf73ec566f0662a9cf2b94f86a99659, and 898faa97287a756231c663a3ed5165672b417207. The --info-memory bit hasn't been finished yet even though it's already mentioned in this commit under --memlimit-mt-decompress and --threads.
2022-08-18liblzma: Threaded decoder: Improve LZMA_FAIL_FAST when LZMA_FINISH is used.Lasse Collin2-0/+48
It will now return LZMA_DATA_ERROR (not LZMA_OK or LZMA_BUF_ERROR) if LZMA_FINISH is used and there isn't enough input to finish decoding the Block Header or the Block. The use of LZMA_DATA_ERROR is simpler and the less risky than LZMA_BUF_ERROR but this might be changed before 5.4.0.
2022-07-25liblzma: Refactor lzma_mf_is_supported() to use a switch-statement.Jia Tan1-18/+14
2022-07-24xz: Update the man page that change to --keep will be in 5.2.6.Lasse Collin1-2/+2
2022-07-19xzgrep man page: Document exit statuses.Lasse Collin1-1/+14
2022-07-19xzgrep: Improve error handling, especially signals.Lasse Collin1-19/+53
xzgrep wouldn't exit on SIGPIPE or SIGQUIT when it clearly should have. It's quite possible that it's not perfect still but at least it's much better. If multiple exit statuses compete, now it tries to pick the largest of value. Some comments were added. The exit status handling of signals is still broken if the shell uses values larger than 255 in $? to indicate that a process died due to a signal ***and*** their "exit" command doesn't take this into account. This seems to work well with the ksh and yash versions I tried. However, there is a report in gzip/zgrep that OpenSolaris 5.11 (not 5.10) has a problem with "exit" truncating the argument to 8 bits: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22900#25 Such a bug would break xzgrep but I didn't add a workaround at least for now. 5.11 is old and I don't know if the problem exists in modern descendants, or if the problem exists in other ksh implementations in use.
2022-07-19xzgrep: Make the fix for ZDI-CAN-16587 more robust.Lasse Collin1-1/+4
I don't know if this can make a difference in the real world but it looked kind of suspicious (what happens with sed implementations that cannot process very long lines?). At least this commit shouldn't make it worse.
2022-07-18xzgrep: Use grep -H --label when available (GNU, *BSDs).Lasse Collin1-0/+21
It avoids the use of sed for prefixing filenames to output lines. Using sed for that is slower and prone to security bugs so now the sed method is only used as a fallback. This also fixes an actual bug: When grepping a binary file, GNU grep nowadays prints its diagnostics to stderr instead of stdout and thus the sed-method for prefixing the filename doesn't work. So with this commit grepping binary files gives reasonable output with GNU grep now. This was inspired by zgrep but the implementation is different.
2022-07-18xzgrep: Use -e to specify the pattern to grep.Lasse Collin1-8/+4
Now we don't need the separate test for adding the -q option as it can be added directly in the two places where it's needed.
2022-07-18Scripts: Use printf instead of echo in a few places.Lasse Collin4-11/+11
It's a good habbit as echo has some portability corner cases when the string contents can be anything.
2022-07-17xzgrep: Add more LC_ALL=C to avoid bugs with multibyte characters.Lasse Collin1-6/+8
Also replace one use of expr with printf. The rationale for LC_ALL=C was already mentioned in 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 that fixed a security issue. However, unrelated uses weren't changed in that commit yet. POSIX says that with sed and such tools one should use LC_ALL=C to ensure predictable behavior when strings contain byte sequences that aren't valid multibyte characters in the current locale. See under "Application usage" in here: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html With GNU sed invalid multibyte strings would work without this; it's documented in its Texinfo manual. Some other implementations aren't so forgiving.
2022-07-17xzgrep: Fix parsing of certain options.Lasse Collin1-2/+17
Fix handling of "xzgrep -25 foo" (in GNU grep "grep -25 foo" is an alias for "grep -C25 foo"). xzgrep would treat "foo" as filename instead of as a pattern. This bug was fixed in zgrep in gzip in 2012. Add -E, -F, -G, and -P to the "no argument required" list. Add -X to "argument required" list. It is an intentionally-undocumented GNU grep option so this isn't an important option for xzgrep but it seems that other grep implementations (well, those that I checked) don't support -X so I hope this change is an improvement still. grep -d (grep --directories=ACTION) requires an argument. In contrast to zgrep, I kept -d in the "no argument required" list because it's not supported in xzgrep (or zgrep). This way "xzgrep -d" gives an error about option being unsupported instead of telling that it requires an argument. Both zgrep and xzgrep tell that it's unsupported if an argument is specified. Add comments.
2022-07-14liblzma: Rename a variable and improve a comment.Lasse Collin1-4/+9
2022-07-13liblzma: Add optional autodetection of LZMA end marker.Lasse Collin6-37/+86
Turns out that this is needed for .lzma files as the spec in LZMA SDK says that end marker may be present even if the size is stored in the header. Such files are rare but exist in the real world. The code in liblzma is so old that the spec didn't exist in LZMA SDK back then and I had understood that such files weren't possible (the lzma tool in LZMA SDK didn't create such files). This modifies the internal API so that LZMA decoder can be told if EOPM is allowed even when the uncompressed size is known. It's allowed with .lzma and not with other uses. Thanks to Karl Beldan for reporting the problem.
2022-07-12xz: Document the special memlimit case of 2000 MiB on MIPS32.Lasse Collin1-2/+6
See commit fc3d3a7296ef58bb799a73943636b8bfd95339f7.
2022-05-23liblzma: Silence a warning.Lasse Collin1-1/+1
The actual initialization is done via mythread_sync and seems that GCC doesn't necessarily see that it gets initialized there.
2022-04-14xz: Fix build with --disable-threads.Lasse Collin1-0/+4
2022-04-14xz: Change the cap of the default -T0 memlimit for 32-bit xz.Lasse Collin1-1/+3
The SIZE_MAX / 3 was 1365 MiB. 1400 MiB gives little more room and it looks like a round (artificial) number in --info-memory once --info-memory is made to display it. Also, using #if avoids useless code on 64-bit builds.
2022-04-14xz: Add a default soft memory usage limit for --threads=0.Lasse Collin3-11/+82
This is a soft limit in sense that it only affects the number of threads. It never makes xz fail and it never makes xz change settings that would affect the compressed output. The idea is to make -T0 have more reasonable behavior when the system has very many cores or when a memory-hungry compression options are used. This also helps with 32-bit xz, preventing it from running out of address space. The downside of this commit is that now the number of threads might become too low compared to what the user expected. I hope this to be an acceptable compromise as the old behavior has been a source of well-argued complaints for a long time.
2022-04-14xz: Make -T0 use multithreaded mode on single-core systems.Lasse Collin3-9/+27
The main problem withi the old behavior is that the compressed output is different on single-core systems vs. multicore systems. This commit fixes it by making -T0 one thread in multithreaded mode on single-core systems. The downside of this is that it uses more memory. However, if --memlimit-compress is used, xz can (thanks to the previous commit) drop to the single-threaded mode still.
2022-04-14xz: Changes to --memlimit-compress and --no-adjust.Lasse Collin1-20/+43
In single-threaded mode, --memlimit-compress can make xz scale down the LZMA2 dictionary size to meet the memory usage limit. This obviously affects the compressed output. However, if xz was in threaded mode, --memlimit-compress could make xz reduce the number of threads but it wouldn't make xz switch from multithreaded mode to single-threaded mode or scale down the LZMA2 dictionary size. This seemed illogical and there was even a "FIXME?" about it. Now --memlimit-compress can make xz switch to single-threaded mode if one thread in multithreaded mode uses too much memory. If memory usage is still too high, then the LZMA2 dictionary size can be scaled down too. The option --no-adjust was also changed so that it no longer prevents xz from scaling down the number of threads as that doesn't affect compressed output (only performance). After this commit --no-adjust only prevents adjustments that affect compressed output, that is, with --no-adjust xz won't switch from multithreaded mode to single-threaded mode and won't scale down the LZMA2 dictionary size. The man page wasn't updated yet.
2022-04-12xz: Add --memlimit-mt-decompress along with a default limit value.Lasse Collin5-42/+97
--memlimit-mt-decompress allows specifying the limit for multithreaded decompression. This matches memlimit_threading in liblzma. This limit can only affect the number of threads being used; it will never prevent xz from decompressing a file. The old --memlimit-decompress option is still used at the same time. If the value of --memlimit-decompress (the default value or one specified by the user) is less than the value of --memlimit-mt-decompress , then --memlimit-mt-decompress is reduced to match --memlimit-decompress. Man page wasn't updated yet.
2022-04-06liblzma: Threaded decoder: Improve setting of pending_error.Lasse Collin1-16/+35
It doesn't need to be done conditionally. The comments try to explain it.
2022-04-06liblzma: Add a new flag LZMA_FAIL_FAST for threaded decoder.Lasse Collin3-26/+56
In most cases if the input file is corrupt the application won't care about the uncompressed content at all. With this new flag the threaded decoder will return an error as soon as any thread has detected an error; it won't wait to copy out the data before the location of the error. I don't plan to use this in xz to keep the behavior consistent between single-threaded and multi-threaded modes.
2022-04-05liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.Lasse Collin1-2/+24
This makes the behavior consistent with the single-threaded decoder when handling truncated .xz files. Thanks to Jia Tan for finding this issue.
2022-04-02liblzma: Threaded decoder: Support zpipe.c-style decoding loop.Lasse Collin1-10/+67
This makes it possible to call lzma_code() in a loop that only reads new input when lzma_code() didn't fill the output buffer completely. That isn't the calling style suggested by the liblzma example program 02_decompress.c so perhaps the usefulness of this feature is limited. Also, it is possible to write such a loop so that it works with the single-threaded decoder but not with the threaded decoder even after this commit, or so that it works only if lzma_mt.timeout = 0. The zlib tutorial <https://zlib.net/zlib_how.html> is a well-known example of a loop where more input is read only when output isn't full. Porting this as is to liblzma would work with the single-threaded decoder (if LZMA_CONCATENATED isn't used) but it wouldn't work with threaded decoder even after this commit because the loop assumes that no more output is possible when it cannot read more input ("if (strm.avail_in == 0) break;"). This cannot be fixed at liblzma side; the loop has to be modified at least a little. I'm adding this in any case because the actual code is simple and short and should have no harmful side-effects in other situations.
2022-03-29xzgrep: Fix escaping of malicious filenames (ZDI-CAN-16587).Lasse Collin1-8/+12
Malicious filenames can make xzgrep to write to arbitrary files or (with a GNU sed extension) lead to arbitrary code execution. xzgrep from XZ Utils versions up to and including 5.2.5 are affected. 5.3.1alpha and 5.3.2alpha are affected as well. This patch works for all of them. This bug was inherited from gzip's zgrep. gzip 1.12 includes a fix for zgrep. The issue with the old sed script is that with multiple newlines, the N-command will read the second line of input, then the s-commands will be skipped because it's not the end of the file yet, then a new sed cycle starts and the pattern space is printed and emptied. So only the last line or two get escaped. One way to fix this would be to read all lines into the pattern space first. However, the included fix is even simpler: All lines except the last line get a backslash appended at the end. To ensure that shell command substitution doesn't eat a possible trailing newline, a colon is appended to the filename before escaping. The colon is later used to separate the filename from the grep output so it is fine to add it here instead of a few lines later. The old code also wasn't POSIX compliant as it used \n in the replacement section of the s-command. Using \<newline> is the POSIX compatible method. LC_ALL=C was added to the two critical sed commands. POSIX sed manual recommends it when using sed to manipulate pathnames because in other locales invalid multibyte sequences might cause issues with some sed implementations. In case of GNU sed, these particular sed scripts wouldn't have such problems but some other scripts could have, see: info '(sed)Locale Considerations' This vulnerability was discovered by: cleemy desu wayo working with Trend Micro Zero Day Initiative Thanks to Jim Meyering and Paul Eggert discussing the different ways to fix this and for coordinating the patch release schedule with gzip.
2022-03-26liblzma: Fix a deadlock in threaded decoder.Lasse Collin1-19/+52
If a worker thread has consumed all input so far and it's waiting on thr->cond and then the main thread enables partial update for that thread, the code used to deadlock. This commit allows one dummy decoding pass to occur in this situation which then also does the partial update. As part of the fix, this moves thr->progress_* updates to avoid the second thr->mutex locking. Thanks to Jia Tan for finding, debugging, and reporting the bug.
2022-03-23liblzma: Threaded decoder: Don't stop threads on LZMA_TIMED_OUT.Lasse Collin1-1/+1
LZMA_TIMED_OUT is not an error and thus stopping threads on LZMA_TIMED_OUT breaks the decoder badly. Thanks to Jia Tan for finding the bug and for the patch.
2022-03-07xz: Add initial support for threaded decompression.Lasse Collin1-1/+35
If threading support is enabled at build time, this will use lzma_stream_decoder_mt() even for single-threaded mode. With memlimit_threading=0 the behavior should be identical. This needs some work like adding --memlimit-threading=LIMIT. The original patch from Sebastian Andrzej Siewior included a method to get currently available RAM on Linux. It might be one way to go but as it is Linux-only, the available-RAM approach needs work for portability or using a fallback method on other OSes. The man page wasn't updated yet.
2022-03-07liblzma: Add threaded .xz decompressor.Lasse Collin5-7/+1907
I realize that this is about a decade late. Big thanks to Sebastian Andrzej Siewior for the original patch. I made a bunch of smaller changes but after a while quite a few things got rewritten. So any bugs in the commit were created by me.
2022-03-06liblzma: Fix docs: lzma_block_decoder() cannot return LZMA_UNSUPPORTED_CHECK.Lasse Collin1-3/+0
If Check is unsupported, it will be silently ignored. It's the caller's job to handle it.
2022-03-06liblzma: Add new output queue (lzma_outq) features.Lasse Collin2-8/+79
Add lzma_outq_clear_cache2() which may leave one buffer allocated in the cache. Add lzma_outq_outbuf_memusage() to get the memory needed for a single lzma_outbuf. This is now used internally in outqueue.c too. Track both the total amount of memory allocated and the amount of memory that is in active use (not in cache). In lzma_outbuf, allow storing the current input position that matches the current output position. This way the main thread can notice when no more output is possible without first providing more input. Allow specifying return code for lzma_outq_read() in a finished lzma_outbuf.
2022-03-06liblzma: Index hash: Change return value type of hash_append() to void.Lasse Collin1-6/+5
2022-02-22liblzma: Minor addition to lzma_vli_size() API doc.Lasse Collin1-0/+2
Thanks to Jia Tan.
2022-02-22liblzma: Check the return value of lzma_index_append() in threaded encoder.Lasse Collin1-2/+5
If lzma_index_append() failed (most likely memory allocation failure) it could have gone unnoticed and the resulting .xz file would have an incorrect Index. Decompressing such a file would produce the correct uncompressed data but then an error would occur when verifying the Index field.
2022-02-22liblzma: Use non-executable stack on FreeBSD as on LinuxEd Maste2-4/+4
2022-02-20liblzma: Make Block decoder catch certain types of errors better.Lasse Collin1-25/+54
Now it limits the input and output buffer sizes that are passed to a raw decoder. This way there's no need to check if the sizes can grow too big or overflow when updating Compressed Size and Uncompressed Size counts. This also means that a corrupt file cannot cause the raw decoder to process useless extra input or output that would exceed the size info in Block Header (and thus cause LZMA_DATA_ERROR anyway). More importantly, now the size information is verified more carefully in case raw decoder returns LZMA_OK. This doesn't really matter with the current single-threaded .xz decoder as the errors would be detected slightly later anyway. But this helps avoiding corner cases in the upcoming threaded decompressor, and it might help other Block decoder uses outside liblzma too. The test files bad-1-lzma2-{9,10,11}.xz test these conditions. With the single-threaded .xz decoder the only difference is that LZMA_DATA_ERROR is detected in a difference place now.
2022-02-07liblzma: Add NULL checks to LZMA and LZMA2 properties encoders.jiat752-0/+6
Previously lzma_lzma_props_encode() and lzma_lzma2_props_encode() assumed that the options pointers must be non-NULL because the with these filters the API says it must never be NULL. It is good to do these checks anyway.
2022-02-06liblzma: Fix uint64_t vs. size_t confusion.Lasse Collin1-4/+7
This broke 32-bit builds due to a pointer type mismatch. This bug was introduced with the output-size-limited encoding in 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. Thanks to huangqinjin for the bug report.
2021-11-13xzgrep: Update man page timestamp.Lasse Collin1-1/+1
2021-11-13xzgrep: use `grep -E/-F` instead of `egrep` and `fgrep`Ville Skyttä2-6/+6
`egrep` and `fgrep` have been deprecated in GNU grep since 2007, and in current post 3.7 Git they have been made to emit obsolescence warnings: https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a9515624709865d480e3142fd959bccd1c9372d1
2021-10-28Bump the version number for 5.3.2alpha.larhzu/v5.3.2alphaLasse Collin2-2/+2
2021-10-27xz: Change the coding style of the previous commit.Lasse Collin1-5/+6
It isn't any better now but it's consistent with the rest of the code base.
2021-10-27xz: Avoid fchown(2) failure.Alexander Bluhm1-1/+7
OpenBSD does not allow to change the group of a file if the user does not belong to this group. In contrast to Linux, OpenBSD also fails if the new group is the same as the old one. Do not call fchown(2) in this case, it would change nothing anyway. This fixes an issue with Perl Alien::Build module. https://github.com/PerlAlien/Alien-Build/issues/62
2021-09-17liblzma: Fix liblzma.map for the lzma_microlzma_* symbols.Lasse Collin1-2/+2
This should have been part of d267d109c370a40b502e73f8664b154b15e4f253. Thanks to Gao Xiang.
2021-09-09liblzma: Use _MSVC_LANG to detect when "noexcept" can be used with MSVC.Lasse Collin1-1/+2
By default, MSVC always sets __cplusplus to 199711L. The real C++ standard version is available in _MSVC_LANG (or one could use /Zc:__cplusplus to set __cplusplus correctly). Fixes <https://sourceforge.net/p/lzmautils/discussion/708858/thread/f6bc3b108a/>. Thanks to Dan Weiss.
2021-09-05liblzma: Rename EROFS LZMA to MicroLZMA.Lasse Collin4-47/+52
It still exists primarily for EROFS but MicroLZMA is a more generic name (that hopefully doesn't clash with something that already exists).
2021-06-04xzdiff: Update the man page about the exit status.Lasse Collin1-2/+2
This was forgotten from 194029ffaf74282a81f0c299c07f73caca3232ca.
2021-06-04xzless: Fix less(1) version detection when it contains a dot.Lasse Collin1-1/+1
Sometimes the version number from "less -V" contains a dot, sometimes not. xzless failed detect the version number when it does contain a dot. This fixes it. Thanks to nick87720z for reporting this. Apparently it had been reported here <https://bugs.gentoo.org/489362> in 2013.
2021-04-11Reduce maximum possible memory limit on MIPS32Ivan A. Melnikov1-0/+6
Due to architectural limitations, address space available to a single userspace process on MIPS32 is limited to 2 GiB, not 4, even on systems that have more physical RAM -- e.g. 64-bit systems with 32-bit userspace, or systems that use XPA (an extension similar to x86's PAE). So, for MIPS32, we have to impose stronger memory limits. I've chosen 2000MiB to give the process some headroom.
2021-01-29liblzma: Fix unitialized variable.Lasse Collin1-0/+1
This was introduced two weeks ago in the commit 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. Thanks to Nathan Moinvaziri.
2021-01-24liblzma: Fix a wrong comment in stream_encoder_mt.c.Lasse Collin1-3/+7
2021-01-17liblzma: In EROFS LZMA decoder, verify that comp_size matches at the end.Lasse Collin1-1/+6
When the uncompressed size is known to be exact, after decompressing the stream exactly comp_size bytes of input must have been consumed. This is a minor improvement to error detection.
2021-01-17liblzma: Make EROFS LZMA decoder work when exact uncomp_size isn't known.Lasse Collin2-12/+91
The caller must still not specify an uncompressed size bigger than the actual uncompressed size. As a downside, this now needs the exact compressed size.
2021-01-14liblzma: Fix missing normalization in rc_encode_dummy().Lasse Collin1-1/+6
Without this fix it could attempt to create too much output.
2021-01-14liblzma: Add EROFS LZMA encoder and decoder.Lasse Collin5-0/+367
Right now this is just a planned extra-compact format for use in the EROFS file system in Linux. At this point it's possible that the format will either change or be abandoned and removed completely. The special thing about the encoder is that it uses the output-size-limited encoding added in the previous commit. EROFS uses fixed-sized blocks (e.g. 4 KiB) to hold compressed data so the compressors must be able to create valid streams that fill the given block size.
2021-01-14liblzma: Add rough support for output-size-limited encoding in LZMA1.Lasse Collin6-35/+246
With this it is possible to encode LZMA1 data without EOPM so that the encoder will encode as much input as it can without exceeding the specified output size limit. The resulting LZMA1 stream will be a normal LZMA1 stream without EOPM. The actual uncompressed size will be available to the caller via the uncomp_size pointer. One missing thing is that the LZMA layer doesn't inform the LZ layer when the encoding is finished and thus the LZ may read more input when it won't be used. However, this doesn't matter if encoding is done with a single call (which is the planned use case for now). For proper multi-call encoding this should be improved. This commit only adds the functionality for internal use. Nothing uses it yet.
2021-01-11Scripts: Add zstd support to xzdiff.Lasse Collin2-7/+15
2021-01-11xz: Make --keep accept symlinks, hardlinks, and setuid/setgid/sticky.Lasse Collin2-5/+20
Previously this required using --force but that has other effects too which might be undesirable. Changing the behavior of --keep has a small risk of breaking existing scripts but since this is a fairly special corner case I expect the likehood of breakage to be low enough. I think the new behavior is more logical. The only reason for the old behavior was to be consistent with gzip and bzip2. Thanks to Vincent Lefevre and Sebastian Andrzej Siewior.
2021-01-11Scripts: Fix exit status of xzgrep.Lasse Collin1-7/+13
Omit the -q option from xz, gzip, and bzip2. With xz this shouldn't matter. With gzip it's important because -q makes gzip replace SIGPIPE with exit status 2. With bzip2 it's important because with -q bzip2 is completely silent if input is corrupt while other decompressors still give an error message. Avoiding exit status 2 from gzip is important because bzip2 uses exit status 2 to indicate corrupt input. Before this commit xzgrep didn't recognize corrupt .bz2 files because xzgrep was treating exit status 2 as SIGPIPE for gzip compatibility. zstd still needs -q because otherwise it is noisy in normal operation. The code to detect real SIGPIPE didn't check if the exit status was due to a signal (>= 128) and so could ignore some other exit status too.
2021-01-11Scripts: Fix exit status of xzdiff/xzcmp.Lasse Collin1-14/+21
This is a minor fix since this affects only the situation when the files differ and the exit status is something else than 0. In such case there could be SIGPIPE from a decompression tool and that would result in exit status of 2 from xzdiff/xzcmp while the correct behavior would be to return 1 or whatever else diff or cmp may have returned. This commit omits the -q option from xz/gzip/bzip2/lzop arguments. I'm not sure why the -q was used in the first place, perhaps it hides warnings in some situation that I cannot see at the moment. Hopefully the removal won't introduce a new bug. With gzip the -q option was harmful because it made gzip return 2 instead of >= 128 with SIGPIPE. Ignoring exit status 2 (warning from gzip) isn't practical because bzip2 uses exit status 2 to indicate corrupt input file. It's better if SIGPIPE results in exit status >= 128. With bzip2 the removal of -q seems to be good because with -q it prints nothing if input is corrupt. The other tools aren't silent in this situation even with -q. On the other hand, if zstd support is added, it will need -q since otherwise it's noisy in normal situations. Thanks to Étienne Mollier and Sebastian Andrzej Siewior.
2021-01-09liblzma: Make lzma_outq usable for threaded decompression too.Lasse Collin3-157/+301
Before this commit all output queue buffers were allocated as a single big allocation. Now each buffer is allocated separately when needed. Used buffers are cached to avoid reallocation overhead but the cache will keep only one buffer size at a time. This should make things work OK in the decompression where most of the time the buffer sizes will be the same but with some less common files the buffer sizes may vary. While this should work fine, it's still a bit preliminary and may even get reverted if it turns out to be useless for decompression.
2020-12-23liblzma: Enable Intel CET in x86 CRC assembly codesH.J. Lu2-0/+18
When Intel CET is enabled, we need to include <cet.h> in assembly codes to mark Intel CET support and add _CET_ENDBR to indirect jump targets. Tested on Intel Tiger Lake under CET enabled Linux.
2020-12-05Scripts: Add zstd support to xzgrep.Adam Borowski2-3/+7
Thanks to Adam Borowski.
2020-11-01xz: Avoid unneeded \f escapes on the man page.Lasse Collin1-9/+22
I don't want to use \c in macro arguments but groff_man(7) suggests that \f has better portability. \f would be needed for the .TP strings for portability reasons anyway. Thanks to Bjarni Ingi Gislason.
2020-11-01xz: Use non-breaking spaces when intentionally using more than one space.Lasse Collin1-1/+1
This silences some style checker warnings. Seems that spaces in the beginning of a line don't need this treatment. Thanks to Bjarni Ingi Gislason.
2020-11-01xz: Protect the ellipsis (...) on the man page with \&.Lasse Collin1-2/+2
This does it only when ... appears outside macro calls. Thanks to Bjarni Ingi Gislason.
2020-11-01xz: Avoid the abbreviation "e.g." on the man page.Lasse Collin1-33/+33
A few are simply omitted, most are converted to "for example" and surrounded with commas. Sounds like that this is better style, for example, man-pages(7) recommends avoiding such abbreviations except in parenthesis. Thanks to Bjarni Ingi Gislason.
2020-07-12xz man page: Change \- (minus) to \(en (en-dash) for a numeric range.Lasse Collin1-8/+8
Docs of ancient troff/nroff mention \(em (em-dash) but not \(en and \- was used for both minus and en-dash. I don't know how portable \(en is nowadays but it can be changed back if someone complains. At least GNU groff and OpenBSD's mandoc support it. Thanks to Bjarni Ingi Gislason for the patch.
2020-07-12Windows: Fix building of resource files when config.h isn't used.Lasse Collin1-1/+3
Now CMake + Visual Studio works for building liblzma.dll. Thanks to Markus Rickert.
2020-04-06src/scripts/xzgrep.1: Filenames to xzgrep are optional.Lasse Collin1-1/+1
xzgrep --help was correct already.
2020-04-06src/script/xzgrep.1: Remove superfluous '.RB'Bjarni Ingi Gislason1-6/+6
Output is from: test-groff -b -e -mandoc -T utf8 -rF0 -t -w w -z [ "test-groff" is a developmental version of "groff" ] Input file is ./src/scripts/xzgrep.1 <src/scripts/xzgrep.1>:20 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:23 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:26 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:29 (macro RB): only 1 argument, but more are expected <src/scripts/xzgrep.1>:32 (macro RB): only 1 argument, but more are expected "abc..." does not mean the same as "abc ...". The output from nroff and troff is unchanged except for the space between "file" and "...". Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2020-04-06xzgrep.1: Delete superfluous '.PP'Bjarni Ingi Gislason1-1/+0
Summary: mandoc -T lint xzgrep.1 : mandoc: xzgrep.1:79:2: WARNING: skipping paragraph macro: PP empty There is no change in the output of "nroff" and "troff". Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2020-04-06src/xz/xz.1: Correct misused two-fonts macrosBjarni Ingi Gislason1-5/+5
Output is from: test-groff -b -e -mandoc -T utf8 -rF0 -t -w w -z [ "test-groff" is a developmental version of "groff" ] Input file is ./src/xz/xz.1 <src/xz/xz.1>:408 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1009 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1743 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:1920 (macro BR): only 1 argument, but more are expected <src/xz/xz.1>:2213 (macro BR): only 1 argument, but more are expected Output from nroff and troff is unchanged, except for a font change of a full stop (.). Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
2020-03-23Typo fixes from fossies.org.Lasse Collin4-5/+5
https://fossies.org/linux/misc/xz-5.2.5.tar.xz/codespell.html
2020-03-11xz: Never use thousand separators in DJGPP builds.Lasse Collin1-2/+12
DJGPP 2.05 added support for thousands separators but it's broken at least under WinXP with Finnish locale that uses a non-breaking space as the thousands separator. Workaround by disabling thousands separators for DJGPP builds.
2020-03-02liblzma: Fix a comment and RC_SYMBOLS_MAX.Lasse Collin1-2/+2
The comment didn't match the value of RC_SYMBOLS_MAX and the value itself was slightly larger than actually needed. The only harm about this was that memory usage was a few bytes larger.
2020-02-24liblzma: Remove unneeded <sys/types.h> from fastpos_tablegen.c.Lasse Collin1-1/+0
This file only generates fastpos_table.c. It isn't built as a part of liblzma.
2020-02-22Use defined(__GNUC__) before __GNUC__ in preprocessor lines.Lasse Collin2-3/+5
This should silence the equivalent of -Wundef in compilers that don't define __GNUC__.
2020-02-21liblzma: Add more uses of lzma_memcmplen() to the normal mode of LZMA.Lasse Collin1-6/+10
This gives a tiny encoder speed improvement. This could have been done in 2014 after the commit 544aaa3d13554e8640f9caf7db717a96360ec0f6 but it was forgotten.
2020-02-21xz: Silence a warning when sig_atomic_t is long int.Lasse Collin1-1/+1
It can be true at least on z/OS.
2020-02-21xz: Avoid unneeded access of a volatile variable.Lasse Collin1-1/+1
2020-02-20tuklib_exit: Add missing header.Lasse Collin1-0/+1
strerror() needs <string.h> which happened to be included via tuklib_common.h -> tuklib_config.h -> sysdefs.h if HAVE_CONFIG_H was defined. This wasn't tested without config.h before so it had worked fine.
2020-02-18Revert the previous commit and add a comment.Lasse Collin1-7/+10
The previous commit broke crc32_tablegen.c. If the whole package is built without config.h (with defines set on the compiler command line) this should still work fine as long as these headers conform to C99 well enough.
2020-02-17Do not check for HAVE_CONFIG_H in tuklib_config.h.Lasse Collin1-8/+7
In XZ Utils sysdefs.h takes care of it and the required headers.
2020-02-16sysdefs.h: Omit the conditionals around string.h and limits.h.Lasse Collin1-6/+2
string.h is used unconditionally elsewhere in the project and configure has always stopped if limits.h is missing, so these headers must have been always available even on the weirdest systems.
2020-02-07Build: Add support for translated man pages using po4a.Lasse Collin3-42/+127
The dependency on po4a is optional. It's never required to install the translated man pages when xz is built from a release tarball. If po4a is missing when building from xz.git, the translated man pages won't be generated but otherwise the build will work normally. The translations are only updated automatically by autogen.sh and by "make mydist". This makes it easy to keep po4a as an optional dependency and ensures that I won't forget to put updated translations to a release tarball. The translated man pages aren't installed if --disable-nls is used. The installation of translated man pages abuses Automake internals by calling "install-man" with redefined dist_man_MANS and man_MANS. This makes the hairy script code slightly less hairy. If it breaks some day, this code needs to be fixed; don't blame Automake developers. Also, this adds more quotes to the existing shell script code in the Makefile.am "-hook"s.
2020-02-05xz: Make it a fatal error if enabling the sandbox fails.Lasse Collin1-1/+1
Perhaps it's too drastic but on the other hand it will let me learn about possible problems if people report the errors. This won't be backported to the v5.2 branch.
2020-02-05xz: Comment out annoying sandboxing messages.Lasse Collin1-3/+7
2020-02-01xz: Limit --memlimit-compress to at most 4020 MiB for 32-bit xz.Lasse Collin2-2/+51
See the code comment for reasoning. It's far from perfect but hopefully good enough for certain cases while hopefully doing nothing bad in other situations. At presets -5 ... -9, 4020 MiB vs. 4096 MiB makes no difference on how xz scales down the number of threads. The limit has to be a few MiB below 4096 MiB because otherwise things like "xz --lzma2=dict=500MiB" won't scale down the dict size enough and xz cannot allocate enough memory. With "ulimit -v $((4096 * 1024))" on x86-64, the limit in xz had to be no more than 4085 MiB. Some safety margin is good though. This is hack but it should be useful when running 32-bit xz on a 64-bit kernel that gives full 4 GiB address space to xz. Hopefully this is enough to solve this: https://bugzilla.redhat.com/show_bug.cgi?id=1196786 FreeBSD has a patch that limits the result in tuklib_physmem() to SIZE_MAX on 32-bit systems. While I think it's not the way to do it, the results on --memlimit-compress have been good. This commit should achieve practically identical results for compression while leaving decompression and tuklib_physmem() and thus lzma_physmem() unaffected.
2020-01-26xz: Set the --flush-timeout deadline when the first input byte arrives.Lasse Collin3-7/+6
xz --flush-timeout=2000, old version: 1. xz is started. The next flush will happen after two seconds. 2. No input for one second. 3. A burst of a few kilobytes of input. 4. No input for one second. 5. Two seconds have passed and flushing starts. The first second counted towards the flush-timeout even though there was no pending data. This can cause flushing to occur more often than needed. xz --flush-timeout=2000, after this commit: 1. xz is started. 2. No input for one second. 3. A burst of a few kilobytes of input. The next flush will happen after two seconds counted from the time when the first bytes of the burst were read. 4. No input for one second. 5. No input for another second. 6. Two seconds have passed and flushing starts.
2020-01-26xz: Move flush_needed from mytime.h to file_pair struct in file_io.h.Lasse Collin5-9/+7
2020-01-26xz: coder.c: Make writing output a separate function.Lasse Collin1-13/+17
The same code sequence repeats so it's nicer as a separate function. Note that in one case there was no test for opt_mode != MODE_TEST, but that was only because that condition would always be true, so this commit doesn't change the behavior there.
2020-01-26xz: Fix semi-busy-waiting in xz --flush-timeout.Lasse Collin3-4/+19
When input blocked, xz --flush-timeout=1 would wake up every millisecond and initiate flushing which would have nothing to flush and thus would just waste CPU time. The fix disables the timeout when no input has been seen since the previous flush.
2020-01-26xz: Refactor io_read() a bit.Lasse Collin1-9/+8
2020-01-26xz: Update a comment in file_io.h.Lasse Collin1-1/+4
2020-01-26xz: Move the setting of flush_needed in file_io.c to a nicer location.Lasse Collin1-4/+2
2019-12-31Rename unaligned_read32ne to read32ne, and similarly for the others.Lasse Collin13-57/+49
2019-12-31Rename read32ne to aligned_read32ne, and similarly for the others.Lasse Collin3-32/+32
Using the aligned methods requires more care to ensure that the address really is aligned, so it's nicer if the aligned methods are prefixed. The next commit will remove the unaligned_ prefix from the unaligned methods which in liblzma are used in more places than the aligned ones.
2019-12-31Revise tuklib_integer.h and .m4.Lasse Collin1-217/+271
Add a configure option --enable-unsafe-type-punning to get the old non-conforming memory access methods. It can be useful with old compilers or in some other less typical situations but shouldn't normally be used. Omit the packed struct trick for unaligned access. While it's best in some cases, this is simpler. If the memcpy trick doesn't work, one can request unsafe type punning from configure. Because CRC32/CRC64 code needs fast aligned reads, if no very safe way to do it is found, type punning is used as a fallback. This sucks but since it currently works in practice, it seems to be the least bad option. It's never needed with GCC >= 4.7 or Clang >= 3.6 since these support __builtin_assume_aligned and thus fast aligned access can be done with the memcpy trick. Other things: - Support GCC/Clang __builtin_bswapXX - Cleaner bswap fallback macros - Minor cleanups
2019-09-24Scripts: Put /usr/xpg4/bin to the beginning of PATH on Solaris.Lasse Collin4-0/+4
This adds a configure option --enable-path-for-scripts=PREFIX which defaults to empty except on Solaris it is /usr/xpg4/bin to make POSIX grep and others available. The Solaris case had been documented in INSTALL with a manual fix but it's better to do this automatically since it is needed on most Solaris systems anyway. Thanks to Daniel Richard G.
2019-07-12Fix comment typos in tuklib_mbstr* files.Lasse Collin3-3/+3
2019-07-12Add missing include to tuklib_mbstr_width.c.Lasse Collin1-0/+1
It didn't matter in XZ Utils because sysdefs.h includes string.h anyway.
2019-07-12Update tuklib base headers to include stdbool.h.Lasse Collin2-1/+2
2019-06-28xz: Automatically align the strings in --info-memory.Lasse Collin1-11/+34
This makes it easier to translate the strings. Also, the string for amount of RAM was shortened.
2019-06-25liblzma: Fix a buggy comment.Lasse Collin1-1/+1
2019-06-24liblzma: Add a comment.Lasse Collin1-1/+1
2019-06-24liblzma: Silence clang -Wmissing-variable-declarations.Lasse Collin2-0/+6
2019-06-24Add LZMA_RET_INTERNAL1..8 to lzma_ret and use one for LZMA_TIMED_OUT.Lasse Collin4-7/+25
LZMA_TIMED_OUT is *internally* used as a value for lzma_ret enumeration. Previously it was #defined to 32 and cast to lzma_ret. That way it wasn't visible in the public API, but this was hackish. Now the public API has eight LZMA_RET_INTERNALx members and LZMA_TIMED_OUT is #defined to LZMA_RET_INTERNAL1. This way the code is cleaner overall although the public API has a few extra mysterious enum members.
2019-06-24xz: Silence a warning from clang -Wsign-conversion in main.c.Lasse Collin1-1/+1
2019-06-24xz: Make "headings" static in list.c.Lasse Collin1-1/+1
Caught by clang -Wmissing-variable-declarations.
2019-06-24liblzma: Remove incorrect uses of lzma_attribute((__unused__)).Lasse Collin3-6/+3
Caught by clang -Wused-but-marked-unused.
2019-06-24xz: Fix an integer overflow with 32-bit off_t.Lasse Collin1-2/+9
Or any off_t which isn't very big (like signed 64 bit integer that most system have). A small off_t could overflow if the file being decompressed had long enough run of zero bytes, which would result in corrupt output.
2019-06-24xz: Cleanup io_seek_src() a bit.Lasse Collin1-3/+1
lseek() returns -1 on error and checking for -1 is nicer.
2019-06-24xz: Change io_seek_src and io_pread arguments from off_t to uint64_t.Lasse Collin3-11/+18
This helps fixing warnings from -Wsign-conversion and makes the code look better too.
2019-06-24xz: list.c: Fix some warnings from -Wsign-conversion.Lasse Collin1-3/+4
2019-06-23tuklib_mbstr_width: Fix a warning from -Wsign-conversion.Lasse Collin1-1/+1
2019-06-23xz: Fix some of the warnings from -Wsign-conversion.Lasse Collin7-13/+14
2019-06-23tuklib_cpucores: Silence warnings from -Wsign-conversion.Lasse Collin1-5/+5
2019-06-23xzdec: Fix warnings from -Wsign-conversion.Lasse Collin1-1/+1
2019-06-23liblzma: Fix warnings from -Wsign-conversion.Lasse Collin11-29/+31
Also, more parentheses were added to the literal_subcoder macro in lzma_comon.h (better style but no functional change in the current usage).
2019-06-23tuklib_integer: Silence warnings from -Wsign-conversion.Lasse Collin1-3/+3
2019-06-20tuklib_integer: Fix usage of conv macros.Lasse Collin1-4/+8
Use a temporary variable instead of e.g. conv32le(unaligned_read32ne(buf)) because the macro can evaluate its argument multiple times.
2019-06-03liblzma: Fix comments.Lasse Collin6-7/+7
Thanks to Bruce Stark.
2019-06-02liblzma: Fix one more unaligned read to use unaligned_read16ne().Lasse Collin1-1/+1
2019-06-01liblzma: memcmplen: Use ctz32() from tuklib_integer.h.Lasse Collin1-9/+1
The same compiler-specific #ifdefs are already in tuklib_integer.h
2019-06-01tuklib_integer: Cleanup MSVC-specific code.Lasse Collin1-11/+9
2019-06-01liblzma: Use unaligned_readXXne functions instead of type punning.Lasse Collin2-7/+7
Now gcc -fsanitize=undefined should be clean. Thanks to Jeffrey Walton.
2019-06-01tuklib_integer: Improve unaligned memory access.Lasse Collin1-12/+168
Now memcpy() or GNU C packed structs for unaligned access instead of type punning. See the comment in this commit for details. Avoiding type punning with unaligned access is needed to silence gcc -fsanitize=undefined. New functions: unaliged_readXXne and unaligned_writeXXne where XX is 16, 32, or 64.
2019-05-13liblzma: Avoid memcpy(NULL, foo, 0) because it is undefined behavior.Lasse Collin3-5/+23
I should have always known this but I didn't. Here is an example as a reminder to myself: int mycopy(void *dest, void *src, size_t n) { memcpy(dest, src, n); return dest == NULL; } In the example, a compiler may assume that dest != NULL because passing NULL to memcpy() would be undefined behavior. Testing with GCC 8.2.1, mycopy(NULL, NULL, 0) returns 1 with -O0 and -O1. With -O2 the return value is 0 because the compiler infers that dest cannot be NULL because it was already used with memcpy() and thus the test for NULL gets optimized out. In liblzma, if a null-pointer was passed to memcpy(), there were no checks for NULL *after* the memcpy() call, so I cautiously suspect that it shouldn't have caused bad behavior in practice, but it's hard to be sure, and the problematic cases had to be fixed anyway. Thanks to Jeffrey Walton.
2019-05-11xz: Update xz man page date.Lasse Collin1-1/+1
2019-05-11spellingAntoine Cœur18-22/+22
2019-05-01xz: In xz -lvv look at the widths of the check names too.Lasse Collin1-6/+26
Now the widths of the check names is used to adjust the width of the Check column. This way there no longer is a need to restrict the widths of the check names to be at most ten terminal-columns.
2019-05-01xz: Fix xz -lvv column alignment to look at the translated strings.Lasse Collin1-2/+2
2019-03-04xz: Automatically align column headings in xz -lvv.Lasse Collin1-51/+212
2019-03-04xz: Automatically align strings ending in a colon in --list output.Lasse Collin1-12/+102
This should avoid alignment errors in translations with these strings.
2018-12-20xz: Fix a crash in progress indicator when in passthru mode.Lasse Collin3-7/+25
"xz -dcfv not_an_xz_file" crashed (all four options are required to trigger it). It caused xz to call lzma_get_progress(&strm, ...) when no coder was initialized in strm. In this situation strm.internal is NULL which leads to a crash in lzma_get_progress(). The bug was introduced when xz started using lzma_get_progress() to get progress info for multi-threaded compression, so the bug is present in versions 5.1.3alpha and higher. Thanks to Filip Palian <Filip.Palian@pjwstk.edu.pl> for the bug report.
2018-11-22xz: Update man page timestamp.Lasse Collin1-1/+1
2018-11-22'have have' typosPavel Raiskup2-2/+2
2018-10-26liblzma: Don't verify header CRC32s if building for fuzz testing.Lasse Collin4-5/+20
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is #defined when liblzma is being built for fuzz testing. Most fuzzed inputs would normally get rejected because of incorrect CRC32 and the actual header decoding code wouldn't get fuzzed. Disabling CRC32 checks avoids this problem. The fuzzer program must still use LZMA_IGNORE_CHECK flag to disable verification of integrity checks of uncompressed data.
2018-07-27xzless: Rename unused variables to silence static analysers.Lasse Collin1-1/+1
In this particular case I don't see this affecting readability of the code. Thanks to Pavel Raiskup.
2018-07-27liblzma: Remove an always-true condition from lzma_index_cat().Lasse Collin1-2/+2
This should help static analysis tools to see that newg isn't leaked. Thanks to Pavel Raiskup.
2018-05-19liblzma: Improve lzma_properties_decode() API documentation.Lasse Collin1-3/+4
2018-04-29Bump the version number to 5.3.1alpha.larhzu/v5.3.1alphaLasse Collin2-2/+2
2018-02-06nothrow: use noexcept for C++11 and newerBen Boeckel1-1/+5
In C++11, the `throw()` specifier is deprecated and `noexcept` is preffered instead.
2018-02-06liblzma: Remove incorrect #ifdef from range_common.h.Lasse Collin1-3/+1
In most cases it was harmless but it could affect some custom build systems. Thanks to Pippijn van Steenhoven.
2018-01-10tuklib_integer: New Intel C compiler needs immintrin.h.Lasse Collin1-0/+11
Thanks to Melanie Blower (Intel) for the patch.
2017-08-14Fix or hide warnings from GCC 7's -Wimplicit-fallthrough.Lasse Collin2-0/+8
2017-05-23xz: Fix "xz --list --robot missing_or_bad_file.xz".Lasse Collin1-2/+6
It ended up printing an uninitialized char-array when trying to print the check names (column 7) on the "totals" line. This also changes the column 12 (minimum xz version) to 50000002 (xz 5.0.0) instead of 0 when there are no valid input files. Thanks to kidmin for the bug report.
2017-04-24xz: Use lzma_file_info_decoder() for --list.Lasse Collin1-210/+44
2017-04-24liblzma: Add lzma_file_info_decoder().Lasse Collin4-1/+928
2017-04-21liblzma: Rename LZMA_SEEK to LZMA_SEEK_NEEDED and seek_in to seek_pos.Lasse Collin3-11/+11
2017-04-19Update the home page URLs to HTTPS.Lasse Collin3-6/+6
2017-04-05xz: Add io_seek_src().Lasse Collin2-3/+30
2017-03-30xz: Use POSIX_FADV_RANDOM for in "xz --list" mode.Lasse Collin1-2/+8
xz --list is random access so POSIX_FADV_SEQUENTIAL was clearly wrong.
2017-03-30liblzma: Make lzma_index_decoder_init() visible to other liblzma funcs.Lasse Collin3-5/+30
This is to allow other functions to use it without going via the public API (lzma_index_decoder()).
2017-03-30liblzma: Add generic support for input seeking (LZMA_SEEK).Lasse Collin3-2/+42
Also mention LZMA_SEEK in xz/message.c to silence a warning.
2017-03-30liblzma: Fix lzma_memlimit_set(strm, 0).Lasse Collin3-3/+13
The 0 got treated specially in a buggy way and as a result the function did nothing. The API doc said that 0 was supposed to return LZMA_PROG_ERROR but it didn't. Now 0 is treated as if 1 had been specified. This is done because 0 is already used to indicate an error from lzma_memlimit_get() and lzma_memusage(). In addition, lzma_memlimit_set() no longer checks that the new limit is at least LZMA_MEMUSAGE_BASE. It's counter-productive for the Index decoder and was actually needed only by the auto decoder. Auto decoder has now been modified to check for LZMA_MEMUSAGE_BASE.
2017-03-30liblzma: Similar memlimit fix for stream_, alone_, and auto_decoder.Lasse Collin4-16/+20
2017-03-30liblzma: Fix handling of memlimit == 0 in lzma_index_decoder().Lasse Collin2-9/+13
It returned LZMA_PROG_ERROR, which was done to avoid zero as the limit (because it's a special value elsewhere), but using LZMA_PROG_ERROR is simply inconvenient and can cause bugs. The fix/workaround is to treat 0 as if it were 1 byte. It's effectively the same thing. The only weird consequence is that then lzma_memlimit_get() will return 1 even when 0 was specified as the limit. This fixes a very rare corner case in xz --list where a specific memory usage limit and a multi-stream file could print the error message "Internal error (bug)" instead of saying that the memory usage limit is too low.
2016-11-21liblzma: Avoid multiple definitions of lzma_coder structures.Lasse Collin35-423/+532
Only one definition was visible in a translation unit. It avoided a few casts and temp variables but seems that this hack doesn't work with link-time optimizations in compilers as it's not C99/C11 compliant. Fixes: http://www.mail-archive.com/xz-devel@tukaani.org/msg00279.html
2016-10-24tuklib_cpucores: Add support for sched_getaffinity().Lasse Collin1-0/+9
It's available in glibc (GNU/Linux, GNU/kFreeBSD). It's better than sysconf(_SC_NPROCESSORS_ONLN) because sched_getaffinity() gives the number of cores available to the process instead of the total number of cores online. As a side effect, this commit fixes a bug on GNU/kFreeBSD where configure would detect the FreeBSD-specific cpuset_getaffinity() but it wouldn't actually work because on GNU/kFreeBSD it requires using -lfreebsd-glue when linking. Now the glibc-specific function will be used instead. Thanks to Sebastian Andrzej Siewior for the original patch and testing.
2016-06-30xz: Fix copying of timestamps on Windows.Lasse Collin1-0/+18
xz used to call utime() on Windows, but its result gets lost on close(). Using _futime() seems to work. Thanks to Martok for reporting the bug: http://www.mail-archive.com/xz-devel@tukaani.org/msg00261.html
2016-06-16xz: Silence warnings from -Wlogical-op.Lasse Collin1-2/+10
Thanks to Evan Nemerson.
2016-04-10Build: Fix = to += for xz_SOURCES in src/xz/Makefile.am.Lasse Collin1-1/+1
Thanks to Christian Kujau.
2016-03-13liblzma: Disable external SHA-256 by default.Lasse Collin1-10/+6
This is the sane thing to do. The conflict with OpenSSL on some OSes and especially that the OS-provided versions can be significantly slower makes it clear that it was a mistake to have the external SHA-256 support enabled by default. Those who want it can now pass --enable-external-sha256 to configure. INSTALL was updated with notes about OSes where this can be a bad idea. The SHA-256 detection code in configure.ac had some bugs that could lead to a build failure in some situations. These were fixed, although it doesn't matter that much now that the external SHA-256 is disabled by default. MINIX >= 3.2.0 uses NetBSD's libc and thus has SHA256_Init in libc instead of libutil. Support for the libutil version was removed.
2015-11-08tuklib_physmem: Hopefully silence a warning on Windows.Lasse Collin1-1/+2
2015-11-04liblzma: Make Valgrind happier with optimized (gcc -O2) liblzma.Lasse Collin1-0/+4
When optimizing, GCC can reorder code so that an uninitialized value gets used in a comparison, which makes Valgrind unhappy. It doesn't happen when compiled with -O0, which I tend to use when running Valgrind. Thanks to Rich Prohaska. I remember this being mentioned long ago by someone else but nothing was done back then.
2015-11-03liblzma: Rename lzma_presets.c back to lzma_encoder_presets.c.Lasse Collin2-2/+2
It would be too annoying to update other build systems just because of this.
2015-11-03xz: Make xz buildable even when encoders or decoders are disabled.Lasse Collin5-13/+58
The patch is quite long but it's mostly about adding new #ifdefs to omit code when encoders or decoders have been disabled. This adds two new #defines to config.h: HAVE_ENCODERS and HAVE_DECODERS.
2015-11-03Build: Build LZMA1/2 presets also when only decoder is wanted.Lasse Collin2-2/+7
People shouldn't rely on the presets when decoding raw streams, but xz uses the presets as the starting point for raw decoder options anyway. lzma_encocder_presets.c was renamed to lzma_presets.c to make it clear it's not used solely by the encoder code.
2015-11-03Build: Don't omit lzma_cputhreads() unless using --disable-threads.Lasse Collin1-1/+4
Previously it was omitted if encoders were disabled with --disable-encoders. It didn't make sense and it also broke the build.
2015-11-02liblzma: Fix a build failure related to external SHA-256 support.Lasse Collin1-9/+23
If an appropriate header and structure were found by configure, but a library with a usable SHA-256 functions wasn't, the build failed.
2015-11-02xz: Always close the file before trying to delete it.Lasse Collin1-13/+12
unlink() can return EBUSY in errno for open files on some operating systems and file systems.
2015-10-12liblzma: Fix lzma_index_dup() for empty Streams.Lasse Collin1-5/+6
Stream Flags and Stream Padding weren't copied from empty Streams.
2015-10-12liblzma: Add a note to index.c for those using static analyzers.Lasse Collin1-0/+3
2015-10-12liblzma: Fix a memory leak in error path of lzma_index_dup().Lasse Collin1-9/+9
lzma_index_dup() calls index_dup_stream() which, in case of an error, calls index_stream_end() to free memory allocated by index_stream_init(). However, it illogically didn't actually free the memory. To make it logical, the tree handling code was modified a bit in addition to changing index_stream_end(). Thanks to Evan Nemerson for the bug report.
2015-07-12liblzma: A MSVC-specific hack isn't needed with MSVC 2013 and newer.Lasse Collin1-5/+13
2015-05-11xz: Document that threaded decompression hasn't been implemented yet.Lasse Collin1-1/+9
2015-04-20Revert "xz: Use pipe2() if available."Lasse Collin1-8/+1
This reverts commit 7a11c4a8e5e15f13d5fa59233b3172e65428efdd. It is a problem when libc has pipe2() but the kernel is too old to have pipe2() and thus pipe2() fails. In xz it's pointless to have a fallback for non-functioning pipe2(); it's better to avoid pipe2() completely. Thanks to Michael Fox for the bug report.
2015-04-01xz: Fix the Capsicum rights on user_abort_pipe.Lasse Collin1-1/+5
2015-03-31xz: Add support for sandboxing with Capsicum.Lasse Collin5-1/+110
The sandboxing is used conditionally as described in main.c. This isn't optimal but it was much easier to implement than a full sandboxing solution and it still covers the most common use cases where xz is writing to standard output. This should have practically no effect on performance even with small files as fork() isn't needed. C and locale libraries can open files as needed. This has been fine in the past, but it's a problem with things like Capsicum. io_sandbox_enter() tries to ensure that various locale-related files have been loaded before cap_enter() is called, but it's possible that there are other similar problems which haven't been seen yet. Currently Capsicum is available on FreeBSD 10 and later and there is a port to Linux too. Thanks to Loganaden Velvindron for help.
2015-03-30Bump version to 5.3.0alpha and soname to 5.3.99.Lasse Collin2-4/+4
The idea of 99 is that it looks a bit weird in this context. For new features there's no API/ABI stability in devel versions.
2015-03-29Fix the detection of installed RAM on QNX.Lasse Collin1-1/+13
The earlier version compiled but didn't actually work since sysconf(_SC_PHYS_PAGES) always fails (or so I was told). Thanks to Ole André Vadla Ravnås for the patch and testing.
2015-03-07xz: size_t/uint32_t cleanup in options.c.Lasse Collin1-6/+6
2015-03-07xz: Fix a comment and silence a warning in message.c.Lasse Collin1-2/+3
2015-03-07liblzma: Silence more uint32_t vs. size_t warnings.Lasse Collin2-2/+2
2015-03-07xz: Make arg_count an unsigned int to silence a warning.Lasse Collin2-2/+2
Actually the value of arg_count cannot exceed INT_MAX but it's nicer as an unsigned int.
2015-03-07liblzma: Fix a warning in index.c.Lasse Collin1-1/+3
2015-02-26Bump version and soname for 5.2.1.larhzu/v5.2.1Lasse Collin2-2/+2