aboutsummaryrefslogtreecommitdiff
path: root/src/liblzma (follow)
AgeCommit message (Collapse)AuthorFilesLines
2022-11-27liblzma: Pass the Filter ID to LZ encoder and decoder.Lasse Collin14-12/+26
This allows using two Filter IDs with the same initialization function and data structures.
2022-11-27liblzma: Remove two FIXME comments.Lasse Collin1-2/+2
2022-11-26liblzma: Use lzma_filters_free() in more places.Lasse Collin3-38/+8
2022-11-25liblzma: Omit simple coder init functions if they are disabled.Lasse Collin6-0/+24
2022-11-24liblzma: Allow nice_len 2 and 3 even if match finder requires 3 or 4.Lasse Collin3-8/+26
That is, if the specified nice_len is smaller than the minimum of the match finder, silently use the match finder's minimum value instead of reporting an error. The old behavior is annoying to users and it complicates xz options handling too.
2022-11-24liblzma: Add lzma_filters_update() support to the multi-threaded encoder.Lasse Collin2-23/+109
A tiny downside of this is that now a 1-4 tiny allocations are made for every Block because each worker thread needs its own copy of the filter chain.
2022-11-24Build: Don't put GNU/Linux-specific symbol versions into static liblzma.Lasse Collin1-0/+12
It not only makes no sense to put symbol versions into a static library but it can also cause breakage. By default Libtool #defines PIC if building a shared library and doesn't define it for static libraries. This is documented in the Libtool manual. It can be overriden using --with-pic or --without-pic. configure.ac detects if --with-pic or --without-pic is used and then gives an error if neither --disable-shared nor --disable-static was used at the same time. Thus, in normal situations it works to build both shared and static library at the same time on GNU/Linux, only --with-pic or --without-pic requires that only one type of library is built. Thanks to John Paul Adrian Glaubitz from Debian for reporting the problem that occurred on ia64: https://www.mail-archive.com/xz-devel@tukaani.org/msg00610.html
2022-11-24liblzma: Refactor to use lzma_filters_free().Lasse Collin2-21/+6
lzma_filters_free() sets the options to NULL and ids to LZMA_VLI_UNKNOWN so there is no need to do it by caller; the filter arrays will always be left in a safe state. Also use memcpy() instead of a loop to copy a filter chain when it is known to be safe to copy LZMA_FILTERS_MAX + 1 (even if the elements past the terminator might be uninitialized).
2022-11-24liblzma: Fix another invalid free() after memory allocation failure.Lasse Collin1-0/+4
This time it can happen when lzma_stream_encoder_mt() is used to reinitialize an existing multi-threaded Stream encoder and one of 1-4 tiny allocations in lzma_filters_copy() fail. It's very similar to the previous bug 10430fbf3820dafd4eafd38ec8be161a6978ed2b, happening with an array of lzma_filter structures whose old options are freed but the replacement never arrives due to a memory allocation failure in lzma_filters_copy().
2022-11-24liblzma: Add support for LZMA_SYNC_FLUSH in the Block encoder.Jia Tan1-0/+1
The documentation mentions that lzma_block_encoder() supports LZMA_SYNC_FLUSH but it was never added to supported_actions[] in the internal structure. Because of this, LZMA_SYNC_FLUSH could not be used with the Block encoder unless it was the next coder after something like stream_encoder() or stream_encoder_mt().
2022-11-24liblzma: Add new API function lzma_filters_free().Lasse Collin4-0/+49
This is small but convenient and should have been added a long time ago.
2022-11-23liblzma: Add lzma_attr_warn_unused_result to lzma_filters_copy().Lasse Collin1-1/+2
2022-11-23liblzma: Fix invalid free() after memory allocation failure.Lasse Collin1-8/+31
The bug was in the single-threaded .xz Stream encoder in the code that is used for both re-initialization and for lzma_filters_update(). To trigger it, an application had to either re-initialize an existing encoder instance with lzma_stream_encoder() or use lzma_filters_update(), and then one of the 1-4 tiny allocations in lzma_filters_copy() (called from stream_encoder_update()) must fail. An error was correctly reported but the encoder state was corrupted. This is related to the recent fix in f8ee61e74eb40600445fdb601c374d582e1e9c8a which is good but it wasn't enough to fix the main problem in stream_encoder.c.
2022-11-22liblzma: Fix language in a comment.Lasse Collin1-1/+1
2022-11-22liblzma: Fix infinite loop in LZMA encoder init with dict_size >= 2 GiB.Lasse Collin1-4/+15
The encoder doesn't support dictionary sizes larger than 1536 MiB. This is validated, for example, when calculating the memory usage via lzma_raw_encoder_memusage(). It is also enforced by the LZ part of the encoder initialization. However, LZMA encoder with LZMA_MODE_NORMAL did an unsafe calculation with dict_size before such validation and that results in an infinite loop if dict_size was 2 << 30 or greater.
2022-11-21liblzma: Fix two Doxygen commands in the API headers.Lasse Collin2-2/+2
These were caught by clang -Wdocumentation.
2022-11-15Bump version number for 5.3.4alpha.larhzu/v5.3.4alphaLasse Collin3-3/+3
2022-11-14Revert "liblzma: Simple/BCJ filters: Allow disabling generic BCJ options."Lasse Collin9-11/+10
This reverts commit 177bdc922cb17bd0fd831ab8139dfae912a5c2b8 and also does equivalent change to arm64.c. Now that ARM64 filter will use lzma_options_bcj, this change is not needed anymore.
2022-11-14Replace the experimental ARM64 filter with a new experimental version.Lasse Collin7-237/+136
This is incompatible with the previous version. This has space/tab fixes in filter_*.c and bcj.h too.
2022-11-14liblzma: Add fast CRC64 for 32/64-bit x86 using SSSE3 + SSE4.1 + CLMUL.Lasse Collin2-9/+461
It also works on E2K as it supports these intrinsics. On x86-64 runtime detection is used so the code keeps working on older processors too. A CLMUL-only build can be done by using -msse4.1 -mpclmul in CFLAGS and this will reduce the library size since the generic implementation and its 8 KiB lookup table will be omitted. On 32-bit x86 this isn't used by default for now because by default on 32-bit x86 the separate assembly file crc64_x86.S is used. If --disable-assembler is used then this new CLMUL code is used the same way as on 64-bit x86. However, a CLMUL-only build (-msse4.1 -mpclmul) won't omit the 8 KiB lookup table on 32-bit x86 due to a currently-missing check for disabled assembler usage. The configure.ac check should be such that the code won't be built if something in the toolchain doesn't support it but --disable-clmul-crc option can be used to unconditionally disable this feature. CLMUL speeds up decompression of files that have compressed very well (assuming CRC64 is used as a check type). It is know that the CLMUL code is significantly slower than the generic code for tiny inputs (especially 1-8 bytes but up to 16 bytes). If that is a real-world problem then there is already a commented-out variant that uses the generic version for small inputs. Thanks to Ilya Kurdyukov for the original patch which was derived from a white paper from Intel [1] (published in 2009) and public domain code from [2] (released in 2016). [1] https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf [2] https://github.com/rawrunprotected/crc
2022-11-14liblzma: Use __attribute__((__constructor__)) if available.Lasse Collin3-1/+13
This uses it for CRC table initializations when using --disable-small. It avoids mythread_once() overhead. It also means that then --disable-small --disable-threads is thread-safe if this attribute is supported.
2022-11-11liblzma: Fix building with Intel ICC (the classic compiler).Lasse Collin1-1/+1
It claims __GNUC__ >= 10 but doesn't support __symver__ attribute. Thanks to Stephen Sachs.
2022-11-11liblzma: Fix incorrect #ifdef for x86 SSE2 support.Lasse Collin1-2/+1
__SSE2__ is the correct macro for SSE2 support with GCC, Clang, and ICC. __SSE2_MATH__ means doing floating point math with SSE2 instead of 387. Often the latter macro is defined if the first one is but it was still a bug.
2022-11-09liblzma: Add .lz support to lzma_auto_decoder().Lasse Collin4-11/+47
Thanks to Michał Górny for the original patch.
2022-11-09liblzma: Add .lz (lzip) decompression support (format versions 0 and 1).Lasse Collin5-2/+480
Support for format version 0 was removed from lzip 1.18 for some reason. .lz format version 0 files are rare (and old) but some source packages were released in this format, and some people might have personal files in this format too. It's very little extra code to support it along side format version 1 so this commits adds support for both. The Sync Flush marker extentension to the original .lz format version 1 isn't supported. It would require changes to the LZMA decoder itself. Such files are very rare anyway. See the API doc for lzma_lzip_decoder() for more details about the .lz format support. Thanks to Michał Górny for the original patch.
2022-11-09liblzma: Add the missing Makefile.inc change for --disable-microlzma.Lasse Collin1-2/+10
This was forgotten from commit 59c4d6e1390f6f4176f43ac1dad1f7ac03c449b8.
2022-11-08liblzma: Update API docs about decoder flags.Lasse Collin1-4/+17
2022-11-08liblzma: Use the return_if_error() macro in alone_decoder.c.Lasse Collin1-4/+2
2022-11-08liblzma: Fix a comment in auto_decoder.c.Lasse Collin1-2/+2
2022-11-07liblzma: Include cached memory in reported memusage in threaded decoder.Lasse Collin1-3/+19
This affects lzma_memusage() and lzma_memlimit_set() when used with the threaded decompressor. Now all allocations are reported by lzma_memusage() (so it's not misleading) and lzma_memlimit_set() cannot lower the limit below that value. The alternative would have been to allow lowering the limit if doing so is possible by freeing the cached memory but since the primary use case of lzma_memlimit_set() is to increase memlimit after LZMA_MEMLIMIT_ERROR this simple approach was selected. The cached memory was always included when enforcing the memory usage limit while decoding. Thanks to Jia Tan.
2022-10-31liblzma: Silence -Wconversion warning from crc64_fast.c.Lasse Collin1-2/+3
2022-10-24liblzma: Threaded decoder: Stop the worker threads on errors.Lasse Collin1-7/+26
It's waste of CPU time and electricity to leave the unfinished worker threads running when it is known that their output will get ignored.
2022-10-05liblzma: Fix a compilation issue when encoders are disabled.Jia Tan1-3/+4
When encoders were disabled and threading enabled, outqueue.c and outqueue.h were not compiled. The multi threaded decoder required these files, so compilation failed.
2022-10-05tuklib_integer: Add 64-bit endianness-converting reads and writes.Lasse Collin1-6/+3
Also update the comment in liblzma's memcmplen.h. Thanks to Michał Górny for the original patch for the reads.
2022-09-30liblzma: Add API doc note about the .xz decoder LZMA_MEMLIMIT_ERROR bug.Lasse Collin1-0/+11
The bug was fixed in 660739f99ab211edec4071de98889fb32ed04e98.
2022-09-28liblzma: Add dest and src NULL checks to lzma_index_cat.Jia Tan1-0/+3
The documentation states LZMA_PROG_ERROR can be returned from lzma_index_cat. Previously, lzma_index_cat could not return LZMA_PROG_ERROR. Now, the validation is similar to lzma_index_append, which does a NULL check on the index parameter.
2022-09-28liblzma: Fix copying of check type statistics in lzma_index_cat().Jia Tan1-1/+6
The check type of the last Stream in dest was never copied to dest->checks (the code tried to copy it but it was done too late). This meant that the value returned by lzma_index_checks() would only include the check type of the last Stream when multiple lzma_indexes had been concatenated. In xz --list this meant that the summary would only list the check type of the last Stream, so in this sense this was only a visual bug. However, it's possible that some applications use this information for purposes other than merely showing it to the users in an informational message. I'm not aware of such applications though and it's quite possible that such applications don't exist. Regular streamed decompression in xz or any other application doesn't use lzma_index_cat() and so this bug cannot affect them.
2022-09-28liblzma: Stream decoder: Fix restarting after LZMA_MEMLIMIT_ERROR.Lasse Collin1-3/+13
If lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible to use lzma_memlimit_set() to increase the limit and continue decoding. This was supposed to work from the beginning but there was a bug. With other decoders (.lzma or threaded .xz) this already worked correctly.
2022-09-28liblzma: Stream decoder: Fix comments.Lasse Collin1-7/+5
2022-09-20liblzma: ARM64: Add comments.Lasse Collin1-0/+13
2022-09-20liblzma: ARM64: Fix wrong comment in API doc.Lasse Collin1-2/+2
Thanks to Jia Tan.
2022-09-19liblzma: Add experimental ARM64 BCJ filter with a temporary Filter ID.Lasse Collin9-1/+308
That is, the Filter ID will be changed once the design is final. The current version will be removed. So files created with the tempoary Filter ID won't be supported in the future.
2022-09-17liblzma: Simple/BCJ filters: Allow disabling generic BCJ options.Lasse Collin8-9/+10
This will be needed for the ARM64 BCJ filter as it will use its own options struct.
2022-09-09liblzma: lzma_filters_copy: Keep dest[] unmodified if an error occurs.Lasse Collin2-7/+15
lzma_stream_encoder() and lzma_stream_encoder_mt() always assumed this. Before this patch, failing lzma_filters_copy() could result in free(invalid_pointer) or invalid memory reads in stream_encoder.c or stream_encoder_mt.c. To trigger this, allocating memory for a filter options structure has to fail. These are tiny allocations so in practice they very rarely fail. Certain badness in the filter chain array could also make lzma_filters_copy() fail but both stream_encoder.c and stream_encoder_mt.c validate the filter chain before trying to copy it, so the crash cannot occur this way.
2022-09-08liblzma: lzma_index_append: Add missing integer overflow check.Jia Tan1-0/+4
The documentation in src/liblzma/api/lzma/index.h suggests that both the unpadded (compressed) size and the uncompressed size are checked for overflow, but only the unpadded size was checked. The uncompressed check is done first since that is more likely to occur than the unpadded or index field size overflows.
2022-09-08liblzma: Vaccinate against an ill patch from RHEL/CentOS 7.Lasse Collin9-14/+360
RHEL/CentOS 7 shipped with 5.1.2alpha, including the threaded encoder that is behind #ifdef LZMA_UNSTABLE in the API headers. In 5.1.2alpha these symbols are under XZ_5.1.2alpha in liblzma.map. API/ABI compatibility tracking isn't done between development releases so newer releases didn't have XZ_5.1.2alpha anymore. Later RHEL/CentOS 7 updated xz to 5.2.2 but they wanted to keep the exported symbols compatible with 5.1.2alpha. After checking the ABI changes it turned out that >= 5.2.0 ABI is backward compatible with the threaded encoder functions from 5.1.2alpha (but not vice versa as fixes and extensions to these functions were made between 5.1.2alpha and 5.2.0). In RHEL/CentOS 7, XZ Utils 5.2.2 was patched with xz-5.2.2-compat-libs.patch to modify liblzma.map: - XZ_5.1.2alpha was added with lzma_stream_encoder_mt and lzma_stream_encoder_mt_memusage. This matched XZ Utils 5.1.2alpha. - XZ_5.2 was replaced with XZ_5.2.2. It is clear that this was an error; the intention was to keep using XZ_5.2 (XZ_5.2.2 has never been used in XZ Utils). So XZ_5.2.2 lists all symbols that were listed under XZ_5.2 before the patch. lzma_stream_encoder_mt and _mt_memusage are included too so they are listed both here and under XZ_5.1.2alpha. The patch didn't add any __asm__(".symver ...") lines to the .c files. Thus the resulting liblzma.so exports the threaded encoder functions under XZ_5.1.2alpha only. Listing the two functions also under XZ_5.2.2 in liblzma.map has no effect without matching .symver lines. The lack of XZ_5.2 in RHEL/CentOS 7 means that binaries linked against unpatched XZ Utils 5.2.x won't run on RHEL/CentOS 7. This is unfortunate but this alone isn't too bad as the problem is contained within RHEL/CentOS 7 and doesn't affect users of other distributions. It could also be fixed internally in RHEL/CentOS 7. The second problem is more serious: In XZ Utils 5.2.2 the API headers don't have #ifdef LZMA_UNSTABLE for obvious reasons. This is true in RHEL/CentOS 7 version too. Thus now programs using new APIs can be compiled without an extra #define. However, the programs end up depending on symbol version XZ_5.1.2alpha (and possibly also XZ_5.2.2) instead of XZ_5.2 as they would with an unpatched XZ Utils 5.2.2. This means that such binaries won't run on other distributions shipping XZ Utils >= 5.2.0 as they don't provide XZ_5.1.2alpha or XZ_5.2.2; they only provide XZ_5.2 (and XZ_5.0). (This includes RHEL/CentOS 8 as the patch luckily isn't included there anymore with XZ Utils 5.2.4.) Binaries built by RHEL/CentOS 7 users get distributed and then people wonder why they don't run on some other distribution. Seems that people have found out about the patch and been copying it to some build scripts, seemingly curing the symptoms but actually spreading the illness further and outside RHEL/CentOS 7. The ill patch seems to be from late 2016 (RHEL 7.3) and in 2017 it had spread at least to EasyBuild. I heard about the events only recently. :-( This commit splits liblzma.map into two versions: one for GNU/Linux and another for other OSes that can use symbol versioning (FreeBSD, Solaris, maybe others). The Linux-specific file and the matching additions to .c files add full compatibility with binaries that have been built against a RHEL/CentOS-patched liblzma. Builds for OSes other than GNU/Linux won't get the vaccine as they should be immune to the problem (I really hope that no build script uses the RHEL/CentOS 7 patch outside GNU/Linux). The RHEL/CentOS compatibility symbols XZ_5.1.2alpha and XZ_5.2.2 are intentionally put *after* XZ_5.2 in liblzma_linux.map. This way if one forgets to #define HAVE_SYMBOL_VERSIONS_LINUX when building, the resulting liblzma.so.5 will have lzma_stream_encoder_mt@@XZ_5.2 since XZ_5.2 {...} is the first one that lists that function. Without HAVE_SYMBOL_VERSIONS_LINUX @XZ_5.1.2alpha and @XZ_5.2.2 will be missing but that's still a minor problem compared to only having lzma_stream_encoder_mt@@XZ_5.1.2alpha! The "local: *;" line was moved to XZ_5.0 so that it doesn't need to be moved around. It doesn't matter where it is put. Having two similar liblzma_*.map files is a bit silly as it is, at least for now, easily possible to generate the generic one from the Linux-specific file. But that adds extra steps and increases the risk of mistakes when supporting more than one build system. So I rather maintain two files in parallel and let validate_map.sh check that they are in sync when "make mydist" is run. This adds .symver lines for lzma_stream_encoder_mt@XZ_5.2.2 and lzma_stream_encoder_mt_memusage@XZ_5.2.2 even though these weren't exported by RHEL/CentOS 7 (only @@XZ_5.1.2alpha was for these two). I added these anyway because someone might misunderstand the RHEL/CentOS 7 patch and think that @XZ_5.2.2 (@@XZ_5.2.2) versions were exported too. At glance one could suggest using __typeof__ to copy the function prototypes when making aliases. However, this doesn't work trivially because __typeof__ won't copy attributes (lzma_nothrow, lzma_pure) and it won't change symbol visibility from hidden to default (done by LZMA_API()). Attributes could be copied with __copy__ attribute but that needs GCC 9 and a fallback method would be needed anyway. This uses __symver__ attribute with GCC >= 10 and __asm__(".symver ...") with everything else. The attribute method is required for LTO (-flto) support with GCC. Using -flto with GCC older than 10 is now broken on GNU/Linux and will not be fixed (can silently result in a broken liblzma build that has dangerously incorrect symbol versions). LTO builds with Clang seem to work with the traditional __asm__(".symver ...") method. Thanks to Boud Roukema for reporting the problem and discussing the details and testing the fix.
2022-08-22Bump version number for 5.3.3alpha.larhzu/v5.3.3alphaLasse Collin2-2/+2
2022-08-18liblzma: Threaded decoder: Improve LZMA_FAIL_FAST when LZMA_FINISH is used.Lasse Collin2-0/+48
It will now return LZMA_DATA_ERROR (not LZMA_OK or LZMA_BUF_ERROR) if LZMA_FINISH is used and there isn't enough input to finish decoding the Block Header or the Block. The use of LZMA_DATA_ERROR is simpler and the less risky than LZMA_BUF_ERROR but this might be changed before 5.4.0.
2022-07-25liblzma: Refactor lzma_mf_is_supported() to use a switch-statement.Jia Tan1-18/+14
2022-07-14liblzma: Rename a variable and improve a comment.Lasse Collin1-4/+9
2022-07-13liblzma: Add optional autodetection of LZMA end marker.Lasse Collin6-37/+86
Turns out that this is needed for .lzma files as the spec in LZMA SDK says that end marker may be present even if the size is stored in the header. Such files are rare but exist in the real world. The code in liblzma is so old that the spec didn't exist in LZMA SDK back then and I had understood that such files weren't possible (the lzma tool in LZMA SDK didn't create such files). This modifies the internal API so that LZMA decoder can be told if EOPM is allowed even when the uncompressed size is known. It's allowed with .lzma and not with other uses. Thanks to Karl Beldan for reporting the problem.
2022-05-23liblzma: Silence a warning.Lasse Collin1-1/+1
The actual initialization is done via mythread_sync and seems that GCC doesn't necessarily see that it gets initialized there.
2022-04-06liblzma: Threaded decoder: Improve setting of pending_error.Lasse Collin1-16/+35
It doesn't need to be done conditionally. The comments try to explain it.
2022-04-06liblzma: Add a new flag LZMA_FAIL_FAST for threaded decoder.Lasse Collin3-26/+56
In most cases if the input file is corrupt the application won't care about the uncompressed content at all. With this new flag the threaded decoder will return an error as soon as any thread has detected an error; it won't wait to copy out the data before the location of the error. I don't plan to use this in xz to keep the behavior consistent between single-threaded and multi-threaded modes.
2022-04-05liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.Lasse Collin1-2/+24
This makes the behavior consistent with the single-threaded decoder when handling truncated .xz files. Thanks to Jia Tan for finding this issue.
2022-04-02liblzma: Threaded decoder: Support zpipe.c-style decoding loop.Lasse Collin1-10/+67
This makes it possible to call lzma_code() in a loop that only reads new input when lzma_code() didn't fill the output buffer completely. That isn't the calling style suggested by the liblzma example program 02_decompress.c so perhaps the usefulness of this feature is limited. Also, it is possible to write such a loop so that it works with the single-threaded decoder but not with the threaded decoder even after this commit, or so that it works only if lzma_mt.timeout = 0. The zlib tutorial <https://zlib.net/zlib_how.html> is a well-known example of a loop where more input is read only when output isn't full. Porting this as is to liblzma would work with the single-threaded decoder (if LZMA_CONCATENATED isn't used) but it wouldn't work with threaded decoder even after this commit because the loop assumes that no more output is possible when it cannot read more input ("if (strm.avail_in == 0) break;"). This cannot be fixed at liblzma side; the loop has to be modified at least a little. I'm adding this in any case because the actual code is simple and short and should have no harmful side-effects in other situations.
2022-03-26liblzma: Fix a deadlock in threaded decoder.Lasse Collin1-19/+52
If a worker thread has consumed all input so far and it's waiting on thr->cond and then the main thread enables partial update for that thread, the code used to deadlock. This commit allows one dummy decoding pass to occur in this situation which then also does the partial update. As part of the fix, this moves thr->progress_* updates to avoid the second thr->mutex locking. Thanks to Jia Tan for finding, debugging, and reporting the bug.
2022-03-23liblzma: Threaded decoder: Don't stop threads on LZMA_TIMED_OUT.Lasse Collin1-1/+1
LZMA_TIMED_OUT is not an error and thus stopping threads on LZMA_TIMED_OUT breaks the decoder badly. Thanks to Jia Tan for finding the bug and for the patch.
2022-03-07liblzma: Add threaded .xz decompressor.Lasse Collin5-7/+1907
I realize that this is about a decade late. Big thanks to Sebastian Andrzej Siewior for the original patch. I made a bunch of smaller changes but after a while quite a few things got rewritten. So any bugs in the commit were created by me.
2022-03-06liblzma: Fix docs: lzma_block_decoder() cannot return LZMA_UNSUPPORTED_CHECK.Lasse Collin1-3/+0
If Check is unsupported, it will be silently ignored. It's the caller's job to handle it.
2022-03-06liblzma: Add new output queue (lzma_outq) features.Lasse Collin2-8/+79
Add lzma_outq_clear_cache2() which may leave one buffer allocated in the cache. Add lzma_outq_outbuf_memusage() to get the memory needed for a single lzma_outbuf. This is now used internally in outqueue.c too. Track both the total amount of memory allocated and the amount of memory that is in active use (not in cache). In lzma_outbuf, allow storing the current input position that matches the current output position. This way the main thread can notice when no more output is possible without first providing more input. Allow specifying return code for lzma_outq_read() in a finished lzma_outbuf.
2022-03-06liblzma: Index hash: Change return value type of hash_append() to void.Lasse Collin1-6/+5
2022-02-22liblzma: Minor addition to lzma_vli_size() API doc.Lasse Collin1-0/+2
Thanks to Jia Tan.
2022-02-22liblzma: Check the return value of lzma_index_append() in threaded encoder.Lasse Collin1-2/+5
If lzma_index_append() failed (most likely memory allocation failure) it could have gone unnoticed and the resulting .xz file would have an incorrect Index. Decompressing such a file would produce the correct uncompressed data but then an error would occur when verifying the Index field.
2022-02-22liblzma: Use non-executable stack on FreeBSD as on LinuxEd Maste2-4/+4
2022-02-20liblzma: Make Block decoder catch certain types of errors better.Lasse Collin1-25/+54
Now it limits the input and output buffer sizes that are passed to a raw decoder. This way there's no need to check if the sizes can grow too big or overflow when updating Compressed Size and Uncompressed Size counts. This also means that a corrupt file cannot cause the raw decoder to process useless extra input or output that would exceed the size info in Block Header (and thus cause LZMA_DATA_ERROR anyway). More importantly, now the size information is verified more carefully in case raw decoder returns LZMA_OK. This doesn't really matter with the current single-threaded .xz decoder as the errors would be detected slightly later anyway. But this helps avoiding corner cases in the upcoming threaded decompressor, and it might help other Block decoder uses outside liblzma too. The test files bad-1-lzma2-{9,10,11}.xz test these conditions. With the single-threaded .xz decoder the only difference is that LZMA_DATA_ERROR is detected in a difference place now.
2022-02-07liblzma: Add NULL checks to LZMA and LZMA2 properties encoders.jiat752-0/+6
Previously lzma_lzma_props_encode() and lzma_lzma2_props_encode() assumed that the options pointers must be non-NULL because the with these filters the API says it must never be NULL. It is good to do these checks anyway.
2022-02-06liblzma: Fix uint64_t vs. size_t confusion.Lasse Collin1-4/+7
This broke 32-bit builds due to a pointer type mismatch. This bug was introduced with the output-size-limited encoding in 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. Thanks to huangqinjin for the bug report.
2021-10-28Bump the version number for 5.3.2alpha.larhzu/v5.3.2alphaLasse Collin2-2/+2
2021-09-17liblzma: Fix liblzma.map for the lzma_microlzma_* symbols.Lasse Collin1-2/+2
This should have been part of d267d109c370a40b502e73f8664b154b15e4f253. Thanks to Gao Xiang.
2021-09-09liblzma: Use _MSVC_LANG to detect when "noexcept" can be used with MSVC.Lasse Collin1-1/+2
By default, MSVC always sets __cplusplus to 199711L. The real C++ standard version is available in _MSVC_LANG (or one could use /Zc:__cplusplus to set __cplusplus correctly). Fixes <https://sourceforge.net/p/lzmautils/discussion/708858/thread/f6bc3b108a/>. Thanks to Dan Weiss.
2021-09-05liblzma: Rename EROFS LZMA to MicroLZMA.Lasse Collin4-47/+52
It still exists primarily for EROFS but MicroLZMA is a more generic name (that hopefully doesn't clash with something that already exists).
2021-01-29liblzma: Fix unitialized variable.Lasse Collin1-0/+1
This was introduced two weeks ago in the commit 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. Thanks to Nathan Moinvaziri.
2021-01-24liblzma: Fix a wrong comment in stream_encoder_mt.c.Lasse Collin1-3/+7
2021-01-17liblzma: In EROFS LZMA decoder, verify that comp_size matches at the end.Lasse Collin1-1/+6
When the uncompressed size is known to be exact, after decompressing the stream exactly comp_size bytes of input must have been consumed. This is a minor improvement to error detection.
2021-01-17liblzma: Make EROFS LZMA decoder work when exact uncomp_size isn't known.Lasse Collin2-12/+91
The caller must still not specify an uncompressed size bigger than the actual uncompressed size. As a downside, this now needs the exact compressed size.
2021-01-14liblzma: Fix missing normalization in rc_encode_dummy().Lasse Collin1-1/+6
Without this fix it could attempt to create too much output.
2021-01-14liblzma: Add EROFS LZMA encoder and decoder.Lasse Collin5-0/+367
Right now this is just a planned extra-compact format for use in the EROFS file system in Linux. At this point it's possible that the format will either change or be abandoned and removed completely. The special thing about the encoder is that it uses the output-size-limited encoding added in the previous commit. EROFS uses fixed-sized blocks (e.g. 4 KiB) to hold compressed data so the compressors must be able to create valid streams that fill the given block size.
2021-01-14liblzma: Add rough support for output-size-limited encoding in LZMA1.Lasse Collin6-35/+246
With this it is possible to encode LZMA1 data without EOPM so that the encoder will encode as much input as it can without exceeding the specified output size limit. The resulting LZMA1 stream will be a normal LZMA1 stream without EOPM. The actual uncompressed size will be available to the caller via the uncomp_size pointer. One missing thing is that the LZMA layer doesn't inform the LZ layer when the encoding is finished and thus the LZ may read more input when it won't be used. However, this doesn't matter if encoding is done with a single call (which is the planned use case for now). For proper multi-call encoding this should be improved. This commit only adds the functionality for internal use. Nothing uses it yet.
2021-01-09liblzma: Make lzma_outq usable for threaded decompression too.Lasse Collin3-157/+301
Before this commit all output queue buffers were allocated as a single big allocation. Now each buffer is allocated separately when needed. Used buffers are cached to avoid reallocation overhead but the cache will keep only one buffer size at a time. This should make things work OK in the decompression where most of the time the buffer sizes will be the same but with some less common files the buffer sizes may vary. While this should work fine, it's still a bit preliminary and may even get reverted if it turns out to be useless for decompression.
2020-12-23liblzma: Enable Intel CET in x86 CRC assembly codesH.J. Lu2-0/+18
When Intel CET is enabled, we need to include <cet.h> in assembly codes to mark Intel CET support and add _CET_ENDBR to indirect jump targets. Tested on Intel Tiger Lake under CET enabled Linux.
2020-03-23Typo fixes from fossies.org.Lasse Collin3-3/+3
https://fossies.org/linux/misc/xz-5.2.5.tar.xz/codespell.html
2020-03-02liblzma: Fix a comment and RC_SYMBOLS_MAX.Lasse Collin1-2/+2
The comment didn't match the value of RC_SYMBOLS_MAX and the value itself was slightly larger than actually needed. The only harm about this was that memory usage was a few bytes larger.
2020-02-24liblzma: Remove unneeded <sys/types.h> from fastpos_tablegen.c.Lasse Collin1-1/+0
This file only generates fastpos_table.c. It isn't built as a part of liblzma.
2020-02-22Use defined(__GNUC__) before __GNUC__ in preprocessor lines.Lasse Collin1-2/+3
This should silence the equivalent of -Wundef in compilers that don't define __GNUC__.
2020-02-21liblzma: Add more uses of lzma_memcmplen() to the normal mode of LZMA.Lasse Collin1-6/+10
This gives a tiny encoder speed improvement. This could have been done in 2014 after the commit 544aaa3d13554e8640f9caf7db717a96360ec0f6 but it was forgotten.
2019-12-31Rename unaligned_read32ne to read32ne, and similarly for the others.Lasse Collin12-23/+19
2019-12-31Rename read32ne to aligned_read32ne, and similarly for the others.Lasse Collin2-4/+4
Using the aligned methods requires more care to ensure that the address really is aligned, so it's nicer if the aligned methods are prefixed. The next commit will remove the unaligned_ prefix from the unaligned methods which in liblzma are used in more places than the aligned ones.
2019-06-25liblzma: Fix a buggy comment.Lasse Collin1-1/+1
2019-06-24liblzma: Add a comment.Lasse Collin1-1/+1
2019-06-24liblzma: Silence clang -Wmissing-variable-declarations.Lasse Collin2-0/+6
2019-06-24Add LZMA_RET_INTERNAL1..8 to lzma_ret and use one for LZMA_TIMED_OUT.Lasse Collin3-7/+17
LZMA_TIMED_OUT is *internally* used as a value for lzma_ret enumeration. Previously it was #defined to 32 and cast to lzma_ret. That way it wasn't visible in the public API, but this was hackish. Now the public API has eight LZMA_RET_INTERNALx members and LZMA_TIMED_OUT is #defined to LZMA_RET_INTERNAL1. This way the code is cleaner overall although the public API has a few extra mysterious enum members.
2019-06-24liblzma: Remove incorrect uses of lzma_attribute((__unused__)).Lasse Collin3-6/+3
Caught by clang -Wused-but-marked-unused.
2019-06-23liblzma: Fix warnings from -Wsign-conversion.Lasse Collin11-29/+31
Also, more parentheses were added to the literal_subcoder macro in lzma_comon.h (better style but no functional change in the current usage).
2019-06-03liblzma: Fix comments.Lasse Collin6-7/+7
Thanks to Bruce Stark.
2019-06-02liblzma: Fix one more unaligned read to use unaligned_read16ne().Lasse Collin1-1/+1
2019-06-01liblzma: memcmplen: Use ctz32() from tuklib_integer.h.Lasse Collin1-9/+1
The same compiler-specific #ifdefs are already in tuklib_integer.h
2019-06-01liblzma: Use unaligned_readXXne functions instead of type punning.Lasse Collin2-7/+7
Now gcc -fsanitize=undefined should be clean. Thanks to Jeffrey Walton.
2019-05-13liblzma: Avoid memcpy(NULL, foo, 0) because it is undefined behavior.Lasse Collin3-5/+23
I should have always known this but I didn't. Here is an example as a reminder to myself: int mycopy(void *dest, void *src, size_t n) { memcpy(dest, src, n); return dest == NULL; } In the example, a compiler may assume that dest != NULL because passing NULL to memcpy() would be undefined behavior. Testing with GCC 8.2.1, mycopy(NULL, NULL, 0) returns 1 with -O0 and -O1. With -O2 the return value is 0 because the compiler infers that dest cannot be NULL because it was already used with memcpy() and thus the test for NULL gets optimized out. In liblzma, if a null-pointer was passed to memcpy(), there were no checks for NULL *after* the memcpy() call, so I cautiously suspect that it shouldn't have caused bad behavior in practice, but it's hard to be sure, and the problematic cases had to be fixed anyway. Thanks to Jeffrey Walton.
2019-05-11spellingAntoine Cœur10-12/+12
2018-10-26liblzma: Don't verify header CRC32s if building for fuzz testing.Lasse Collin4-5/+20
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is #defined when liblzma is being built for fuzz testing. Most fuzzed inputs would normally get rejected because of incorrect CRC32 and the actual header decoding code wouldn't get fuzzed. Disabling CRC32 checks avoids this problem. The fuzzer program must still use LZMA_IGNORE_CHECK flag to disable verification of integrity checks of uncompressed data.
2018-07-27liblzma: Remove an always-true condition from lzma_index_cat().Lasse Collin1-2/+2
This should help static analysis tools to see that newg isn't leaked. Thanks to Pavel Raiskup.
2018-05-19liblzma: Improve lzma_properties_decode() API documentation.Lasse Collin1-3/+4
2018-04-29Bump the version number to 5.3.1alpha.larhzu/v5.3.1alphaLasse Collin2-2/+2
2018-02-06nothrow: use noexcept for C++11 and newerBen Boeckel1-1/+5
In C++11, the `throw()` specifier is deprecated and `noexcept` is preffered instead.
2018-02-06liblzma: Remove incorrect #ifdef from range_common.h.Lasse Collin1-3/+1
In most cases it was harmless but it could affect some custom build systems. Thanks to Pippijn van Steenhoven.
2017-08-14Fix or hide warnings from GCC 7's -Wimplicit-fallthrough.Lasse Collin1-0/+6
2017-04-24liblzma: Add lzma_file_info_decoder().Lasse Collin4-1/+928
2017-04-21liblzma: Rename LZMA_SEEK to LZMA_SEEK_NEEDED and seek_in to seek_pos.Lasse Collin2-10/+10
2017-03-30liblzma: Make lzma_index_decoder_init() visible to other liblzma funcs.Lasse Collin3-5/+30
This is to allow other functions to use it without going via the public API (lzma_index_decoder()).
2017-03-30liblzma: Add generic support for input seeking (LZMA_SEEK).Lasse Collin2-2/+41
Also mention LZMA_SEEK in xz/message.c to silence a warning.
2017-03-30liblzma: Fix lzma_memlimit_set(strm, 0).Lasse Collin3-3/+13
The 0 got treated specially in a buggy way and as a result the function did nothing. The API doc said that 0 was supposed to return LZMA_PROG_ERROR but it didn't. Now 0 is treated as if 1 had been specified. This is done because 0 is already used to indicate an error from lzma_memlimit_get() and lzma_memusage(). In addition, lzma_memlimit_set() no longer checks that the new limit is at least LZMA_MEMUSAGE_BASE. It's counter-productive for the Index decoder and was actually needed only by the auto decoder. Auto decoder has now been modified to check for LZMA_MEMUSAGE_BASE.
2017-03-30liblzma: Similar memlimit fix for stream_, alone_, and auto_decoder.Lasse Collin4-16/+20
2017-03-30liblzma: Fix handling of memlimit == 0 in lzma_index_decoder().Lasse Collin2-9/+13
It returned LZMA_PROG_ERROR, which was done to avoid zero as the limit (because it's a special value elsewhere), but using LZMA_PROG_ERROR is simply inconvenient and can cause bugs. The fix/workaround is to treat 0 as if it were 1 byte. It's effectively the same thing. The only weird consequence is that then lzma_memlimit_get() will return 1 even when 0 was specified as the limit. This fixes a very rare corner case in xz --list where a specific memory usage limit and a multi-stream file could print the error message "Internal error (bug)" instead of saying that the memory usage limit is too low.
2016-11-21liblzma: Avoid multiple definitions of lzma_coder structures.Lasse Collin35-423/+532
Only one definition was visible in a translation unit. It avoided a few casts and temp variables but seems that this hack doesn't work with link-time optimizations in compilers as it's not C99/C11 compliant. Fixes: http://www.mail-archive.com/xz-devel@tukaani.org/msg00279.html
2016-03-13liblzma: Disable external SHA-256 by default.Lasse Collin1-10/+6
This is the sane thing to do. The conflict with OpenSSL on some OSes and especially that the OS-provided versions can be significantly slower makes it clear that it was a mistake to have the external SHA-256 support enabled by default. Those who want it can now pass --enable-external-sha256 to configure. INSTALL was updated with notes about OSes where this can be a bad idea. The SHA-256 detection code in configure.ac had some bugs that could lead to a build failure in some situations. These were fixed, although it doesn't matter that much now that the external SHA-256 is disabled by default. MINIX >= 3.2.0 uses NetBSD's libc and thus has SHA256_Init in libc instead of libutil. Support for the libutil version was removed.
2015-11-04liblzma: Make Valgrind happier with optimized (gcc -O2) liblzma.Lasse Collin1-0/+4
When optimizing, GCC can reorder code so that an uninitialized value gets used in a comparison, which makes Valgrind unhappy. It doesn't happen when compiled with -O0, which I tend to use when running Valgrind. Thanks to Rich Prohaska. I remember this being mentioned long ago by someone else but nothing was done back then.
2015-11-03liblzma: Rename lzma_presets.c back to lzma_encoder_presets.c.Lasse Collin2-2/+2
It would be too annoying to update other build systems just because of this.
2015-11-03Build: Build LZMA1/2 presets also when only decoder is wanted.Lasse Collin2-2/+7
People shouldn't rely on the presets when decoding raw streams, but xz uses the presets as the starting point for raw decoder options anyway. lzma_encocder_presets.c was renamed to lzma_presets.c to make it clear it's not used solely by the encoder code.
2015-11-03Build: Don't omit lzma_cputhreads() unless using --disable-threads.Lasse Collin1-1/+4
Previously it was omitted if encoders were disabled with --disable-encoders. It didn't make sense and it also broke the build.
2015-11-02liblzma: Fix a build failure related to external SHA-256 support.Lasse Collin1-9/+23
If an appropriate header and structure were found by configure, but a library with a usable SHA-256 functions wasn't, the build failed.
2015-10-12liblzma: Fix lzma_index_dup() for empty Streams.Lasse Collin1-5/+6
Stream Flags and Stream Padding weren't copied from empty Streams.
2015-10-12liblzma: Add a note to index.c for those using static analyzers.Lasse Collin1-0/+3
2015-10-12liblzma: Fix a memory leak in error path of lzma_index_dup().Lasse Collin1-9/+9
lzma_index_dup() calls index_dup_stream() which, in case of an error, calls index_stream_end() to free memory allocated by index_stream_init(). However, it illogically didn't actually free the memory. To make it logical, the tree handling code was modified a bit in addition to changing index_stream_end(). Thanks to Evan Nemerson for the bug report.
2015-07-12liblzma: A MSVC-specific hack isn't needed with MSVC 2013 and newer.Lasse Collin1-5/+13
2015-03-30Bump version to 5.3.0alpha and soname to 5.3.99.Lasse Collin2-4/+4
The idea of 99 is that it looks a bit weird in this context. For new features there's no API/ABI stability in devel versions.
2015-03-07liblzma: Silence more uint32_t vs. size_t warnings.Lasse Collin2-2/+2
2015-03-07liblzma: Fix a warning in index.c.Lasse Collin1-1/+3
2015-02-26Bump version and soname for 5.2.1.larhzu/v5.2.1Lasse Collin2-2/+2
2015-02-21liblzma: Fix a compression-ratio regression in LZMA1/2 in fast mode.Lasse Collin1-1/+1
The bug was added in the commit f48fce093b07aeda95c18850f5e086d9f2383380 and thus affected 5.1.4beta and 5.2.0. Luckily the bug cannot cause data corruption or other nasty things.
2015-01-26liblzma: Set LZMA_MEMCMPLEN_EXTRA depending on the compare method.Lasse Collin1-5/+10
2015-01-26liblzma: Silence harmless Valgrind errors.Lasse Collin1-0/+6
Thanks to Torsten Rupp for reporting this. I had forgotten to run Valgrind before the 5.2.0 release.
2014-12-21Bump version and soname for 5.2.0.Lasse Collin3-5/+5
I know that soname != app version, but I skip AGE=1 in -version-info to make the soname match the liblzma version anyway. It doesn't hurt anything as long as it doesn't conflict with library versioning rules.
2014-12-02liblzma: Document how lzma_mt.block_size affects memory usage.Lasse Collin1-0/+4
2014-11-26Remove LZMA_UNSTABLE macro.Lasse Collin2-6/+0
2014-11-26liblzma: Update lzma_stream_encoder_mt() API docs.Lasse Collin1-2/+3
2014-11-25liblzma: Verify the filter chain in threaded encoder initialization.Lasse Collin1-3/+6
This way an invalid filter chain is detected at the Stream encoder initialization instead of delaying it to the first call to lzma_code() which triggers the initialization of the actual filter encoder(s).
2014-11-10liblzma: Fix lzma_mt.preset in lzma_stream_encoder_mt_memusage().Lasse Collin1-2/+1
It read the filter chain from a wrong variable. This is a similar bug that was fixed in 9494fb6d0ff41c585326f00aa8f7fe58f8106a5e.
2014-10-29Build: Prepare to support Automake's subdir-objects.Lasse Collin1-2/+2
Due to a bug in Automake, subdir-objects won't be enabled for now. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17354 Thanks to Daniel Richard G. for the original patches.
2014-09-20liblzma: Fix a portability problem in Makefile.am.Lasse Collin1-1/+1
POSIX supports $< only in inference rules (suffix rules). Using it elsewhere is a GNU make extension and doesn't work e.g. with OpenBSD make. Thanks to Christian Weisgerber for the patch.
2014-09-14Bump the version number to 5.1.4beta.larhzu/v5.1.4betaLasse Collin2-3/+3
2014-08-05liblzma: Add support for LZMA_IGNORE_CHECK.Lasse Collin3-2/+37
2014-08-05liblzma: Add support for lzma_block.ignore_check.Lasse Collin7-20/+68
Note that this slightly changes how lzma_block_header_decode() has been documented. Earlier it said that the .version is set to the lowest required value, but now it says that the .version field is kept unchanged if possible. In practice this doesn't affect any old code, because before this commit the only possible .version was 0.
2014-08-04liblzma: Use lzma_memcmplen() in the BT3 match finder.Lasse Collin1-3/+2
I had missed this when writing the commit 5db75054e900fa06ef5ade5f2c21dffdd5d16141. Thanks to Jun I Jin.
2014-08-03liblzma: SHA-256: Optimize the Maj macro slightly.Lasse Collin1-1/+1
The Maj macro is used where multiple things are added together, so making Maj a sum of two expressions allows some extra freedom for the compiler to schedule the instructions. I learned this trick from <http://www.hackersdelight.org/corres.txt>.
2014-08-03liblzma: SHA-256: Optimize the way rotations are done.Lasse Collin1-6/+11
This looks weird because the rotations become sequential, but it helps quite a bit on both 32-bit and 64-bit x86: - It requires fewer instructions on two-operand instruction sets like x86. - It requires one register less which matters especially on 32-bit x86. I hope this doesn't hurt other archs. I didn't invent this idea myself, but I don't remember where I saw it first.
2014-08-03liblzma: SHA-256: Remove the GCC #pragma that became unneeded.Lasse Collin1-5/+0
The unrolling in the previous commit should avoid the situation where a compiler may think that an uninitialized variable might be accessed.
2014-08-03liblzma: SHA-256: Unroll a little more.Lasse Collin1-9/+16
This way a branch isn't needed for each operation to choose between blk0 and blk2, and still the code doesn't grow as much as it would with full unrolling.
2014-08-03liblzma: SHA-256: Do the byteswapping without a temporary buffer.Lasse Collin1-12/+1
2014-07-25liblzma: Use lzma_memcmplen() in normal mode of LZMA.Lasse Collin1-15/+5
Two locations were not changed yet because the simplest change assumes that the initial "len" may be greater than "limit".
2014-07-25liblzma: Simplify LZMA fast mode code by using memcmp().Lasse Collin1-10/+1
2014-07-25liblzma: Use lzma_memcmplen() in fast mode of LZMA.Lasse Collin1-3/+3
2014-07-25liblzma: Use lzma_memcmplen() in the match finders.Lasse Collin2-23/+23
This doesn't change the match finder output.
2014-07-25liblzma: Add lzma_memcmplen() for fast memory comparison.Lasse Collin2-0/+171
This commit just adds the function. Its uses will be in separate commits. This hasn't been tested much yet and it's perhaps a bit early to commit it but if there are bugs they should get found quite quickly. Thanks to Jun I Jin from Intel for help and for pointing out that string comparison needs to be optimized in liblzma.
2014-06-18liblzma: Add lzma_cputhreads().Lasse Collin5-1/+45
2014-05-25liblzma: Use lzma_alloc_zero() in LZ encoder initialization.Lasse Collin3-55/+62
This avoids a memzero() call for a newly-allocated memory, which can be expensive when encoding small streams with an over-sized dictionary. To avoid using lzma_alloc_zero() for memory that doesn't need to be zeroed, lzma_mf.son is now allocated separately, which requires handling it separately in normalize() too. Thanks to Vincenzo Innocente for reporting the problem.
2014-05-25liblzma: Add the internal function lzma_alloc_zero().Lasse Collin2-0/+27
2014-05-04liblzma: Rename the private API header lzma/lzma.h to lzma/lzma12.h.Lasse Collin3-3/+3
It can be confusing that two header files have the same name. The public API file is still lzma.h.
2014-01-29liblzma: Fix lzma_mt.preset not working with lzma_stream_encoder_mt().Lasse Collin1-2/+2
It read the filter chain from a wrong variable.
2014-01-20liblzma: Fix typo in a comment.Lasse Collin1-1/+1
2014-01-12liblzma: Avoid C99 compound literal arrays.Lasse Collin1-3/+5
MSVC 2013 doesn't like them. Maybe they aren't so good for readability either since many aren't used to them.
2014-01-12liblzma: Remove a useless C99ism from sha256.c.Lasse Collin1-1/+1
Unsurprisingly it makes no difference in compiled output.
2014-01-12Fix typos in comments.Lasse Collin1-1/+1
2013-11-26liblzma: Document the need for block->check for lzma_block_header_decode().Lasse Collin1-0/+3
Thanks to Tomer Chachamu.
2013-10-26Bump the version number to 5.1.3alpha.larhzu/v5.1.3alphaLasse Collin2-2/+2
2013-10-02liblzma: Support LZMA_FULL_FLUSH and _BARRIER in threaded encoder.Lasse Collin1-16/+39
Now --block-list=SIZES works with in the threaded mode too, although the performance is still bad due to the use of LZMA_FULL_FLUSH instead of the new LZMA_FULL_BARRIER.
2013-10-02liblzma: Add LZMA_FULL_BARRIER support to single-threaded encoder.Lasse Collin4-11/+54
In the single-threaded encoder LZMA_FULL_BARRIER is simply an alias for LZMA_FULL_FLUSH.
2013-09-17liblzma: Add block_buffer_encoder.h into Makefile.inc.Lasse Collin1-0/+1
This should have been in b465da5988dd59ad98fda10c2e4ea13d0b9c73bc.
2013-09-17Add native threading support on Windows.Lasse Collin1-47/+36
Now liblzma only uses "mythread" functions and types which are defined in mythread.h matching the desired threading method. Before Windows Vista, there is no direct equivalent to pthread condition variables. Since this package doesn't use pthread_cond_broadcast(), pre-Vista threading can still be kept quite simple. The pre-Vista code doesn't use anything that wasn't already available in Windows 95, so the binaries should run even on Windows 95 if someone happens to care.
2013-09-09Build: Create liblzma.pc in a src/liblzma/Makefile.am.Lasse Collin1-0/+20
Previously it was done in configure, but doing that goes against the Autoconf manual. Autoconf requires that it is possible to override e.g. prefix after running configure and that doesn't work correctly if liblzma.pc is created by configure. A potential downside of this change is that now e.g. libdir in liblzma.pc is a standalone string instead of being defined via ${prefix}, so if one overrides prefix when running pkg-config the libdir won't get the new value. I don't know if this matters in practice. Thanks to Vincent Torri.
2013-06-23liblzma: Avoid a warning about a shadowed variable.Lasse Collin1-2/+2
On Mac OS X wait() is declared in <sys/wait.h> that we include one way or other so don't use "wait" as a variable name. Thanks to Christian Kujau.
2013-03-23liblzma: Be less picky in lzma_alone_decoder().Lasse Collin3-11/+18
To avoid false positives when detecting .lzma files, rare values in dictionary size and uncompressed size fields were rejected. They will still be rejected if .lzma files are decoded with lzma_auto_decoder(), but when using lzma_alone_decoder() directly, such files will now be accepted. Hopefully this is an OK compromise. This doesn't affect xz because xz still has its own file format detection code. This does affect lzmadec though. So after this commit lzmadec will accept files that xz or xz-emulating-lzma doesn't. NOTE: lzma_alone_decoder() still won't decode all .lzma files because liblzma's LZMA decoder doesn't support lc + lp > 4. Reported here: http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/7068827
2013-03-23liblzma: Use lzma_block_buffer_bound64() in threaded encoder.Lasse Collin1-16/+50
Now it uses lzma_block_uncomp_encode() if the data doesn't fit into the space calculated by lzma_block_buffer_bound64().
2013-03-23liblzma: Fix another deadlock in the threaded encoder.Lasse Collin1-3/+6
This race condition could cause a deadlock if lzma_end() was called before finishing the encoding. This can happen with xz with debugging enabled (non-debugging version doesn't call lzma_end() before exiting).
2013-03-23liblzma: Add lzma_block_uncomp_encode().Lasse Collin4-31/+106
This also adds a new internal function lzma_block_buffer_bound64() which is similar to lzma_block_buffer_bound() but uses uint64_t instead of size_t.
2012-12-14Make the progress indicator smooth in threaded mode.Lasse Collin5-6/+116
This adds lzma_get_progress() to liblzma and takes advantage of it in xz. lzma_get_progress() collects progress information from the thread-specific structures so that fairly accurate progress information is available to applications. Adding a new function seemed to be a better way than making the information directly available in lzma_stream (like total_in and total_out are) because collecting the information requires locking mutexes. It's waste of time to do it more often than the up to date information is actually needed by an application.
2012-08-24A few typo fixes to comments and the xz man page.Lasse Collin1-1/+0
Thanks to Jim Meyering.
2012-07-17liblzma: Make the use of lzma_allocator const-correct.Lasse Collin71-219/+269
There is a tiny risk of causing breakage: If an application assigns lzma_stream.allocator to a non-const pointer, such code won't compile anymore. I don't know why anyone would do such a thing though, so in practice this shouldn't cause trouble. Thanks to Jan Kratochvil for the patch.
2012-07-05Build: Include validate_map.sh in the distribution.Lasse Collin1-1/+1
It's required by "make mydist". Fix also the location of EXTRA_DIST+= so that those files get distributed also if symbol versioning isn't enabled.
2012-07-04Bump the version number to 5.1.2alpha.larhzu/v5.1.2alphaLasse Collin2-2/+2
2012-06-28liblzma: Check that the first byte of range encoded data is 0x00.Lasse Collin2-5/+15
It is just to be more pedantic and thus perhaps catch broken files slightly earlier.
2012-05-28liblzma: Fix possibility of incorrect LZMA_BUF_ERROR.Lasse Collin1-1/+1
lzma_code() could incorrectly return LZMA_BUF_ERROR if all of the following was true: - The caller knows how many bytes of output to expect and only provides that much output space. - When the last output bytes are decoded, the caller-provided input buffer ends right before the LZMA2 end of payload marker. So LZMA2 won't provide more output anymore, but it won't know it yet and thus won't return LZMA_STREAM_END yet. - A BCJ filter is in use and it hasn't left any unfiltered bytes in the temp buffer. This can happen with any BCJ filter, but in practice it's more likely with filters other than the x86 BCJ. Another situation where the bug can be triggered happens if the uncompressed size is zero bytes and no output space is provided. In this case the decompression can fail even if the whole input file is given to lzma_code(). A similar bug was fixed in XZ Embedded on 2011-09-19.
2012-04-19liblzma: Remove outdated comments.Lasse Collin2-5/+1
2012-04-19liblzma: Fix Libs.private in liblzma.pc to include -lrt when needed.Lasse Collin1-1/+1
2011-10-23liblzma: Fix invalid free() in the threaded encoder.Lasse Collin1-0/+4
It was triggered if initialization failed e.g. due to running out of memory. Thanks to Arkadiusz Miskiewicz.
2011-10-23liblzma: Fix a deadlock in the threaded encoder.Lasse Collin1-1/+3
It was triggered when reinitializing the encoder, e.g. when encoding two files.
2011-06-16liblzma: Remove unneeded semicolon.Lasse Collin1-1/+1
2011-05-28liblzma: Use symbol versioning.Lasse Collin3-0/+179
Symbol versioning is enabled by default on GNU/Linux, other GNU-based systems, and FreeBSD. I'm not sure how stable this is, so it may need backward-incompatible changes before the next release. The idea is that alpha and beta symbols are considered unstable and require recompiling the applications that use those symbols. Once a symbol is stable, it may get extended with new features in ways that don't break compatibility with older ABI & API. The mydist target runs validate_map.sh which should catch some probable problems in liblzma.map. Otherwise I would forget to update the map file for new releases.
2011-05-27liblzma: Handle allocation failures correctly in lzma_index_init().Lasse Collin1-2/+5
Thanks to Jim Meyering.
2011-05-21liblzma: Try to use SHA-256 from the operating system.Lasse Collin2-8/+77
If the operating system libc or other base libraries provide SHA-256, use that instead of our own copy. Note that this doesn't use OpenSSL or libgcrypt or such libraries to avoid creating dependencies to other packages. This supports at least FreeBSD, NetBSD, OpenBSD, Solaris, MINIX, and Darwin. They all provide similar but not identical SHA-256 APIs; everyone is a little different. Thanks to Wim Lewis for the original patch, improvements, and testing.
2011-05-17Add underscores to attributes (__attribute((__foo__))).Lasse Collin16-23/+25
2011-04-12Bump the version number to 5.1.1alpha and liblzma soname to 5.0.99.larhzu/v5.1.1alphaLasse Collin2-2/+2
2011-04-12Put the unstable APIs behind #ifdef LZMA_UNSTABLE.Lasse Collin2-0/+6
This way people hopefully won't complain if these APIs change and break code that used an older API.
2011-04-12Remove doubled words from documentation and comments.Lasse Collin2-2/+2
Spot candidates by running these commands: git ls-files |xargs perl -0777 -n \ -e 'while (/\b(then?|[iao]n|i[fst]|but|f?or|at|and|[dt]o)\s+\1\b/gims)' \ -e '{$n=($` =~ tr/\n/\n/ + 1); ($v=$&)=~s/\n/\\n/g; print "$ARGV:$n:$v\n"}' Thanks to Jim Meyering for the original patch.
2011-04-11liblzma: Add lzma_stream_encoder_mt() for threaded compression.Lasse Collin7-1/+1538
This is the simplest method to do threading, which splits the uncompressed data into blocks and compresses them independently from each other. There's room for improvement especially to reduce the memory usage, but nevertheless, this is a good start.
2011-04-11liblzma: Add the forgotten lzma_lzma2_block_size().Lasse Collin2-0/+12
This should have been in 5eefc0086d24a65e136352f8c1d19cefb0cbac7a.
2011-04-11liblzma: Document lzma_easy_(enc|dec)oder_memusage() better too.Lasse Collin1-0/+9
2011-04-11liblzma: Document lzma_raw_(enc|dec)oder_memusage() better.Lasse Collin1-2/+6
It didn't mention the return value that is used if an error occurs.
2011-04-11liblzma: Use memzero() to initialize supported_actions[].Lasse Collin1-4/+2
This is cleaner and makes it simpler to add new members to lzma_action enumeration.
2011-04-11liblzma: API comment about lzma_allocator with threaded coding.Lasse Collin1-5/+13