aboutsummaryrefslogtreecommitdiff
path: root/src/liblzma/common (follow)
AgeCommit message (Collapse)AuthorFilesLines
2023-01-12liblzma: Silence warnings from clang -Wconditional-uninitialized.Lasse Collin2-4/+6
This is similar to 2ce4f36f179a81d0c6e182a409f363df759d1ad0. The actual initialization of the variables is done inside mythread_sync() macro. Clang doesn't seem to see that the initialization code inside the macro is always executed.
2023-01-05liblzma: Remove common.h include from common/index.h.Jia Tan4-2/+10
common/index.h is needed by liblzma internally and tests. common.h will include and define many things that are not needed by the tests. Also, this prevents include order problems because common.h will redefine LZMA_API resulting in a warning.
2023-01-02liblzma: Add NULL check to lzma_index_hash_append.Jia Tan1-1/+1
This is for consistency with lzma_index_append.
2023-01-02liblzma: Replaced hardcoded 0x0 index indicator byte with macroJia Tan6-5/+9
2022-12-30liblzma: Update documentation for lzma_filter_encoder.Jia Tan1-2/+5
2022-12-30liblzma: Fix lzma_microlzma_encoder() return value.Jia Tan1-1/+2
Using return_if_error on lzma_lzma_lclppb_encode was improper because return_if_error is expecting an lzma_ret value, but lzma_lzma_lclppb_encode returns a boolean. This could result in lzma_microlzma_encoder, which would be misleading for applications.
2022-12-08liblzma: Check for unexpected NULL pointers in block_header_decode().Lasse Collin1-0/+4
The API docs gave an impression that such checks are done but they actually weren't done. In practice it made little difference since the calling code has a bug if these are NULL. Thanks to Jia Tan for the original patch that checked for block->filters == NULL.
2022-12-01liblzma: Use __has_attribute(__symver__) to fix Clang detection.Lasse Collin1-1/+14
If someone sets up Clang to define __GNUC__ to 10 or greater then symvers broke. __has_attribute is supported by such GCC and Clang versions that don't support __symver__ so this should be much better and simpler way to detect if __symver__ is actually supported. Thanks to Tomasz Gajc for the bug report.
2022-12-01liblzma: Silence unused variable warning when BCJ filters are disabled.Lasse Collin1-0/+15
Thanks to Jia Tan for the original patch.
2022-11-28liblzma: Add lzma_str_to_filters, _from_filters, and _list_filters.Lasse Collin2-0/+1303
lzma_str_to_filters() uses static error messages which makes them not very precise. It tells the position in the string where an error occurred though which helps quite a bit if applications take advantage of it. Dynamic error messages can be added later with a new flag if it seems important enough.
2022-11-28liblzma: Make lzma_validate_chain() available outside filter_common.c.Lasse Collin2-4/+7
2022-11-28liblzma: Use LZMA1EXT feature in lzma_microlzma_decoder().Lasse Collin1-7/+8
Here too this avoids the slightly ugly method to set the uncompressed size. Also moved the setting of dict_size to the struct initializer.
2022-11-28liblzma: Use LZMA1EXT feature in lzma_alone_decoder().Lasse Collin1-5/+13
This avoids the need to use the slightly ugly method to set the uncompressed size.
2022-11-27liblzma: Add LZMA_FILTER_LZMA1EXT to support LZMA1 without end marker.Lasse Collin3-0/+22
Some file formats need support for LZMA1 streams that don't use the end of payload marker (EOPM) alias end of stream (EOS) marker. So far liblzma API has supported decompressing such streams via lzma_alone_decoder() when .lzma header specifies a known uncompressed size. Encoding support hasn't been available in the API. Instead of adding a new LZMA1-only API for this purpose, this commit adds a new filter ID for use with raw encoder and decoder. The main benefit of this approach is that then also filter chains are possible, for example, if someone wants to implement support for .7z files that use the x86 BCJ filter with LZMA1 (not BCJ2 as that isn't supported in liblzma).
2022-11-27liblzma: Pass the Filter ID to LZ encoder and decoder.Lasse Collin6-2/+10
This allows using two Filter IDs with the same initialization function and data structures.
2022-11-27liblzma: Remove two FIXME comments.Lasse Collin1-2/+2
2022-11-26liblzma: Use lzma_filters_free() in more places.Lasse Collin3-38/+8
2022-11-24liblzma: Add lzma_filters_update() support to the multi-threaded encoder.Lasse Collin1-8/+88
A tiny downside of this is that now a 1-4 tiny allocations are made for every Block because each worker thread needs its own copy of the filter chain.
2022-11-24Build: Don't put GNU/Linux-specific symbol versions into static liblzma.Lasse Collin1-0/+12
It not only makes no sense to put symbol versions into a static library but it can also cause breakage. By default Libtool #defines PIC if building a shared library and doesn't define it for static libraries. This is documented in the Libtool manual. It can be overriden using --with-pic or --without-pic. configure.ac detects if --with-pic or --without-pic is used and then gives an error if neither --disable-shared nor --disable-static was used at the same time. Thus, in normal situations it works to build both shared and static library at the same time on GNU/Linux, only --with-pic or --without-pic requires that only one type of library is built. Thanks to John Paul Adrian Glaubitz from Debian for reporting the problem that occurred on ia64: https://www.mail-archive.com/xz-devel@tukaani.org/msg00610.html
2022-11-24liblzma: Refactor to use lzma_filters_free().Lasse Collin2-21/+6
lzma_filters_free() sets the options to NULL and ids to LZMA_VLI_UNKNOWN so there is no need to do it by caller; the filter arrays will always be left in a safe state. Also use memcpy() instead of a loop to copy a filter chain when it is known to be safe to copy LZMA_FILTERS_MAX + 1 (even if the elements past the terminator might be uninitialized).
2022-11-24liblzma: Fix another invalid free() after memory allocation failure.Lasse Collin1-0/+4
This time it can happen when lzma_stream_encoder_mt() is used to reinitialize an existing multi-threaded Stream encoder and one of 1-4 tiny allocations in lzma_filters_copy() fail. It's very similar to the previous bug 10430fbf3820dafd4eafd38ec8be161a6978ed2b, happening with an array of lzma_filter structures whose old options are freed but the replacement never arrives due to a memory allocation failure in lzma_filters_copy().
2022-11-24liblzma: Add support for LZMA_SYNC_FLUSH in the Block encoder.Jia Tan1-0/+1
The documentation mentions that lzma_block_encoder() supports LZMA_SYNC_FLUSH but it was never added to supported_actions[] in the internal structure. Because of this, LZMA_SYNC_FLUSH could not be used with the Block encoder unless it was the next coder after something like stream_encoder() or stream_encoder_mt().
2022-11-24liblzma: Add new API function lzma_filters_free().Lasse Collin1-0/+26
This is small but convenient and should have been added a long time ago.
2022-11-23liblzma: Fix invalid free() after memory allocation failure.Lasse Collin1-8/+31
The bug was in the single-threaded .xz Stream encoder in the code that is used for both re-initialization and for lzma_filters_update(). To trigger it, an application had to either re-initialize an existing encoder instance with lzma_stream_encoder() or use lzma_filters_update(), and then one of the 1-4 tiny allocations in lzma_filters_copy() (called from stream_encoder_update()) must fail. An error was correctly reported but the encoder state was corrupted. This is related to the recent fix in f8ee61e74eb40600445fdb601c374d582e1e9c8a which is good but it wasn't enough to fix the main problem in stream_encoder.c.
2022-11-22liblzma: Fix language in a comment.Lasse Collin1-1/+1
2022-11-14Replace the experimental ARM64 filter with a new experimental version.Lasse Collin3-22/+21
This is incompatible with the previous version. This has space/tab fixes in filter_*.c and bcj.h too.
2022-11-11liblzma: Fix building with Intel ICC (the classic compiler).Lasse Collin1-1/+1
It claims __GNUC__ >= 10 but doesn't support __symver__ attribute. Thanks to Stephen Sachs.
2022-11-11liblzma: Fix incorrect #ifdef for x86 SSE2 support.Lasse Collin1-2/+1
__SSE2__ is the correct macro for SSE2 support with GCC, Clang, and ICC. __SSE2_MATH__ means doing floating point math with SSE2 instead of 387. Often the latter macro is defined if the first one is but it was still a bug.
2022-11-09liblzma: Add .lz support to lzma_auto_decoder().Lasse Collin3-7/+41
Thanks to Michał Górny for the original patch.
2022-11-09liblzma: Add .lz (lzip) decompression support (format versions 0 and 1).Lasse Collin2-0/+418
Support for format version 0 was removed from lzip 1.18 for some reason. .lz format version 0 files are rare (and old) but some source packages were released in this format, and some people might have personal files in this format too. It's very little extra code to support it along side format version 1 so this commits adds support for both. The Sync Flush marker extentension to the original .lz format version 1 isn't supported. It would require changes to the LZMA decoder itself. Such files are very rare anyway. See the API doc for lzma_lzip_decoder() for more details about the .lz format support. Thanks to Michał Górny for the original patch.
2022-11-09liblzma: Add the missing Makefile.inc change for --disable-microlzma.Lasse Collin1-2/+10
This was forgotten from commit 59c4d6e1390f6f4176f43ac1dad1f7ac03c449b8.
2022-11-08liblzma: Use the return_if_error() macro in alone_decoder.c.Lasse Collin1-4/+2
2022-11-08liblzma: Fix a comment in auto_decoder.c.Lasse Collin1-2/+2
2022-11-07liblzma: Include cached memory in reported memusage in threaded decoder.Lasse Collin1-3/+19
This affects lzma_memusage() and lzma_memlimit_set() when used with the threaded decompressor. Now all allocations are reported by lzma_memusage() (so it's not misleading) and lzma_memlimit_set() cannot lower the limit below that value. The alternative would have been to allow lowering the limit if doing so is possible by freeing the cached memory but since the primary use case of lzma_memlimit_set() is to increase memlimit after LZMA_MEMLIMIT_ERROR this simple approach was selected. The cached memory was always included when enforcing the memory usage limit while decoding. Thanks to Jia Tan.
2022-10-24liblzma: Threaded decoder: Stop the worker threads on errors.Lasse Collin1-7/+26
It's waste of CPU time and electricity to leave the unfinished worker threads running when it is known that their output will get ignored.
2022-10-05liblzma: Fix a compilation issue when encoders are disabled.Jia Tan1-3/+4
When encoders were disabled and threading enabled, outqueue.c and outqueue.h were not compiled. The multi threaded decoder required these files, so compilation failed.
2022-10-05tuklib_integer: Add 64-bit endianness-converting reads and writes.Lasse Collin1-6/+3
Also update the comment in liblzma's memcmplen.h. Thanks to Michał Górny for the original patch for the reads.
2022-09-28liblzma: Add dest and src NULL checks to lzma_index_cat.Jia Tan1-0/+3
The documentation states LZMA_PROG_ERROR can be returned from lzma_index_cat. Previously, lzma_index_cat could not return LZMA_PROG_ERROR. Now, the validation is similar to lzma_index_append, which does a NULL check on the index parameter.
2022-09-28liblzma: Fix copying of check type statistics in lzma_index_cat().Jia Tan1-1/+6
The check type of the last Stream in dest was never copied to dest->checks (the code tried to copy it but it was done too late). This meant that the value returned by lzma_index_checks() would only include the check type of the last Stream when multiple lzma_indexes had been concatenated. In xz --list this meant that the summary would only list the check type of the last Stream, so in this sense this was only a visual bug. However, it's possible that some applications use this information for purposes other than merely showing it to the users in an informational message. I'm not aware of such applications though and it's quite possible that such applications don't exist. Regular streamed decompression in xz or any other application doesn't use lzma_index_cat() and so this bug cannot affect them.
2022-09-28liblzma: Stream decoder: Fix restarting after LZMA_MEMLIMIT_ERROR.Lasse Collin1-3/+13
If lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible to use lzma_memlimit_set() to increase the limit and continue decoding. This was supposed to work from the beginning but there was a bug. With other decoders (.lzma or threaded .xz) this already worked correctly.
2022-09-28liblzma: Stream decoder: Fix comments.Lasse Collin1-7/+5
2022-09-19liblzma: Add experimental ARM64 BCJ filter with a temporary Filter ID.Lasse Collin3-0/+28
That is, the Filter ID will be changed once the design is final. The current version will be removed. So files created with the tempoary Filter ID won't be supported in the future.
2022-09-09liblzma: lzma_filters_copy: Keep dest[] unmodified if an error occurs.Lasse Collin1-6/+12
lzma_stream_encoder() and lzma_stream_encoder_mt() always assumed this. Before this patch, failing lzma_filters_copy() could result in free(invalid_pointer) or invalid memory reads in stream_encoder.c or stream_encoder_mt.c. To trigger this, allocating memory for a filter options structure has to fail. These are tiny allocations so in practice they very rarely fail. Certain badness in the filter chain array could also make lzma_filters_copy() fail but both stream_encoder.c and stream_encoder_mt.c validate the filter chain before trying to copy it, so the crash cannot occur this way.
2022-09-08liblzma: lzma_index_append: Add missing integer overflow check.Jia Tan1-0/+4
The documentation in src/liblzma/api/lzma/index.h suggests that both the unpadded (compressed) size and the uncompressed size are checked for overflow, but only the unpadded size was checked. The uncompressed check is done first since that is more likely to occur than the unpadded or index field size overflows.
2022-09-08liblzma: Vaccinate against an ill patch from RHEL/CentOS 7.Lasse Collin5-0/+114
RHEL/CentOS 7 shipped with 5.1.2alpha, including the threaded encoder that is behind #ifdef LZMA_UNSTABLE in the API headers. In 5.1.2alpha these symbols are under XZ_5.1.2alpha in liblzma.map. API/ABI compatibility tracking isn't done between development releases so newer releases didn't have XZ_5.1.2alpha anymore. Later RHEL/CentOS 7 updated xz to 5.2.2 but they wanted to keep the exported symbols compatible with 5.1.2alpha. After checking the ABI changes it turned out that >= 5.2.0 ABI is backward compatible with the threaded encoder functions from 5.1.2alpha (but not vice versa as fixes and extensions to these functions were made between 5.1.2alpha and 5.2.0). In RHEL/CentOS 7, XZ Utils 5.2.2 was patched with xz-5.2.2-compat-libs.patch to modify liblzma.map: - XZ_5.1.2alpha was added with lzma_stream_encoder_mt and lzma_stream_encoder_mt_memusage. This matched XZ Utils 5.1.2alpha. - XZ_5.2 was replaced with XZ_5.2.2. It is clear that this was an error; the intention was to keep using XZ_5.2 (XZ_5.2.2 has never been used in XZ Utils). So XZ_5.2.2 lists all symbols that were listed under XZ_5.2 before the patch. lzma_stream_encoder_mt and _mt_memusage are included too so they are listed both here and under XZ_5.1.2alpha. The patch didn't add any __asm__(".symver ...") lines to the .c files. Thus the resulting liblzma.so exports the threaded encoder functions under XZ_5.1.2alpha only. Listing the two functions also under XZ_5.2.2 in liblzma.map has no effect without matching .symver lines. The lack of XZ_5.2 in RHEL/CentOS 7 means that binaries linked against unpatched XZ Utils 5.2.x won't run on RHEL/CentOS 7. This is unfortunate but this alone isn't too bad as the problem is contained within RHEL/CentOS 7 and doesn't affect users of other distributions. It could also be fixed internally in RHEL/CentOS 7. The second problem is more serious: In XZ Utils 5.2.2 the API headers don't have #ifdef LZMA_UNSTABLE for obvious reasons. This is true in RHEL/CentOS 7 version too. Thus now programs using new APIs can be compiled without an extra #define. However, the programs end up depending on symbol version XZ_5.1.2alpha (and possibly also XZ_5.2.2) instead of XZ_5.2 as they would with an unpatched XZ Utils 5.2.2. This means that such binaries won't run on other distributions shipping XZ Utils >= 5.2.0 as they don't provide XZ_5.1.2alpha or XZ_5.2.2; they only provide XZ_5.2 (and XZ_5.0). (This includes RHEL/CentOS 8 as the patch luckily isn't included there anymore with XZ Utils 5.2.4.) Binaries built by RHEL/CentOS 7 users get distributed and then people wonder why they don't run on some other distribution. Seems that people have found out about the patch and been copying it to some build scripts, seemingly curing the symptoms but actually spreading the illness further and outside RHEL/CentOS 7. The ill patch seems to be from late 2016 (RHEL 7.3) and in 2017 it had spread at least to EasyBuild. I heard about the events only recently. :-( This commit splits liblzma.map into two versions: one for GNU/Linux and another for other OSes that can use symbol versioning (FreeBSD, Solaris, maybe others). The Linux-specific file and the matching additions to .c files add full compatibility with binaries that have been built against a RHEL/CentOS-patched liblzma. Builds for OSes other than GNU/Linux won't get the vaccine as they should be immune to the problem (I really hope that no build script uses the RHEL/CentOS 7 patch outside GNU/Linux). The RHEL/CentOS compatibility symbols XZ_5.1.2alpha and XZ_5.2.2 are intentionally put *after* XZ_5.2 in liblzma_linux.map. This way if one forgets to #define HAVE_SYMBOL_VERSIONS_LINUX when building, the resulting liblzma.so.5 will have lzma_stream_encoder_mt@@XZ_5.2 since XZ_5.2 {...} is the first one that lists that function. Without HAVE_SYMBOL_VERSIONS_LINUX @XZ_5.1.2alpha and @XZ_5.2.2 will be missing but that's still a minor problem compared to only having lzma_stream_encoder_mt@@XZ_5.1.2alpha! The "local: *;" line was moved to XZ_5.0 so that it doesn't need to be moved around. It doesn't matter where it is put. Having two similar liblzma_*.map files is a bit silly as it is, at least for now, easily possible to generate the generic one from the Linux-specific file. But that adds extra steps and increases the risk of mistakes when supporting more than one build system. So I rather maintain two files in parallel and let validate_map.sh check that they are in sync when "make mydist" is run. This adds .symver lines for lzma_stream_encoder_mt@XZ_5.2.2 and lzma_stream_encoder_mt_memusage@XZ_5.2.2 even though these weren't exported by RHEL/CentOS 7 (only @@XZ_5.1.2alpha was for these two). I added these anyway because someone might misunderstand the RHEL/CentOS 7 patch and think that @XZ_5.2.2 (@@XZ_5.2.2) versions were exported too. At glance one could suggest using __typeof__ to copy the function prototypes when making aliases. However, this doesn't work trivially because __typeof__ won't copy attributes (lzma_nothrow, lzma_pure) and it won't change symbol visibility from hidden to default (done by LZMA_API()). Attributes could be copied with __copy__ attribute but that needs GCC 9 and a fallback method would be needed anyway. This uses __symver__ attribute with GCC >= 10 and __asm__(".symver ...") with everything else. The attribute method is required for LTO (-flto) support with GCC. Using -flto with GCC older than 10 is now broken on GNU/Linux and will not be fixed (can silently result in a broken liblzma build that has dangerously incorrect symbol versions). LTO builds with Clang seem to work with the traditional __asm__(".symver ...") method. Thanks to Boud Roukema for reporting the problem and discussing the details and testing the fix.
2022-08-18liblzma: Threaded decoder: Improve LZMA_FAIL_FAST when LZMA_FINISH is used.Lasse Collin1-0/+42
It will now return LZMA_DATA_ERROR (not LZMA_OK or LZMA_BUF_ERROR) if LZMA_FINISH is used and there isn't enough input to finish decoding the Block Header or the Block. The use of LZMA_DATA_ERROR is simpler and the less risky than LZMA_BUF_ERROR but this might be changed before 5.4.0.
2022-07-13liblzma: Add optional autodetection of LZMA end marker.Lasse Collin2-2/+2
Turns out that this is needed for .lzma files as the spec in LZMA SDK says that end marker may be present even if the size is stored in the header. Such files are rare but exist in the real world. The code in liblzma is so old that the spec didn't exist in LZMA SDK back then and I had understood that such files weren't possible (the lzma tool in LZMA SDK didn't create such files). This modifies the internal API so that LZMA decoder can be told if EOPM is allowed even when the uncompressed size is known. It's allowed with .lzma and not with other uses. Thanks to Karl Beldan for reporting the problem.
2022-05-23liblzma: Silence a warning.Lasse Collin1-1/+1
The actual initialization is done via mythread_sync and seems that GCC doesn't necessarily see that it gets initialized there.
2022-04-06liblzma: Threaded decoder: Improve setting of pending_error.Lasse Collin1-16/+35
It doesn't need to be done conditionally. The comments try to explain it.
2022-04-06liblzma: Add a new flag LZMA_FAIL_FAST for threaded decoder.Lasse Collin2-25/+32
In most cases if the input file is corrupt the application won't care about the uncompressed content at all. With this new flag the threaded decoder will return an error as soon as any thread has detected an error; it won't wait to copy out the data before the location of the error. I don't plan to use this in xz to keep the behavior consistent between single-threaded and multi-threaded modes.
2022-04-05liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.Lasse Collin1-2/+24
This makes the behavior consistent with the single-threaded decoder when handling truncated .xz files. Thanks to Jia Tan for finding this issue.
2022-04-02liblzma: Threaded decoder: Support zpipe.c-style decoding loop.Lasse Collin1-10/+67
This makes it possible to call lzma_code() in a loop that only reads new input when lzma_code() didn't fill the output buffer completely. That isn't the calling style suggested by the liblzma example program 02_decompress.c so perhaps the usefulness of this feature is limited. Also, it is possible to write such a loop so that it works with the single-threaded decoder but not with the threaded decoder even after this commit, or so that it works only if lzma_mt.timeout = 0. The zlib tutorial <https://zlib.net/zlib_how.html> is a well-known example of a loop where more input is read only when output isn't full. Porting this as is to liblzma would work with the single-threaded decoder (if LZMA_CONCATENATED isn't used) but it wouldn't work with threaded decoder even after this commit because the loop assumes that no more output is possible when it cannot read more input ("if (strm.avail_in == 0) break;"). This cannot be fixed at liblzma side; the loop has to be modified at least a little. I'm adding this in any case because the actual code is simple and short and should have no harmful side-effects in other situations.
2022-03-26liblzma: Fix a deadlock in threaded decoder.Lasse Collin1-19/+52
If a worker thread has consumed all input so far and it's waiting on thr->cond and then the main thread enables partial update for that thread, the code used to deadlock. This commit allows one dummy decoding pass to occur in this situation which then also does the partial update. As part of the fix, this moves thr->progress_* updates to avoid the second thr->mutex locking. Thanks to Jia Tan for finding, debugging, and reporting the bug.
2022-03-23liblzma: Threaded decoder: Don't stop threads on LZMA_TIMED_OUT.Lasse Collin1-1/+1
LZMA_TIMED_OUT is not an error and thus stopping threads on LZMA_TIMED_OUT breaks the decoder badly. Thanks to Jia Tan for finding the bug and for the patch.
2022-03-07liblzma: Add threaded .xz decompressor.Lasse Collin3-0/+1823
I realize that this is about a decade late. Big thanks to Sebastian Andrzej Siewior for the original patch. I made a bunch of smaller changes but after a while quite a few things got rewritten. So any bugs in the commit were created by me.
2022-03-06liblzma: Add new output queue (lzma_outq) features.Lasse Collin2-8/+79
Add lzma_outq_clear_cache2() which may leave one buffer allocated in the cache. Add lzma_outq_outbuf_memusage() to get the memory needed for a single lzma_outbuf. This is now used internally in outqueue.c too. Track both the total amount of memory allocated and the amount of memory that is in active use (not in cache). In lzma_outbuf, allow storing the current input position that matches the current output position. This way the main thread can notice when no more output is possible without first providing more input. Allow specifying return code for lzma_outq_read() in a finished lzma_outbuf.
2022-03-06liblzma: Index hash: Change return value type of hash_append() to void.Lasse Collin1-6/+5
2022-02-22liblzma: Check the return value of lzma_index_append() in threaded encoder.Lasse Collin1-2/+5
If lzma_index_append() failed (most likely memory allocation failure) it could have gone unnoticed and the resulting .xz file would have an incorrect Index. Decompressing such a file would produce the correct uncompressed data but then an error would occur when verifying the Index field.
2022-02-20liblzma: Make Block decoder catch certain types of errors better.Lasse Collin1-25/+54
Now it limits the input and output buffer sizes that are passed to a raw decoder. This way there's no need to check if the sizes can grow too big or overflow when updating Compressed Size and Uncompressed Size counts. This also means that a corrupt file cannot cause the raw decoder to process useless extra input or output that would exceed the size info in Block Header (and thus cause LZMA_DATA_ERROR anyway). More importantly, now the size information is verified more carefully in case raw decoder returns LZMA_OK. This doesn't really matter with the current single-threaded .xz decoder as the errors would be detected slightly later anyway. But this helps avoiding corner cases in the upcoming threaded decompressor, and it might help other Block decoder uses outside liblzma too. The test files bad-1-lzma2-{9,10,11}.xz test these conditions. With the single-threaded .xz decoder the only difference is that LZMA_DATA_ERROR is detected in a difference place now.
2021-09-05liblzma: Rename EROFS LZMA to MicroLZMA.Lasse Collin3-33/+33
It still exists primarily for EROFS but MicroLZMA is a more generic name (that hopefully doesn't clash with something that already exists).
2021-01-24liblzma: Fix a wrong comment in stream_encoder_mt.c.Lasse Collin1-3/+7
2021-01-17liblzma: In EROFS LZMA decoder, verify that comp_size matches at the end.Lasse Collin1-1/+6
When the uncompressed size is known to be exact, after decompressing the stream exactly comp_size bytes of input must have been consumed. This is a minor improvement to error detection.
2021-01-17liblzma: Make EROFS LZMA decoder work when exact uncomp_size isn't known.Lasse Collin1-7/+73
The caller must still not specify an uncompressed size bigger than the actual uncompressed size. As a downside, this now needs the exact compressed size.
2021-01-14liblzma: Add EROFS LZMA encoder and decoder.Lasse Collin3-0/+289
Right now this is just a planned extra-compact format for use in the EROFS file system in Linux. At this point it's possible that the format will either change or be abandoned and removed completely. The special thing about the encoder is that it uses the output-size-limited encoding added in the previous commit. EROFS uses fixed-sized blocks (e.g. 4 KiB) to hold compressed data so the compressors must be able to create valid streams that fill the given block size.
2021-01-14liblzma: Add rough support for output-size-limited encoding in LZMA1.Lasse Collin1-0/+11
With this it is possible to encode LZMA1 data without EOPM so that the encoder will encode as much input as it can without exceeding the specified output size limit. The resulting LZMA1 stream will be a normal LZMA1 stream without EOPM. The actual uncompressed size will be available to the caller via the uncomp_size pointer. One missing thing is that the LZMA layer doesn't inform the LZ layer when the encoding is finished and thus the LZ may read more input when it won't be used. However, this doesn't matter if encoding is done with a single call (which is the planned use case for now). For proper multi-call encoding this should be improved. This commit only adds the functionality for internal use. Nothing uses it yet.
2021-01-09liblzma: Make lzma_outq usable for threaded decompression too.Lasse Collin3-157/+301
Before this commit all output queue buffers were allocated as a single big allocation. Now each buffer is allocated separately when needed. Used buffers are cached to avoid reallocation overhead but the cache will keep only one buffer size at a time. This should make things work OK in the decompression where most of the time the buffer sizes will be the same but with some less common files the buffer sizes may vary. While this should work fine, it's still a bit preliminary and may even get reverted if it turns out to be useless for decompression.
2020-03-23Typo fixes from fossies.org.Lasse Collin1-1/+1
https://fossies.org/linux/misc/xz-5.2.5.tar.xz/codespell.html
2019-12-31Rename unaligned_read32ne to read32ne, and similarly for the others.Lasse Collin6-16/+13
2019-06-24liblzma: Add a comment.Lasse Collin1-1/+1
2019-06-24Add LZMA_RET_INTERNAL1..8 to lzma_ret and use one for LZMA_TIMED_OUT.Lasse Collin2-6/+3
LZMA_TIMED_OUT is *internally* used as a value for lzma_ret enumeration. Previously it was #defined to 32 and cast to lzma_ret. That way it wasn't visible in the public API, but this was hackish. Now the public API has eight LZMA_RET_INTERNALx members and LZMA_TIMED_OUT is #defined to LZMA_RET_INTERNAL1. This way the code is cleaner overall although the public API has a few extra mysterious enum members.
2019-06-24liblzma: Remove incorrect uses of lzma_attribute((__unused__)).Lasse Collin2-4/+2
Caught by clang -Wused-but-marked-unused.
2019-06-23liblzma: Fix warnings from -Wsign-conversion.Lasse Collin1-1/+1
Also, more parentheses were added to the literal_subcoder macro in lzma_comon.h (better style but no functional change in the current usage).
2019-06-03liblzma: Fix comments.Lasse Collin6-7/+7
Thanks to Bruce Stark.
2019-06-01liblzma: memcmplen: Use ctz32() from tuklib_integer.h.Lasse Collin1-9/+1
The same compiler-specific #ifdefs are already in tuklib_integer.h
2019-06-01liblzma: Use unaligned_readXXne functions instead of type punning.Lasse Collin1-6/+6
Now gcc -fsanitize=undefined should be clean. Thanks to Jeffrey Walton.
2019-05-13liblzma: Avoid memcpy(NULL, foo, 0) because it is undefined behavior.Lasse Collin1-1/+5
I should have always known this but I didn't. Here is an example as a reminder to myself: int mycopy(void *dest, void *src, size_t n) { memcpy(dest, src, n); return dest == NULL; } In the example, a compiler may assume that dest != NULL because passing NULL to memcpy() would be undefined behavior. Testing with GCC 8.2.1, mycopy(NULL, NULL, 0) returns 1 with -O0 and -O1. With -O2 the return value is 0 because the compiler infers that dest cannot be NULL because it was already used with memcpy() and thus the test for NULL gets optimized out. In liblzma, if a null-pointer was passed to memcpy(), there were no checks for NULL *after* the memcpy() call, so I cautiously suspect that it shouldn't have caused bad behavior in practice, but it's hard to be sure, and the problematic cases had to be fixed anyway. Thanks to Jeffrey Walton.
2019-05-11spellingAntoine Cœur5-7/+7
2018-10-26liblzma: Don't verify header CRC32s if building for fuzz testing.Lasse Collin4-5/+20
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is #defined when liblzma is being built for fuzz testing. Most fuzzed inputs would normally get rejected because of incorrect CRC32 and the actual header decoding code wouldn't get fuzzed. Disabling CRC32 checks avoids this problem. The fuzzer program must still use LZMA_IGNORE_CHECK flag to disable verification of integrity checks of uncompressed data.
2018-07-27liblzma: Remove an always-true condition from lzma_index_cat().Lasse Collin1-2/+2
This should help static analysis tools to see that newg isn't leaked. Thanks to Pavel Raiskup.
2017-04-24liblzma: Add lzma_file_info_decoder().Lasse Collin2-0/+856
2017-04-21liblzma: Rename LZMA_SEEK to LZMA_SEEK_NEEDED and seek_in to seek_pos.Lasse Collin1-1/+1
2017-03-30liblzma: Make lzma_index_decoder_init() visible to other liblzma funcs.Lasse Collin3-5/+30
This is to allow other functions to use it without going via the public API (lzma_index_decoder()).
2017-03-30liblzma: Add generic support for input seeking (LZMA_SEEK).Lasse Collin1-1/+11
Also mention LZMA_SEEK in xz/message.c to silence a warning.
2017-03-30liblzma: Fix lzma_memlimit_set(strm, 0).Lasse Collin2-2/+7
The 0 got treated specially in a buggy way and as a result the function did nothing. The API doc said that 0 was supposed to return LZMA_PROG_ERROR but it didn't. Now 0 is treated as if 1 had been specified. This is done because 0 is already used to indicate an error from lzma_memlimit_get() and lzma_memusage(). In addition, lzma_memlimit_set() no longer checks that the new limit is at least LZMA_MEMUSAGE_BASE. It's counter-productive for the Index decoder and was actually needed only by the auto decoder. Auto decoder has now been modified to check for LZMA_MEMUSAGE_BASE.
2017-03-30liblzma: Similar memlimit fix for stream_, alone_, and auto_decoder.Lasse Collin3-12/+3
2017-03-30liblzma: Fix handling of memlimit == 0 in lzma_index_decoder().Lasse Collin1-2/+2
It returned LZMA_PROG_ERROR, which was done to avoid zero as the limit (because it's a special value elsewhere), but using LZMA_PROG_ERROR is simply inconvenient and can cause bugs. The fix/workaround is to treat 0 as if it were 1 byte. It's effectively the same thing. The only weird consequence is that then lzma_memlimit_get() will return 1 even when 0 was specified as the limit. This fixes a very rare corner case in xz --list where a specific memory usage limit and a multi-stream file could print the error message "Internal error (bug)" instead of saying that the memory usage limit is too low.
2016-11-21liblzma: Avoid multiple definitions of lzma_coder structures.Lasse Collin11-212/+279
Only one definition was visible in a translation unit. It avoided a few casts and temp variables but seems that this hack doesn't work with link-time optimizations in compilers as it's not C99/C11 compliant. Fixes: http://www.mail-archive.com/xz-devel@tukaani.org/msg00279.html
2015-11-03Build: Don't omit lzma_cputhreads() unless using --disable-threads.Lasse Collin1-1/+4
Previously it was omitted if encoders were disabled with --disable-encoders. It didn't make sense and it also broke the build.
2015-10-12liblzma: Fix lzma_index_dup() for empty Streams.Lasse Collin1-5/+6
Stream Flags and Stream Padding weren't copied from empty Streams.
2015-10-12liblzma: Add a note to index.c for those using static analyzers.Lasse Collin1-0/+3
2015-10-12liblzma: Fix a memory leak in error path of lzma_index_dup().Lasse Collin1-9/+9
lzma_index_dup() calls index_dup_stream() which, in case of an error, calls index_stream_end() to free memory allocated by index_stream_init(). However, it illogically didn't actually free the memory. To make it logical, the tree handling code was modified a bit in addition to changing index_stream_end(). Thanks to Evan Nemerson for the bug report.
2015-03-07liblzma: Fix a warning in index.c.Lasse Collin1-1/+3
2015-01-26liblzma: Set LZMA_MEMCMPLEN_EXTRA depending on the compare method.Lasse Collin1-5/+10
2014-11-26Remove LZMA_UNSTABLE macro.Lasse Collin1-2/+0
2014-11-25liblzma: Verify the filter chain in threaded encoder initialization.Lasse Collin1-3/+6
This way an invalid filter chain is detected at the Stream encoder initialization instead of delaying it to the first call to lzma_code() which triggers the initialization of the actual filter encoder(s).
2014-11-10liblzma: Fix lzma_mt.preset in lzma_stream_encoder_mt_memusage().Lasse Collin1-2/+1
It read the filter chain from a wrong variable. This is a similar bug that was fixed in 9494fb6d0ff41c585326f00aa8f7fe58f8106a5e.
2014-08-05liblzma: Add support for LZMA_IGNORE_CHECK.Lasse Collin2-2/+13
2014-08-05liblzma: Add support for lzma_block.ignore_check.Lasse Collin6-10/+28
Note that this slightly changes how lzma_block_header_decode() has been documented. Earlier it said that the .version is set to the lowest required value, but now it says that the .version field is kept unchanged if possible. In practice this doesn't affect any old code, because before this commit the only possible .version was 0.
2014-07-25liblzma: Add lzma_memcmplen() for fast memory comparison.Lasse Collin2-0/+171
This commit just adds the function. Its uses will be in separate commits. This hasn't been tested much yet and it's perhaps a bit early to commit it but if there are bugs they should get found quite quickly. Thanks to Jun I Jin from Intel for help and for pointing out that string comparison needs to be optimized in liblzma.
2014-06-18liblzma: Add lzma_cputhreads().Lasse Collin2-0/+23
2014-05-25liblzma: Add the internal function lzma_alloc_zero().Lasse Collin2-0/+27
2014-01-29liblzma: Fix lzma_mt.preset not working with lzma_stream_encoder_mt().Lasse Collin1-2/+2
It read the filter chain from a wrong variable.
2013-10-02liblzma: Support LZMA_FULL_FLUSH and _BARRIER in threaded encoder.Lasse Collin1-16/+39
Now --block-list=SIZES works with in the threaded mode too, although the performance is still bad due to the use of LZMA_FULL_FLUSH instead of the new LZMA_FULL_BARRIER.
2013-10-02liblzma: Add LZMA_FULL_BARRIER support to single-threaded encoder.Lasse Collin3-4/+24
In the single-threaded encoder LZMA_FULL_BARRIER is simply an alias for LZMA_FULL_FLUSH.
2013-09-17liblzma: Add block_buffer_encoder.h into Makefile.inc.Lasse Collin1-0/+1
This should have been in b465da5988dd59ad98fda10c2e4ea13d0b9c73bc.
2013-09-17Add native threading support on Windows.Lasse Collin1-47/+36
Now liblzma only uses "mythread" functions and types which are defined in mythread.h matching the desired threading method. Before Windows Vista, there is no direct equivalent to pthread condition variables. Since this package doesn't use pthread_cond_broadcast(), pre-Vista threading can still be kept quite simple. The pre-Vista code doesn't use anything that wasn't already available in Windows 95, so the binaries should run even on Windows 95 if someone happens to care.
2013-06-23liblzma: Avoid a warning about a shadowed variable.Lasse Collin1-2/+2
On Mac OS X wait() is declared in <sys/wait.h> that we include one way or other so don't use "wait" as a variable name. Thanks to Christian Kujau.
2013-03-23liblzma: Be less picky in lzma_alone_decoder().Lasse Collin3-11/+18
To avoid false positives when detecting .lzma files, rare values in dictionary size and uncompressed size fields were rejected. They will still be rejected if .lzma files are decoded with lzma_auto_decoder(), but when using lzma_alone_decoder() directly, such files will now be accepted. Hopefully this is an OK compromise. This doesn't affect xz because xz still has its own file format detection code. This does affect lzmadec though. So after this commit lzmadec will accept files that xz or xz-emulating-lzma doesn't. NOTE: lzma_alone_decoder() still won't decode all .lzma files because liblzma's LZMA decoder doesn't support lc + lp > 4. Reported here: http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/7068827
2013-03-23liblzma: Use lzma_block_buffer_bound64() in threaded encoder.Lasse Collin1-16/+50
Now it uses lzma_block_uncomp_encode() if the data doesn't fit into the space calculated by lzma_block_buffer_bound64().
2013-03-23liblzma: Fix another deadlock in the threaded encoder.Lasse Collin1-3/+6
This race condition could cause a deadlock if lzma_end() was called before finishing the encoding. This can happen with xz with debugging enabled (non-debugging version doesn't call lzma_end() before exiting).
2013-03-23liblzma: Add lzma_block_uncomp_encode().Lasse Collin2-31/+87
This also adds a new internal function lzma_block_buffer_bound64() which is similar to lzma_block_buffer_bound() but uses uint64_t instead of size_t.
2012-12-14Make the progress indicator smooth in threaded mode.Lasse Collin3-5/+94
This adds lzma_get_progress() to liblzma and takes advantage of it in xz. lzma_get_progress() collects progress information from the thread-specific structures so that fairly accurate progress information is available to applications. Adding a new function seemed to be a better way than making the information directly available in lzma_stream (like total_in and total_out are) because collecting the information requires locking mutexes. It's waste of time to do it more often than the up to date information is actually needed by an application.
2012-07-17liblzma: Make the use of lzma_allocator const-correct.Lasse Collin36-105/+116
There is a tiny risk of causing breakage: If an application assigns lzma_stream.allocator to a non-const pointer, such code won't compile anymore. I don't know why anyone would do such a thing though, so in practice this shouldn't cause trouble. Thanks to Jan Kratochvil for the patch.
2011-10-23liblzma: Fix invalid free() in the threaded encoder.Lasse Collin1-0/+4
It was triggered if initialization failed e.g. due to running out of memory. Thanks to Arkadiusz Miskiewicz.
2011-10-23liblzma: Fix a deadlock in the threaded encoder.Lasse Collin1-1/+3
It was triggered when reinitializing the encoder, e.g. when encoding two files.
2011-05-27liblzma: Handle allocation failures correctly in lzma_index_init().Lasse Collin1-2/+5
Thanks to Jim Meyering.
2011-05-17Add underscores to attributes (__attribute((__foo__))).Lasse Collin7-14/+16
2011-04-12Put the unstable APIs behind #ifdef LZMA_UNSTABLE.Lasse Collin1-0/+2
This way people hopefully won't complain if these APIs change and break code that used an older API.
2011-04-12Remove doubled words from documentation and comments.Lasse Collin1-1/+1
Spot candidates by running these commands: git ls-files |xargs perl -0777 -n \ -e 'while (/\b(then?|[iao]n|i[fst]|but|f?or|at|and|[dt]o)\s+\1\b/gims)' \ -e '{$n=($` =~ tr/\n/\n/ + 1); ($v=$&)=~s/\n/\\n/g; print "$ARGV:$n:$v\n"}' Thanks to Jim Meyering for the original patch.
2011-04-11liblzma: Add lzma_stream_encoder_mt() for threaded compression.Lasse Collin6-1/+1375
This is the simplest method to do threading, which splits the uncompressed data into blocks and compresses them independently from each other. There's room for improvement especially to reduce the memory usage, but nevertheless, this is a good start.
2011-04-11liblzma: Use memzero() to initialize supported_actions[].Lasse Collin1-4/+2
This is cleaner and makes it simpler to add new members to lzma_action enumeration.
2011-04-11liblzma: Add an internal function lzma_mt_block_size().Lasse Collin2-26/+24
This is based lzma_chunk_size() that was included in some development version of liblzma.
2011-04-11liblzma: Don't create an empty Block in lzma_stream_buffer_encode().Lasse Collin1-7/+13
Empty Block was created if the input buffer was empty. Empty Block wastes a few bytes of space, but more importantly it triggers a bug in XZ Utils 5.0.1 and older when trying to decompress such a file. 5.0.1 and older consider such files to be corrupt. I thought that no encoder creates empty Blocks when releasing 5.0.2 but I was wrong.
2011-04-11liblzma: Validate encoder arguments better.Lasse Collin3-6/+20
The biggest problem was that the integrity check type wasn't validated, and e.g. lzma_easy_buffer_encode() would create a corrupt .xz Stream if given an unsupported Check ID. Luckily applications don't usually try to use an unsupport Check ID, so this bug is unlikely to cause many real-world problems.
2011-04-11liblzma: Make lzma_stream_encoder_init() static (second try).Lasse Collin4-32/+6
It's an internal function and it's not needed by anything outside stream_encoder.c.
2011-04-11Revert "liblzma: Make lzma_stream_encoder_init() static."Lasse Collin3-4/+29
This reverts commit 352ac82db5d3f64585c07b39e4759388dec0e4d7. I don't know what I was thinking.
2011-04-10liblzma: Make lzma_stream_encoder_init() static.Lasse Collin3-29/+4
It's an internal function and it's not needed by anything outside stream_encoder.c.
2011-04-09liblzma: Add missing #ifdefs to filter_common.c.Lasse Collin1-3/+3
Passing --disable-decoders to configure broke a few encoders due to missing #ifdefs in filter_common.c. Thanks to Jason Gorski for the patch.
2011-04-02liblzma: Fix a memory leak in stream_encoder.c.Lasse Collin1-1/+1
It leaks old filter options structures (hundred bytes or so) every time the lzma_stream is reinitialized. With the xz tool, this happens when compressing multiple files.
2011-01-18Add alloc_size and malloc attributes to a few functions.Lasse Collin1-1/+1
Thanks to Cristian Rodríguez for the original patch.
2010-10-23liblzma: Make lzma_code() check the reserved members in lzma_stream.Lasse Collin1-0/+14
If any of the reserved members in lzma_stream are non-zero or non-NULL, LZMA_OPTIONS_ERROR is returned. It is possible that a new feature in the future is indicated by just setting a reserved member to some other value, so the old liblzma version need to catch it as an unsupported feature.
2010-10-23liblzma: Use 512 as INDEX_GROUP_SIZE.Lasse Collin1-1/+1
This lets compiler use shifting instead of 64-bit division.
2010-10-21liblzma: Update the comments in the API headers.Lasse Collin2-0/+2
Adding support for LZMA_FINISH for Index encoding and decoding needed tiny additions to the relevant .c files too.
2010-10-19Clean up a few FIXMEs and TODOs.Lasse Collin2-2/+4
lzma_chunk_size() was commented out because it is currently useless.
2010-05-27Fix lzma_block_compressed_size().Lasse Collin1-3/+3
2010-05-27Remove unused chunk_size.c.Lasse Collin1-67/+0
Thanks to Jonathan Nieder for the reminder.
2010-05-26Rename MIN() and MAX() to my_min() and my_max().Lasse Collin3-3/+3
This should avoid some minor portability issues.
2010-05-26Remove the Subblock filter code for now.Lasse Collin4-43/+0
The spec isn't finished and the code didn't compile anymore. It won't be included in XZ Utils 5.0.0. It's easy to get it back once the spec is done.
2010-03-06Fix missing initialization in lzma_strm_init().Lasse Collin1-0/+1
With bad luck, lzma_code() could return LZMA_BUF_ERROR when it shouldn't. This has been here since the early days of liblzma. It got triggered by the modifications made to the xz tool in commit 18c10c30d2833f394cd7bce0e6a821044b15832f but only when decompressing .lzma files. Somehow I managed to miss testing that with Valgrind earlier. This fixes <http://bugs.gentoo.org/show_bug.cgi?id=305591>. Thanks to Rafał Mużyło for helping to debug it on IRC.
2010-02-12Collection of language fixes to comments and docs.Lasse Collin9-14/+14
Thanks to Jonathan Nieder.
2010-02-07Subtle change to liblzma Block handling API.Lasse Collin1-0/+3
lzma_block.version has to be initialized even for lzma_block_header_decode(). This way a future version of liblzma won't allocate memory in a way that an old application doesn't know how to free it. The subtlety of this change is that all current apps using lzma_block_header_decode() will keep working for now, because the only possible version value is zero, and lzma_block_header_decode() unconditionally sets the version to zero even now. Unless fixed, these apps will break in the future if a new version of the Block options is ever needed.
2010-01-01Fix _memconfig() functions.Lasse Collin3-12/+20
This affects lzma_memusage() and lzma_memlimit_get().
2009-12-31Revised the Index handling code.Lasse Collin7-578/+1040
This breaks API and ABI but most apps are not affected since most apps don't use this part of the API. You will get a compile error if you are using anything that got broken. Summary of changes: - Ability to store Stream Flags, which are needed for random-access reading in multi-Stream files. - Separate function to set size of Stream Padding. - Iterator structure makes it possible to read the same lzma_index from multiple threads at the same time. - A lot faster code to locate Blocks. - Removed lzma_index_equal() without adding anything to replace it. I don't know what it should do exactly with the new features and what actually needs this function in the first place other than test_index.c, which now has its own code to compare lzma_indexes.
2009-11-25Fix bugs in lzma_index_read() and lzma_index_cat().Lasse Collin1-11/+19
lzma_index_read() didn't skip over Stream Padding if it was the first record in the Index. lzma_index_cat() didn't combine small Indexes correctly. The test suite was updated to check for these bugs. These bugs didn't affect the xz command line tool or most users of liblzma in any way.
2009-11-25Index decoder fixes.Lasse Collin1-6/+20
The Index decoder code didn't perfectly match the API docs, which said that *i will be set to point to the decoded Index only after decoding has succeeded. The docs were a bit unclear too. Now the decoder will initially set *i to NULL. *i will be set to point to the decoded Index once decoding has succeeded. This simplifies applications too, since it avoids dangling pointers.
2009-11-22Make fastpos.h use tuklib_integer.h instead of bsr.hLasse Collin2-61/+0
when --enable-small has been specified.
2009-11-15Add lzma_physmem().Lasse Collin2-0/+26
I had hoped to keep liblzma as purely a compression library as possible (e.g. file I/O will go into a different library), but it seems that applications linking agaisnt liblzma need some way to determine the memory usage limit, and knowing the amount of RAM is one reasonable way to help making such decisions. Thanks to Jonathan Nieder for the original patch.
2009-11-14Fix a design error in liblzma API.Lasse Collin8-71/+156
Originally the idea was that using LZMA_FULL_FLUSH with Stream encoder would read the filter chain from the same array that was used to intialize the Stream encoder. Since most apps wouldn't use LZMA_FULL_FLUSH, most apps wouldn't need to keep the filter chain available after initializing the Stream encoder. However, due to my mistake, it actually required keeping the array always available. Since setting the new filter chain via the array used at initialization time is not a nice way to do it for a couple of reasons, this commit ditches it and introduces lzma_filters_update(). This new function replaces also the "persistent" flag used by LZMA2 (and to-be-designed Subblock filter), which was also an ugly thing to do. Thanks to Alexey Tourbin for reminding me about the problem that Stream encoder used to require keeping the filter chain allocated.
2009-10-17Fix wrong function name in the previous commit.Lasse Collin1-1/+1
It was meant to be lzma_filters_copy(), not lzma_filters_dup().
2009-10-17Add lzma_filters_copy().Lasse Collin1-0/+82
This will be needed internally by liblzma once I fix a design mistake in the encoder API. This function may be useful to applications too so it's good to export it.
2009-10-04Use a tuklib module for integer handling.Lasse Collin6-10/+10
This replaces bswap.h and integer.h. The tuklib module uses <byteswap.h> on GNU, <sys/endian.h> on *BSDs and <sys/byteorder.h> on Solaris, which may contain optimized code like inline assembly.
2009-09-15Fix incorrect use of "restrict".Lasse Collin2-2/+2
2009-09-12A few grammar fixes.Lasse Collin2-2/+2
Thanks to Christian Weisgerber for pointing out some of these.
2009-09-11Fix a couple of warnings.Lasse Collin1-1/+1
2009-06-30Build system fixesLasse Collin3-84/+77
Don't use libtool convenience libraries to avoid recently discovered long-standing subtle but somewhat severe bugs in libtool (at least 1.5.22 and 2.2.6 are affected). It was found when porting XZ Utils to Windows <http://lists.gnu.org/archive/html/libtool/2009-06/msg00070.html> but the problem is significant also e.g. on GNU/Linux. Unless --disable-shared is passed to configure, static library built from a set of convenience libraries will contain PIC objects. That is, while libtool builds non-PIC objects too, only PIC objects will be used from the convenience libraries. On 32-bit x86 (tested on mobile XP2400+), using PIC instead of non-PIC makes the decompressor 10 % slower with the default CFLAGS. So while xz was linked against static liblzma by default, it got the slower PIC objects unless --disable-shared was used. I tend develop and benchmark with --disable-shared due to faster build time, so I hadn't noticed the problem in benchmarks earlier. This commit also adds support for building Windows resources into liblzma and executables.
2009-06-26Basic support for building with Cygwin and MinGW usingLasse Collin1-1/+1
the Autotools based build system. It's not good yet, more fixes will follow.
2009-06-26Fix @variables@ to $(variables) in Makefile.am files.Lasse Collin1-8/+8
Fix the ordering of libgnu.a and LTLIBINTL on the linker command line and added missing LTLIBINTL to tests/Makefile.am.
2009-05-26Make the raw value of the Check field available to applicationsLasse Collin3-31/+25
via lzma_block structure. This changes ABI but not doesn't break API.
2009-04-28Fixed a crash in liblzma.Lasse Collin11-13/+13
liblzma tries to avoid useless free()/malloc() pairs in initialization when multiple files are handled using the same lzma_stream. This didn't work with filter chains due to comparison of wrong pointers in lzma_next_coder_init(), making liblzma think that no memory reallocation is needed even when it actually is. Easy way to trigger this bug is to decompress two files with a single xz command. The first file should have e.g. x86+LZMA2 as the filter chain, and the second file just LZMA2.
2009-04-13Put the interesting parts of XZ Utils into the public domain.Lasse Collin53-524/+159
Some minor documentation cleanups were made at the same time.
2009-02-18Added files missing from the previous commit.Lasse Collin1-1/+6
2009-02-17Added lzma_easy_buffer_encode(). Splitted easy.c into smallLasse Collin6-47/+175
pieces to avoid unneeded dependencies making statically linked applications bigger than needed.
2009-02-13Changed how the version number is specified in various places.Lasse Collin1-1/+1
Now configure.ac will get the version number directly from src/liblzma/api/lzma/version.h. The intent is to reduce the number of places where the version number is duplicated. In future, support for displaying Git commit ID may be added too.
2009-02-07Introduced LZMA_API_STATIC macro, which the applicationsLasse Collin1-1/+1
need to #define when linking against static liblzma on platforms like Windows. Most developers don't need to care about LZMA_API_STATIC at all.
2009-02-02Modify LZMA_API macro so that it works on Windows withLasse Collin34-80/+80
other compilers than MinGW. This may hurt readability of the API headers slightly, but I don't know any better way to do this.
2009-02-01Fix a bug in lzma_block_buffer_decode(), although thisLasse Collin1-4/+4
function should be rewritten anyway.
2009-02-01Added initial version of raw buffer-to-buffer codingLasse Collin3-0/+157
functions, and cleaned up filter.h API header a little. May be very buggy, not tested yet.
2009-02-01Use __cdecl also for function pointers in liblzma API whenLasse Collin1-11/+7
on Windows.
2009-01-31Add LZMA_API to liblzma API headers. It's useful at leastLasse Collin1-2/+11
on Windows. sysdefs.h no longer #includes lzma.h, so lzma.h has to be #included separately where needed.
2009-01-28Fix uninitialized variables in alone_decoder.c. This bug wasLasse Collin1-0/+2
triggered by the previous commit, since these variables were not used by anything before support for a preset dictionary.
2009-01-26Add lzma_block_buffer_decode().Lasse Collin2-0/+88
2009-01-26Add more sanity checks to lzma_stream_buffer_decode().Lasse Collin1-0/+7
2009-01-26Avoid hardcoded constant in easy.c.Lasse Collin1-1/+1
2009-01-26Tiny bit better sanity check in block_util.cLasse Collin1-1/+1
2009-01-25Fix a dumb bug in Block decoder, which made it returnLasse Collin1-3/+3
LZMA_DATA_ERROR with valid data. The bug was added in e114502b2bc371e4a45449832cb69be036360722.
2009-01-23Added lzma_stream_buffer_decode() and made minor cleanups.Lasse Collin2-0/+92
2009-01-22Fix a comment.Lasse Collin1-1/+1
2009-01-20Add some single-call buffer-to-buffer coding functions.Lasse Collin5-20/+567
2009-01-20Block encoder cleanupsLasse Collin2-21/+32
2009-01-20Use LZMA_PROG_ERROR in lzma_code() as documented in base.h.Lasse Collin1-16/+8
2009-01-19Fix handling of non-fatal errors in lzma_code().Lasse Collin1-1/+8
2009-01-17Beta was supposed to be API stable but I had forgot to renameLasse Collin6-9/+9
lzma_memlimit_encoder and lzma_memlimit_decoder to lzma_raw_encoder_memlimit and lzma_raw_decoder_memlimit. :-( Now it is fixed. Hopefully it doesn't cause too much trouble to those who already thought API is stable.
2008-12-31Remove lzma_init() and other init functions from liblzma API.Lasse Collin5-113/+1
Half of developers were already forgetting to use these functions, which could have caused total breakage in some future liblzma version or even now if --enable-small was used. Now liblzma uses pthread_once() to do the initializations unless it has been built with --disable-threads which make these initializations thread-unsafe. When --enable-small isn't used, liblzma currently gets needlessly linked against libpthread (on systems that have it). While it is stupid for now, liblzma will need threads in future anyway, so this stupidity will be temporary only. When --enable-small is used, different code CRC32 and CRC64 is now used than without --enable-small. This made the resulting binary slightly smaller, but the main reason was to clean it up and to handle the lack of lzma_init_check(). The pkg-config file lzma.pc was renamed to liblzma.pc. I'm not sure if it works correctly and portably for static linking (Libs.private includes -pthread or other operating system specific flags). Hopefully someone complains if it is bad. lzma_rc_prices[] is now included as a precomputed array even with --enable-small. It's just 128 bytes now that it uses uint8_t instead of uint32_t. Smaller array seemed to be at least as fast as the more bloated uint32_t array on x86; hopefully it's not bad on other architectures.
2008-12-27Bunch of liblzma tweaks, including some API changes.Lasse Collin17-163/+150
The API and ABI should now be very close to stable, although the code behind it isn't yet.
2008-12-18Updated Makefile.am that was missing from the previous commit.Lasse Collin1-1/+0
2008-12-17Remove the alignment functions for now. Maybe they willLasse Collin1-114/+0
be added back in some form later, but the current version wasn't modular, so it would need fixing anyway.
2008-12-15Fix a dumb bug in .lzma decoder which was introduced inLasse Collin1-15/+14
the previous commit. (Probably the previous commit has other bugs too, it wasn't tested.)
2008-12-15Bunch of liblzma API cleanups and fixes.Lasse Collin10-64/+270
2008-12-01Validate the filter chain before checking filter-specificLasse Collin1-4/+10
memory usage.
2008-12-01Added the changes for Delta filter that should have beenLasse Collin2-2/+2
part of 656ec87882ee74b192c4ea4a233a235eca7b04d4.
2008-11-25Remove the nowadays unneeded memory limitting malloc() wrapper.Lasse Collin2-289/+0
2008-11-23VLI encoder and decoder cleanups. Made encoder returnLasse Collin2-13/+33
LZMA_PROG_ERROR in single-call mode if there's no output space.
2008-11-19Oh well, big messy commit again. Some highlights:Lasse Collin14-331/+348
- Updated to the latest, probably final file format version. - Command line tool reworked to not use threads anymore. Threading will probably go into liblzma anyway. - Memory usage limit is now about 30 % for uncompression and about 90 % for compression. - Progress indicator with --verbose - Simplified --help and full --long-help - Upgraded to the last LGPLv2.1+ getopt_long from gnulib. - Some bug fixes
2008-09-30Fixed Stream decoder to actually use the first_stream variable.Lasse Collin1-0/+5
2008-09-30Fixed uninitialized variable in Stream decoder.Lasse Collin1-0/+1
2008-09-27Fix conflicting Subblock helper filter's ID.Lasse Collin1-1/+1
2008-09-27Changed magic bytes to match the updated spec. FilenameLasse Collin2-3/+3
suffix wasn't changed yet.
2008-09-27Some API changes, bug fixes, cleanups etc.Lasse Collin9-30/+33
2008-09-13Renamed constants:Lasse Collin23-97/+94
- LZMA_VLI_VALUE_MAX -> LZMA_VLI_MAX - LZMA_VLI_VALUE_UNKNOWN -> LZMA_VLI_UNKNOWN - LZMA_HEADER_ERRRO -> LZMA_OPTIONS_ERROR
2008-09-12Improved the Stream Flags handling API.Lasse Collin6-14/+43