| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
lzma_code() could incorrectly return LZMA_BUF_ERROR if
all of the following was true:
- The caller knows how many bytes of output to expect
and only provides that much output space.
- When the last output bytes are decoded, the
caller-provided input buffer ends right before
the LZMA2 end of payload marker. So LZMA2 won't
provide more output anymore, but it won't know it
yet and thus won't return LZMA_STREAM_END yet.
- A BCJ filter is in use and it hasn't left any
unfiltered bytes in the temp buffer. This can happen
with any BCJ filter, but in practice it's more likely
with filters other than the x86 BCJ.
Another situation where the bug can be triggered happens
if the uncompressed size is zero bytes and no output space
is provided. In this case the decompression can fail even
if the whole input file is given to lzma_code().
A similar bug was fixed in XZ Embedded on 2011-09-19.
|
|
|
|
|
|
Thanks to Jim Meyering.
|
|
|
|
|
|
Spot candidates by running these commands:
git ls-files |xargs perl -0777 -n \
-e 'while (/\b(then?|[iao]n|i[fst]|but|f?or|at|and|[dt]o)\s+\1\b/gims)' \
-e '{$n=($` =~ tr/\n/\n/ + 1); ($v=$&)=~s/\n/\\n/g; print "$ARGV:$n:$v\n"}'
Thanks to Jim Meyering for the original patch.
|
|
|
|
It didn't mention the return value that is used if
an error occurs.
|
|
Empty Block was created if the input buffer was empty.
Empty Block wastes a few bytes of space, but more importantly
it triggers a bug in XZ Utils 5.0.1 and older when trying
to decompress such a file. 5.0.1 and older consider such
files to be corrupt. I thought that no encoder creates empty
Blocks when releasing 5.0.2 but I was wrong.
|
|
This return value was missing from the API comments of
four functions.
|
|
The biggest problem was that the integrity check type
wasn't validated, and e.g. lzma_easy_buffer_encode()
would create a corrupt .xz Stream if given an unsupported
Check ID. Luckily applications don't usually try to use
an unsupport Check ID, so this bug is unlikely to cause
many real-world problems.
|
|
Passing --disable-decoders to configure broke a few
encoders due to missing #ifdefs in filter_common.c.
Thanks to Jason Gorski for the patch.
|
|
It leaks old filter options structures (hundred bytes or so)
every time the lzma_stream is reinitialized. With the xz tool,
this happens when compressing multiple files.
|
|
|
|
The decoder considered empty LZMA2 streams to be corrupt.
This shouldn't matter much with .xz files, because no encoder
creates empty LZMA2 streams in .xz. This bug is more likely
to cause problems in applications that use raw LZMA2 streams.
|
|
|
|
Thanks to Cristian Rodríguez for the original patch.
|
|
|
|
|
|
If any of the reserved members in lzma_stream are non-zero
or non-NULL, LZMA_OPTIONS_ERROR is returned. It is possible
that a new feature in the future is indicated by just setting
a reserved member to some other value, so the old liblzma
version need to catch it as an unsupported feature.
|
|
This lets compiler use shifting instead of 64-bit division.
|
|
|
|
Adding support for LZMA_FINISH for Index encoding and
decoding needed tiny additions to the relevant .c files too.
|
|
lzma_chunk_size() was commented out because it is
currently useless.
|
|
|
|
depth=0 was missing.
|
|
This should reduce the cases where --extreme makes
compression worse. On the other hand, some other
files may now benefit slightly less from --extreme.
|
|
"Extreme" mode might need some further tweaking still.
Docs were not updated yet.
|
|
It was 8 + nice_len / 4, now it is 4 + nice_len / 4.
This allows faster settings at lower nice_len values,
even though it seems that I won't use automatic depth
calcuation with HC3 and HC4 in the presets.
|
|
When using -O2 with GCC, it liked to swap two comparisons
in one "if" statement. It's otherwise fine except that
the latter part, which is seemingly never executed, got
executed (nothing wrong with that) and then triggered
warning in Valgrind about conditional jump depending on
uninitialized variable. A few people find this annoying
so do things a bit differently to avoid the warning.
|
|
|
|
|
|
Thanks to Jonathan Nieder for the reminder.
|
|
This should avoid some minor portability issues.
|
|
The spec isn't finished and the code didn't compile anymore.
It won't be included in XZ Utils 5.0.0. It's easy to get it
back once the spec is done.
|
|
It isn't really useful so omitting it makes things
shorter and slightly more readable.
|
|
With bad luck, lzma_code() could return LZMA_BUF_ERROR
when it shouldn't.
This has been here since the early days of liblzma.
It got triggered by the modifications made to the xz
tool in commit 18c10c30d2833f394cd7bce0e6a821044b15832f
but only when decompressing .lzma files. Somehow I managed
to miss testing that with Valgrind earlier.
This fixes <http://bugs.gentoo.org/show_bug.cgi?id=305591>.
Thanks to Rafał Mużyło for helping to debug it on IRC.
|
|
Thanks to Jonathan Nieder.
|
|
|
|
This allows the files to work on HURD.
Thanks to Jonathan Nieder.
|
|
lzma_block.version has to be initialized even for
lzma_block_header_decode(). This way a future version
of liblzma won't allocate memory in a way that an old
application doesn't know how to free it.
The subtlety of this change is that all current apps
using lzma_block_header_decode() will keep working for
now, because the only possible version value is zero,
and lzma_block_header_decode() unconditionally sets the
version to zero even now. Unless fixed, these apps will
break in the future if a new version of the Block options
is ever needed.
|
|
|
|
This affects lzma_memusage() and lzma_memlimit_get().
|
|
This breaks API and ABI but most apps are not affected
since most apps don't use this part of the API. You will
get a compile error if you are using anything that got
broken.
Summary of changes:
- Ability to store Stream Flags, which are needed
for random-access reading in multi-Stream files.
- Separate function to set size of Stream Padding.
- Iterator structure makes it possible to read the same
lzma_index from multiple threads at the same time.
- A lot faster code to locate Blocks.
- Removed lzma_index_equal() without adding anything
to replace it. I don't know what it should do exactly
with the new features and what actually needs this
function in the first place other than test_index.c,
which now has its own code to compare lzma_indexes.
|
|
|
|
I understood that this is nicer, because then people
don't need to worry about the LZMA_API_STATIC macro.
Thanks to Charles Wilson and Keith Marshall.
|
|
lzma_index_read() didn't skip over Stream Padding
if it was the first record in the Index.
lzma_index_cat() didn't combine small Indexes correctly.
The test suite was updated to check for these bugs.
These bugs didn't affect the xz command line tool or
most users of liblzma in any way.
|
|
The Index decoder code didn't perfectly match the API docs,
which said that *i will be set to point to the decoded Index
only after decoding has succeeded. The docs were a bit unclear
too.
Now the decoder will initially set *i to NULL. *i will be set
to point to the decoded Index once decoding has succeeded.
This simplifies applications too, since it avoids dangling
pointers.
|
|
|
|
when --enable-small has been specified.
|
|
in the text editor.
|
|
I had hoped to keep liblzma as purely a compression
library as possible (e.g. file I/O will go into
a different library), but it seems that applications
linking agaisnt liblzma need some way to determine
the memory usage limit, and knowing the amount of RAM
is one reasonable way to help making such decisions.
Thanks to Jonathan Nieder for the original patch.
|
|
Originally the idea was that using LZMA_FULL_FLUSH
with Stream encoder would read the filter chain
from the same array that was used to intialize the
Stream encoder. Since most apps wouldn't use
LZMA_FULL_FLUSH, most apps wouldn't need to keep
the filter chain available after initializing the
Stream encoder. However, due to my mistake, it
actually required keeping the array always available.
Since setting the new filter chain via the array
used at initialization time is not a nice way to do
it for a couple of reasons, this commit ditches it
and introduces lzma_filters_update(). This new function
replaces also the "persistent" flag used by LZMA2
(and to-be-designed Subblock filter), which was also
an ugly thing to do.
Thanks to Alexey Tourbin for reminding me about the problem
that Stream encoder used to require keeping the filter
chain allocated.
|
|
It was meant to be lzma_filters_copy(), not lzma_filters_dup().
|
|
This will be needed internally by liblzma once I fix
a design mistake in the encoder API. This function may
be useful to applications too so it's good to export it.
|
|
This replaces bswap.h and integer.h.
The tuklib module uses <byteswap.h> on GNU,
<sys/endian.h> on *BSDs and <sys/byteorder.h>
on Solaris, which may contain optimized code
like inline assembly.
|
|
in lz_encoder_hash.h.
|
|
Seems that it is a problem in some cases if the same
version of XZ Utils produces different output on different
endiannesses, so this commit fixes that problem. The output
will still vary between different XZ Utils versions, but I
cannot avoid that for now.
This commit bloatens the code on big endian systems by 1 KiB,
which should be OK since liblzma is bloated already. ;-)
|
|
|
|
|
|
Thanks to Christian Weisgerber for pointing out some of these.
|
|
|
|
and use a fix that works on all systems using
GNU assembler.
Maybe the assembler code is used e.g. on Solaris x86
but let's worry about it if this doesn't work on it.
|
|
Thanks to Karl Berry.
|
|
|
|
Added lzma_nothrow for every function. It adds
throw() when the header is used in C++ code.
Some lzma_attrs were added or removed.
Lots of comments were improved.
|
|
Thanks to Jonathan Stott for the bug report.
|
|
|
|
This is a quick and slightly dirty fix to make the code
conform to the latest file format specification. Without
this patch, it's possible to make corrupt files by
specifying start offset that is not a multiple of the
filter's alignment. Custom start offset is almost never
used, so this was only a minor bug.
The xz command line tool doesn't validate the start offset,
so one will get a bit unclear error message if trying to use
an invalid start offset.
|
|
use AC_PROG_SED. We don't do anything fancy with sed,
so this should work OK. libtool 2.2 sets SED but 1.5
doesn't, so $(SED) happened to work when using libtool 2.2.
|
|
|
|
Don't use libtool convenience libraries to avoid recently
discovered long-standing subtle but somewhat severe bugs
in libtool (at least 1.5.22 and 2.2.6 are affected). It
was found when porting XZ Utils to Windows
<http://lists.gnu.org/archive/html/libtool/2009-06/msg00070.html>
but the problem is significant also e.g. on GNU/Linux.
Unless --disable-shared is passed to configure, static
library built from a set of convenience libraries will
contain PIC objects. That is, while libtool builds non-PIC
objects too, only PIC objects will be used from the
convenience libraries. On 32-bit x86 (tested on mobile XP2400+),
using PIC instead of non-PIC makes the decompressor 10 % slower
with the default CFLAGS.
So while xz was linked against static liblzma by default,
it got the slower PIC objects unless --disable-shared was
used. I tend develop and benchmark with --disable-shared
due to faster build time, so I hadn't noticed the problem
in benchmarks earlier.
This commit also adds support for building Windows resources
into liblzma and executables.
|
|
|
|
to prepare for building them with Autotools.
|
|
Autotools based builds on Windows.
|
|
the Autotools based build system. It's not good yet, more
fixes will follow.
|
|
Fix the ordering of libgnu.a and LTLIBINTL on the linker
command line and added missing LTLIBINTL to tests/Makefile.am.
|
|
Released .xz spec 1.0.2 due to this fix too.
|
|
via lzma_block structure.
This changes ABI but not doesn't break API.
|
|
-Wno-uninitialized to silence a bogus warning.
|
|
liblzma tries to avoid useless free()/malloc() pairs in
initialization when multiple files are handled using the
same lzma_stream. This didn't work with filter chains
due to comparison of wrong pointers in lzma_next_coder_init(),
making liblzma think that no memory reallocation is needed
even when it actually is.
Easy way to trigger this bug is to decompress two files with
a single xz command. The first file should have e.g. x86+LZMA2
as the filter chain, and the second file just LZMA2.
|
|
On 64-bit system it would have gone into infinite
loop if a single input buffer was over 4 GiB (unlikely).
|
|
Some minor documentation cleanups were made at the same time.
|
|
Fortunately, this bug had no security risk other than accepting
some corrupt files as valid.
|
|
|
|
pieces to avoid unneeded dependencies making statically
linked applications bigger than needed.
|
|
Now configure.ac will get the version number directly from
src/liblzma/api/lzma/version.h. The intent is to reduce the
number of places where the version number is duplicated. In
future, support for displaying Git commit ID may be added too.
|
|
Here DOS-like means DOS, Windows, and OS/2.
|
|
line. Previously custom CFLAGS worked only when they were
passed to configure.
|
|
table, which is used also by LZ encoder. This was needed
because calling lzma_crc32() and ignoring the result is
a no-op due to lzma_attr_pure.
|
|
avoid __declspec(dllexport) equivalent.
|
|
need to #define when linking against static liblzma on
platforms like Windows. Most developers don't need to
care about LZMA_API_STATIC at all.
|
|
- Use call/ret pair to get instruction pointer for PIC.
- Use PIC only if PIC or __PIC__ is #defined.
- The code should work on MinGW and Darwin in addition
to GNU/Linux and Solaris.
|
|
other compilers than MinGW. This may hurt readability
of the API headers slightly, but I don't know any
better way to do this.
|
|
function should be rewritten anyway.
|
|
functions, and cleaned up filter.h API header a little.
May be very buggy, not tested yet.
|
|
on Windows.
|
|
on Windows. sysdefs.h no longer #includes lzma.h, so lzma.h
has to be #included separately where needed.
|
|
beginning of the line.
|
|
stdint.h or inttypes.h.
|
|
|
|
|
|
triggered by the previous commit, since these variables were
not used by anything before support for a preset dictionary.
|
|
and LZMA2. It is not supported by the .xz format or the xz
command line tool yet.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
LZMA_DATA_ERROR with valid data. The bug was added in
e114502b2bc371e4a45449832cb69be036360722.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
can be used outside lzma2_encoder.c.
|
|
|
|
lzma_memlimit_encoder and lzma_memlimit_decoder to
lzma_raw_encoder_memlimit and lzma_raw_decoder_memlimit. :-(
Now it is fixed. Hopefully it doesn't cause too much trouble
to those who already thought API is stable.
|
|
of 4.999.7beta.
|
|
|
|
|
|
The internal implementation is still using the name "simple".
It may need some cleanups, so I look at it later.
|
|
should have been in the previous commit.
|
|
Half of developers were already forgetting to use these
functions, which could have caused total breakage in some future
liblzma version or even now if --enable-small was used. Now
liblzma uses pthread_once() to do the initializations unless
it has been built with --disable-threads which make these
initializations thread-unsafe.
When --enable-small isn't used, liblzma currently gets needlessly
linked against libpthread (on systems that have it). While it is
stupid for now, liblzma will need threads in future anyway, so
this stupidity will be temporary only.
When --enable-small is used, different code CRC32 and CRC64 is
now used than without --enable-small. This made the resulting
binary slightly smaller, but the main reason was to clean it up
and to handle the lack of lzma_init_check().
The pkg-config file lzma.pc was renamed to liblzma.pc. I'm not
sure if it works correctly and portably for static linking
(Libs.private includes -pthread or other operating system
specific flags). Hopefully someone complains if it is bad.
lzma_rc_prices[] is now included as a precomputed array even
with --enable-small. It's just 128 bytes now that it uses uint8_t
instead of uint32_t. Smaller array seemed to be at least as fast
as the more bloated uint32_t array on x86; hopefully it's not bad
on other architectures.
|
|
that was related to LZMA_MODE_FAST. The original code is slightly
faster although it compresses slightly worse. But since it is fast
mode, it is better to select the faster version.
|
|
The API and ABI should now be very close to stable,
although the code behind it isn't yet.
|
|
|
|
be added back in some form later, but the current version
wasn't modular, so it would need fixing anyway.
|
|
the previous commit. (Probably the previous commit has
other bugs too, it wasn't tested.)
|
|
|
|
which made LZ decoder return too early after dictionary
reset. This fixes it.
|
|
|
|
to validate the filter options. Add missing validation
to LZMA2 encoder when options are changed in the middle
of encoding.
|
|
memory usage.
|
|
to validate the filter options.
|
|
part of 656ec87882ee74b192c4ea4a233a235eca7b04d4.
|
|
also in the first LZMA chunk after a dictionary reset in
uncompressed chunk.
|
|
the options.
|
|
|
|
LZMA_PROG_ERROR in single-call mode if there's no output
space.
|
|
- Updated to the latest, probably final file format version.
- Command line tool reworked to not use threads anymore.
Threading will probably go into liblzma anyway.
- Memory usage limit is now about 30 % for uncompression
and about 90 % for compression.
- Progress indicator with --verbose
- Simplified --help and full --long-help
- Upgraded to the last LGPLv2.1+ getopt_long from gnulib.
- Some bug fixes
|
|
|
|
|
|
|
|
|
|
suffix wasn't changed yet.
|
|
|
|
|
|
|
|
- LZMA_VLI_VALUE_MAX -> LZMA_VLI_MAX
- LZMA_VLI_VALUE_UNKNOWN -> LZMA_VLI_UNKNOWN
- LZMA_HEADER_ERRRO -> LZMA_OPTIONS_ERROR
|
|
|
|
been removed in 2ba01bfa755e47ff6af84a978e3c8d63d7d2775e.
|
|
|
|
|
|
Use LZMA_PROG_ERROR instead of LZMA_HEADER_ERROR if the Filter ID
is in the reserved range. This allows Block Header encoder to
detect unallowed Filter IDs, which is good for Stream encoder.
|
|
|
|
|
|
code from block_private.h to block_decoder.c. Now the Block
encoder doesn't need compressed_size and uncompressed_size
from lzma_block structure to be initialized.
|
|
|
|
|
|
|
|
simply nothing that would use it. Allow LZMA_FINISH to the
decoders, which will usually ignore it (auto decoder and
Stream decoder being exceptions).
|
|
|
|
LZMA_Alone files. Decoding of concatenated LZMA_Alone files is
intentionally not supported, so it is better to put this in
auto decoder than LZMA_Alone decoder.
|
|
|
|
|
|
|
|
broken. API has changed a lot and it will still change a
little more here and there. The command line tool doesn't
have all the required changes to reflect the API changes, so
it's easy to get "internal error" or trigger assertions.
|
|
|
|
literal coder as part of the main LZMA encoder or
decoder structure.
Make the LZMA decoder to rely on the current internal API
to free the allocated memory in case an error occurs.
|
|
|
|
specification. Simplify things by removing most of the
support for known uncompressed size in most places.
There are some miscellaneous changes here and there too.
The API of liblzma has got many changes and still some
more will be done soon. While most of the code has been
updated, some things are not fixed (the command line tool
will choke with invalid filter chain, if nothing else).
Subblock filter is somewhat broken for now. It will be
updated once the encoded format of the Subblock filter
has been decided.
|
|
introduced in 369f72fd656f537a9a8e06f13e6d0d4c242be22f.
|
|
misunderstanding of the code. There's no tiny fix for this
problem, so I also cleaned up the code in general.
This reduces the speed of the encoder 2-5 % in the fastest
compression mode ("lzma -1"). High compression modes should
have no noticeable performance difference.
This commit breaks things (especially LZMA_SYNC_FLUSH) but I
will fix them once the new format and LZMA2 has been roughly
implemented. Plain LZMA won't support LZMA_SYNC_FLUSH at all
and won't be supported in the new .lzma format. This may
change still but this is what it looks like now.
Support for known uncompressed size (that is, LZMA or LZMA2
without EOPM) is likely to go away. This means there will
be API changes.
|
|
memmove, and memset.
|
|
|
|
|
|
liblzma's API.
|
|
size. The "fix" breaks LZMA_SYNC_FLUSH at end of stream
with known uncompressed size, but since it currently seems
likely that support for encoding with known uncompressed
size will go away anyway, I'm not fixing this problem now.
|
|
annoying, because "make dist" put two copies of sysdefs.h
into the tarball instead of the symlink.
|
|
once we have a stable release (won't be very soon). The
version number is no longer related to version of LZMA SDK.
Made some small Automake-related changes to toplevel
Makefile.am and configure.ac.
|
|
lzma_extra_free().
|
|
|
|
|
|
|
|
|
|
|
|
theoretical data corruption, which should be very hard to trigger
even intentionally.
|
|
right shift with as fast version that doesn't need
arithmetic right shift. Removed the related check from
configure.ac.
|
|
|
|
(And BTW, the previous commit actually did change the
program logic slightly.)
|
|
word literal instead of char for better consistency.
There are still some names with _char instead of _literal
in lzma_optimum, these may be changed later.
Renamed length coder variables.
This commit doesn't change the program logic.
|
|
specific to liblzma and was *not* present in LZMA SDK.
|
|
|
|
|
|
|
