| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
|
|
|
|
|
|
In contrast to Automake, skipping of this test when decoders
are disabled is handled at CMake side instead of test_scripts.sh
because CMake-build doesn't create config.h.
|
|
Don't assume that Clang defines __GNUC__ as the extensions
are available in clang-cl as well (and possibly in some other
Clang variants?).
|
|
|
|
|
|
|
|
Translations and doc/xz-file-format.txt and doc/lzma-file-format.txt
were not touched.
COPYING.0BSD was added.
|
|
|
|
|
|
|
|
These test files achieve 100% code coverage in
src/liblzma/simple/riscv.c. They contain all of the instructions that
should be filtered and a few cases that should not.
|
|
|
|
Since GCC version 10, GCC no longer complains about simple implicit
integer conversions with Arithmetic operators.
For instance:
uint8_t a = 5;
uint32_t b = a + 5;
Give a warning on GCC 9 and earlier but this:
uint8_t a = 5;
uint32_t b = (a + 5) * 2;
Gives a warning with GCC 10+.
|
|
Most of these fixes are small typos and tweaks. A few were caused by bad
advice from me. Here is the summary of what is changed:
- Author line edits
- Small comment changes/additions
- Using the return value in the error messages in the fuzz targets'
coder initialization code
- Removed fuzz_encode_stream.options. This set a max length, which may
prevent some worthwhile code paths from being properly exercised.
- Removed the max_len option from fuzz_decode_stream.options for the
same reason as fuzz_encode_stream. The alone decoder fuzz target still
has this restriction.
- Altered the dictionary contents for fuzz_lzma.dict. Instead of keeping
the properties static and varying the dictionary size, the properties
are varied and the dictionary size is kept small. The dictionary size
doesn't have much impact on the code paths but the properties do.
Closes: https://github.com/tukaani-project/xz/pull/73
|
|
This fuzz target handles .xz stream encoding. The first byte of input
is used to dynamically set the preset level in order to increase the
fuzz coverage of complex critical code paths.
|
|
This fuzz target that handles LZMA alone decoding. A new fuzz
dictionary .dict was also created with common LZMA header values to
help speed up the discovery of valid headers.
|
|
All .c files can be built as separate fuzz targets. This simplifies
the Makefile by allowing us to use wildcards instead of having a
Makefile target for each fuzz target.
|
|
|
|
|
|
|
|
This tests some complicated interactions with the --suffix= option.
The suffix option must be used with --format=raw, but can optionally
be used to override the default .xz suffix.
This test also verifies some recent bugs have been correctly solved
and to hopefully avoid further regressions in the future.
|
|
|
|
The tests do not use any Gnulib replacements so they do not need to link
libgnu.a or have /lib in the include path.
|
|
This check was extended to test the code added to fix a failing assert
in ae5c07b22a6b3766b84f409f1b6b5c100469068a.
|
|
|
|
|
|
|
|
Previously if the lzip decoder was not configured then test_files.sh
would pass the lzip tests instead of skipping them.
|
|
|
|
|
|
Fixed a bug where test_compress_* would all fail if arm64 or armthumb
filters were enabled for compression but arm was disabled. Since the
grep tests only checked for "define HAVE_ENCODER_ARM", this would match
on HAVE_ENCODER_ARM64 or HAVE_ENCODER_ARMTHUMB.
Now the config.h feature test requires " 1" at the end to prevent the
prefix problem. have_feature() was also updated for this even though
there were known current bugs affecting it. This is just in case future
features have a similar prefix problem.
|
|
Several tests were missing calls to lzma_index_end() to clean up the
lzma_index structs. The memory leaks were discovered by using
-fsanitize=address with GCC.
|
|
test_block_header was not properly freeing the filter options between
calls to lzma_block_header_decode(). The memory leaks were discovered by
using -fsanitize=address with GCC.
|
|
|
|
|
|
Converts the existing lzma_index tests into tuktests and covers every
API function from index.h except for lzma_file_info_decoder, which can
be tested in the future.
|
|
The static global variables can be disabled if encoders and decoders
are not built. If they are not disabled and -Werror is used, it will
cause an usused warning as an error.
|
|
Tests lzma_str_to_filters(), lzma_str_from_filters(), and
lzma_str_list_filters() API functions.
|
|
Explicitly casting the integer to lzma_check silences the warning.
Since such an invalid value is needed in multiple tests, a constant
INVALID_LZMA_CHECK_ID was added to tests.h.
The use of 0x1000 for lzma_block.check wasn't optimal as if
the underlying type is a char then 0x1000 will be truncated to 0.
However, in these test cases the value is ignored, thus even with
such truncation the test would have passed.
|
|
Note that assigning an unsigned int to lzma_check doesn't warn
on GNU/Linux x86-64 since the enum type is unsigned on that
platform. The enum can be signed on some other platform though
so it's best to use enumeration type lzma_check in these situations.
|
|
|
|
|
|
|
|
This only occurs in test_filter_flags when the BCJ filters are not
configured and built. In this case, ARRAY_SIZE() returns 0 and causes a
type-limits warning with the loop variable since an unsigned number will
always be >= 0.
|
|
One of the global arrays of filters was only used in a test that
required both encoders and decoders to be configured in the build.
|
|
test_index_hash does not use fill_index_hash() unless both encoders
and decoders are configured in the build.
|
|
Here are the list of the most significant issues addressed:
- Avoid using internal common.h header. It's not good to copy the
constants like this but common.h cannot be included for use outside
of liblzma. This is the quickest thing to do that could be fixed later.
- Omit the INIT_FILTER macro. Initialization should be done with just
regular designated initializers.
- Use start_offset = 257 for BCJ tests. It demonstrates that Filter
Flags encoder and decoder don't validate the options thoroughly.
257 is valid only for the x86 filter. This is a bit silly but
not a significant problem in practice because the encoder and
decoder initialization functions will catch bad alignment still.
Perhaps this should be fixed but it's not urgent and doesn't need
to be in 5.4.x.
- Various tweaks to comments such as filter id -> Filter ID
|
|
Converts the existing filter flags tests into tuktests.
|
|
It's not needed in XZ Utils at least for now. It's good to support
it still because if such use is needed later, it wouldn't be
caught on GNU/Linux since malloc(0) from glibc returns non-NULL.
|
|
Tuktest index hash
|
|
|
|
|
|
|
|
|
|
It can trigger warnings from -Wshadow on some systems.
|
|
|
|
The words defined in the .xz file format specification
begin with capital letter to emphasize that they have
a specific meaning.
|
|
|
|
|
|
The shell parameter expansion using # and ## is not supported in
Solaris 10 Bourne shell (/bin/sh). Even though this is POSIX, it is not fully
portable, so we should avoid it.
|
|
Tests all API functions exported from index_hash.h. Does not have a
dedicated test for lzma_index_hash_end.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
It now tries to test as many files as easily possible.
The exit status indicates skipping if any of the files were
skipped. This way it is easy to notice if something is being
skipped when it isn't expected.
|
|
xz (but not xzdec) will normally warn about unsupported check
but since we are testing specifically such a file, it's better
to silence that warning so that it doesn't look suspicious in
test_files.sh.log.
The use of -q and -Q in xzdec is just for consistency and
doesn't affect the result at least for now.
|
|
|
|
|
|
|
|
|
|
This isn't perfect as the scripts can still fail if only
certain filters are disabled. This is still an improvement
as now "make check" has better behavior when all encoders
or decoders are disabled.
Grepping ../config.h is simple and fairly clean but it only
works if config.h was created. CMake builds don't create
config.h but they don't use these test scripts either.
Thanks to Sebastian Andrzej Siewior for reporting the problem.
Thanks to Jia Tan for the original patch which grepped xz
error messages instead of config.h.
|
|
I suspect that I used these in the original version because
Autoconf's manual describes that such a trick is needed in
some specific situations for portability reasons. None of those
situations listed on Autoconf 2.71 manual apply to these test
scripts though so this cleans them up.
|
|
|
|
Converts test_stream_flags to tuktest. Also the test will now
compile and skip properly if encoders or decoders are disabled.
Thanks to Sebastian Andrzej Siewior.
|
|
test_block_header now achieves higher test coverage. Also the
test will now compile and skip properly if encoders or decoders
are disabled.
Thanks to Sebastian Andrzej Siewior.
|
|
test_bcj_exact_size, test_check, test_hardware, and test_index will
all now compile and skip properly if encoders or decoders are disabled.
Also fixed a small typo (disabed -> disabled).
Thanks to Sebastian Andrzej Siewior.
|
|
|
|
Now tests that require threading are skipped when threading
support has been disabled.
Thanks to Sebastian Andrzej Siewior.
|
|
|
|
|
|
This test fails before commit 18d7facd3802b55c287581405c4d49c98708c136.
test_files.sh now runs xz -l for bad-3-index-uncomp-overflow.xz
because only then the previously-buggy code path gets tested.
Normal decompression doesn't use lzma_index_append() at all.
Instead, lzma_index_hash functions are used and those already
did the overflow check.
|
|
Warnings about unused tuktest_run_test conveniently tell which
test programs haven't been converted to tuktest.h yet but I
silenced that warning too for now anyway.
It is fine to use __attribute__((__unused__)) even when the
function is actually used because the attribute only means
that the function might be unused.
|
|
|
|
|
|
The script uses lcov and genhtml after running the tests
to show the code coverage statistics. The script will create
a coverage directory where it is run. It can be run both in
and out of the source directory.
|
|
|
|
Array of pointers to short strings is a bit pointless here
and now it's fully const.
|
|
|
|
|
|
|
|
|
|
lzma_vli is unsigned so trying a signed value results in
a compiler warning from -Wsign-conversion. (lzma_vli)-1
equals to LZMA_VLI_UNKNOWN anyway which is the next assertion.
|
|
|
|
Achieved 100% code coverage vli_encoder.c, vli_decoder.c, and vli_size.c
|
|
It's much nicer this way so that the test data isn't a hardcoded
table inside the C file.
|
|
|
|
|
|
This is from test_bcj_exact_size.c.
It's good to have it as a standalone file.
|
|
Created tests for all API functions exported in
src/liblzma/api/lzma/hardware.h. The tests are fairly trivial
but are helpful because they will inform users if their machines
cannot support these functions. They also improve the code
coverage metrics.
|
|
Thanks to Jia Tan for help with all the tests.
|
|
|
|
The compress() and decompress() functions were merged because
the later depends on the former so they need to be a single
test case.
|
|
This breaks -Werror because none of the tests so far use
tuktest.h and thus there are warnings about unused variables
and functions.
|
|
|
|
test_compress.sh now takes one command line argument:
a filename to be tested. If it begins with "compress_generated_"
the file will be created with create_compress_files.
This will allow parallel execution of the slow tests.
|
|
If a command line argument is given, then only the test file
of that type is created. It's quite dumb in sense that unknown
names don't give an error but it's good enough here.
Also use EXIT_FAILURE instead of 1 as exit status for errors.
|
|
|
|
It will be useless with Automake's parallel tests.
|
|
|
|
|
|
|
|
|
|
https://github.com/google/oss-fuzz/pull/3219#issuecomment-573751048
Thanks to Bhargava Shastry for sending the patch.
|
|
|
|
Thanks to Daniel Richard G.
|
|
Also changed 999 to 99 so it fits even if lzma_check happened
to be 8 bits wide.
|
|
|
|
The duplication was introduced about eleven years ago and
should have been cleaned up back then already.
This was caught by -Wduplicated-branches.
|
|
Thanks to Bhargava Shastry and Github user pdknsk.
|
|
|
|
Thanks to Cristian RodrÃguez.
|
|
Hiding them makes no sense since normally there's no error
when testing the "good" files. With "bad" files errors are
expected and then it makes sense to keep the messages hidden.
|
|
|
|
Now it should be close to the functionality of the original
version by Pavel Raiskup.
|
|
This is a simplified version of Pavel Raiskup's
original patch.
|
|
|
|
lzma_code() could incorrectly return LZMA_BUF_ERROR if
all of the following was true:
- The caller knows how many bytes of output to expect
and only provides that much output space.
- When the last output bytes are decoded, the
caller-provided input buffer ends right before
the LZMA2 end of payload marker. So LZMA2 won't
provide more output anymore, but it won't know it
yet and thus won't return LZMA_STREAM_END yet.
- A BCJ filter is in use and it hasn't left any
unfiltered bytes in the temp buffer. This can happen
with any BCJ filter, but in practice it's more likely
with filters other than the x86 BCJ.
Another situation where the bug can be triggered happens
if the uncompressed size is zero bytes and no output space
is provided. In this case the decompression can fail even
if the whole input file is given to lzma_code().
A similar bug was fixed in XZ Embedded on 2011-09-19.
|
|
Reported here:
http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/4927385
|
|
|
|
The scripts are now made executable in the build tree.
This way the scripts can be run like programs in
test_scripts.sh. Previously test_scripts.sh always
used sh but it's not correct if @POSIX_SHELL@ is set
to something else by configure.
Thanks to Jonathan Nieder for the patch.
|
|
xzdiff was clobbering the exit status from diff in a case
statement used to analyze the exit statuses from "xz" when
its operands were two compressed files. Save and restore
diff's exit status to fix this.
The bug is inherited from zdiff in GNU gzip and was fixed
there on 2009-10-09.
Thanks to Jonathan Nieder for the patch and
to Peter Pallinger for reporting the bug.
|
|
|
|
|
|
For some reason this prevented running the test only
on OS/2 and even on that it broke only recently.
Thanks to Elbert Pol.
|
|
|
|
Most distros want xz linked against shared liblzma, so
it doesn't help much to require --enable-dynamic for that.
Those who want to avoid PIC on x86-32 to get better
performance, can still do it e.g. by using --disable-shared
to compile xz and then another pass to compile shared liblzma.
Part of these static/dynamic tricks were needed for Windows
in the past. Nowadays we rely on GCC and binutils to do the
right thing with auto-import. If the Autotooled build system
needs to support some other toolchain on Windows in the future,
this may need some rethinking.
|
|
Testing compression at level -4 now requires 48 MiB of free store at
compression time and 5 MiB at decompression time.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
|
|
It broke when --memory option was removed from xzdec.
Thanks to Jonathan Nieder.
|
|
Thanks to Jonathan Nieder.
|
|
Thanks to Jonathan Nieder.
|
|
This breaks API and ABI but most apps are not affected
since most apps don't use this part of the API. You will
get a compile error if you are using anything that got
broken.
Summary of changes:
- Ability to store Stream Flags, which are needed
for random-access reading in multi-Stream files.
- Separate function to set size of Stream Padding.
- Iterator structure makes it possible to read the same
lzma_index from multiple threads at the same time.
- A lot faster code to locate Blocks.
- Removed lzma_index_equal() without adding anything
to replace it. I don't know what it should do exactly
with the new features and what actually needs this
function in the first place other than test_index.c,
which now has its own code to compare lzma_indexes.
|
|
This was introduced in
bd13b04e202b6f495a68eb0766f97085b7c50a06.
Thanks to Jim Meyering for noticing it.
|
|
lzma_index_read() didn't skip over Stream Padding
if it was the first record in the Index.
lzma_index_cat() didn't combine small Indexes correctly.
The test suite was updated to check for these bugs.
These bugs didn't affect the xz command line tool or
most users of liblzma in any way.
|
|
This replaces bswap.h and integer.h.
The tuklib module uses <byteswap.h> on GNU,
<sys/endian.h> on *BSDs and <sys/byteorder.h>
on Solaris, which may contain optimized code
like inline assembly.
|
|
|
|
|
|
Fix the ordering of libgnu.a and LTLIBINTL on the linker
command line and added missing LTLIBINTL to tests/Makefile.am.
|
|
|
|
Some minor documentation cleanups were made at the same time.
|
|
were linked with crt*.o, which are copyrighted, and thus the
old test files were not in the public domain as a whole. They
are freely distributable though, but it is better to be careful
and avoid including any copyrighted pieces in the test files.
The new files are just compiled and assembled object files,
and thus don't contain any copyrighted code.
|
|
on Windows. sysdefs.h no longer #includes lzma.h, so lzma.h
has to be #included separately where needed.
|
|
|
|
|
|
disabled by default in configure.ac.
|
|
The internal implementation is still using the name "simple".
It may need some cleanups, so I look at it later.
|
|
Half of developers were already forgetting to use these
functions, which could have caused total breakage in some future
liblzma version or even now if --enable-small was used. Now
liblzma uses pthread_once() to do the initializations unless
it has been built with --disable-threads which make these
initializations thread-unsafe.
When --enable-small isn't used, liblzma currently gets needlessly
linked against libpthread (on systems that have it). While it is
stupid for now, liblzma will need threads in future anyway, so
this stupidity will be temporary only.
When --enable-small is used, different code CRC32 and CRC64 is
now used than without --enable-small. This made the resulting
binary slightly smaller, but the main reason was to clean it up
and to handle the lack of lzma_init_check().
The pkg-config file lzma.pc was renamed to liblzma.pc. I'm not
sure if it works correctly and portably for static linking
(Libs.private includes -pthread or other operating system
specific flags). Hopefully someone complains if it is bad.
lzma_rc_prices[] is now included as a precomputed array even
with --enable-small. It's just 128 bytes now that it uses uint8_t
instead of uint32_t. Smaller array seemed to be at least as fast
as the more bloated uint32_t array on x86; hopefully it's not bad
on other architectures.
|
|
|
|
|
|
|
|
|
|
lzma, unlzma, and lzcat in "make install" for backwards
compatibility with LZMA Utils 4.32.x; I'm not sure if this
should be the default though.
|
|
- Updated to the latest, probably final file format version.
- Command line tool reworked to not use threads anymore.
Threading will probably go into liblzma anyway.
- Memory usage limit is now about 30 % for uncompression
and about 90 % for compression.
- Progress indicator with --verbose
- Simplified --help and full --long-help
- Upgraded to the last LGPLv2.1+ getopt_long from gnulib.
- Some bug fixes
|
|
of 1e8e4fd1f3e50129b4541406ad765d2aa1233943.
|
|
This also fixes a bug related to --suffix option. Some issues
with suffixes with --format=raw were not fixed.
|
|
|
|
|
|
|
|
suffix wasn't changed yet.
|
|
|
|
1dcecfb09b55157b8653d747963069c8bed74f04.
|
|
|
|
- LZMA_VLI_VALUE_MAX -> LZMA_VLI_MAX
- LZMA_VLI_VALUE_UNKNOWN -> LZMA_VLI_UNKNOWN
- LZMA_HEADER_ERRRO -> LZMA_OPTIONS_ERROR
|
|
|
|
|
|
broken. API has changed a lot and it will still change a
little more here and there. The command line tool doesn't
have all the required changes to reflect the API changes, so
it's easy to get "internal error" or trigger assertions.
|
|
|
|
from test_block_header.c.
|
|
specification. Simplify things by removing most of the
support for known uncompressed size in most places.
There are some miscellaneous changes here and there too.
The API of liblzma has got many changes and still some
more will be done soon. While most of the code has been
updated, some things are not fixed (the command line tool
will choke with invalid filter chain, if nothing else).
Subblock filter is somewhat broken for now. It will be
updated once the encoded format of the Subblock filter
has been decided.
|
|
|
|
liblzma's API.
|
|
|
|
|
|
descriptions in the test files' README.
|
|
|
|
|
|
|
|
|
|
bad-single-none-too_long_vli.lzma.
|
|
lzma_metadata.header_metadata_size == LZMA_VLI_VALUE_UNKNOWN
is not allowed at all. To indicate missing Header Metadata
Block, header_metadata_size must be set to zero. This is
what Metadata decoder does after this patch too.
Note that other missing fields in lzma_metadata are still
indicated with LZMA_VLI_VALUE_UNKNOWN. This isn't as
illogical as it sounds at first, because missing Size of
Header Metadata Block means that Header Metadata Block is
not present in the Stream. With other Metadata fields,
a missing field means only that the value is unknown.
|
|
|
|
This new set of tests compress and decompress several
test files with many different compression options.
This set of tests will be extended later.
|
|
|
|
|
