| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Added a install-data-local target to install the Doxygen documentation
only when it has been generated. In order to correctly remove the docs,
a corresponding uninstall-local target was added.
If the doxygen docs exist in the source tree, they will also be included
in the distribution now too.
|
|
Instead of having Doxyfile.in configured by Autoconf, the Doxyfile
can have the tags that need to be configured piped into the doxygen
command through stdin with the overrides after Doxyfile's contents.
Going forward, the documentation should be generated in two different
modes: liblzma or internal.
liblzma is useful for most users. It is the documentation for just
the liblzma API header files. This is the default.
internal is for people who want to understand how xz and liblzma work.
It might be useful for people who want to contribute to the project.
|
|
|
|
|
|
Converts the existing lzma_index tests into tuktests and covers every
API function from index.h except for lzma_file_info_decoder, which can
be tested in the future.
|
|
Also remove unneeded "sandbox_allowed = false;" as this code
will never be run more than once (making it work with multiple
input files isn't trivial).
|
|
|
|
The warning causes the exit status to be 2, so this will cause problems
for many scripted use cases for xz. The sandbox usage is already very
limited already, so silently disabling this allows it to be more usable.
|
|
Thanks to Xin Li for recommending the fix.
|
|
The warning is only used when errno == ENOSYS. Otherwise, xz still
issues a fatal error.
|
|
If a system has the Capsicum header files but does not actually
implement the system calls, then this would render xz unusable. Instead,
we can check if errno == ENOSYS and not issue a fatal error.
|
|
cap_enter() puts the process into the sandbox. If later calls to
cap_rights_limit() fail, then the process can still have some extra
protections.
|
|
lzma_lzma_preset() does not guarentee that the lzma_options_lzma are
usable in an encoder even if it returns false (success). If liblzma
is built with default configurations, then the options will always be
usable. However if the match finders hc3, hc4, or bt4 are disabled, then
the options may not be usable depending on the preset level requested.
The documentation was updated to reflect this complexity, since this
behavior was unclear before.
|
|
Thanks to autoantwort for reporting the issue and suggesting
a different patch:
https://github.com/tukaani-project/xz/pull/42
|
|
The static global variables can be disabled if encoders and decoders
are not built. If they are not disabled and -Werror is used, it will
cause an usused warning as an error.
|
|
The '\n' renders as a newline when the comments are converted to html
by Doxygen.
|
|
Shorten the description for lzma_raw_encoder_memusage() and
lzma_raw_decoder_memusage().
|
|
|
|
All functions now explicitly specify parameter and return values.
The notes and code annotations were moved before the parameter and
return value descriptions for consistency.
Also, the description above lzma_filter_encoder_is_supported() about
not being able to list available filters was removed since
lzma_str_list_filters() will do this.
|
|
|
|
In the C99 and C17 standards, section 6.5.6 paragraph 8 means that
adding 0 to a null pointer is undefined behavior. As of writing,
"clang -fsanitize=undefined" (Clang 15) diagnoses this. However,
I'm not aware of any compiler that would take advantage of this
when optimizing (Clang 15 included). It's good to avoid this anyway
since compilers might some day infer that pointer arithmetic implies
that the pointer is not NULL. That is, the following foo() would then
unconditionally return 0, even for foo(NULL, 0):
void bar(char *a, char *b);
int foo(char *a, size_t n)
{
bar(a, a + n);
return a == NULL;
}
In contrast to C, C++ explicitly allows null pointer + 0. So if
the above is compiled as C++ then there is no undefined behavior
in the foo(NULL, 0) call.
To me it seems that changing the C standard would be the sane
thing to do (just add one sentence) as it would ensure that a huge
amount of old code won't break in the future. Based on web searches
it seems that a large number of codebases (where null pointer + 0
occurs) are being fixed instead to be future-proof in case compilers
will some day optimize based on it (like making the above foo(NULL, 0)
return 0) which in the worst case will cause security bugs.
Some projects don't plan to change it. For example, gnulib and thus
many GNU tools currently require that null pointer + 0 is defined:
https://lists.gnu.org/archive/html/bug-gnulib/2021-11/msg00000.html
https://www.gnu.org/software/gnulib/manual/html_node/Other-portability-assumptions.html
In XZ Utils null pointer + 0 issue should be fixed after this
commit. This adds a few if-statements and thus branches to avoid
null pointer + 0. These check for size > 0 instead of ptr != NULL
because this way bugs where size > 0 && ptr == NULL will likely
get caught quickly. None of them are in hot spots so it shouldn't
matter for performance.
A little less readable version would be replacing
ptr + offset
with
offset != 0 ? ptr + offset : ptr
or creating a macro for it:
#define my_ptr_add(ptr, offset) \
((offset) != 0 ? ((ptr) + (offset)) : (ptr))
Checking for offset != 0 instead of ptr != NULL allows GCC >= 8.1,
Clang >= 7, and Clang-based ICX to optimize it to the very same code
as ptr + offset. That is, it won't create a branch. So for hot code
this could be a good solution to avoid null pointer + 0. Unfortunately
other compilers like ICC 2021 or MSVC 19.33 (VS2022) will create a
branch from my_ptr_add().
Thanks to Marcin Kowalczyk for reporting the problem:
https://github.com/tukaani-project/xz/issues/36
|
|
|
|
|
|
|
|
lzma_microlzma_decoder -> lzma_microlzma_encoder
|
|
Standardizing each function to always specify parameters and return
values. Also moved the parameters and return values to the end of each
function description.
|
|
|
|
|
|
On MicroBlaze, GCC 12 is broken in sense that
__has_attribute(__symver__) returns true but it still doesn't
support the __symver__ attribute even though the platform is ELF
and symbol versioning is supported if using the traditional
__asm__(".symver ...") method. Avoiding the traditional method is
good because it breaks LTO (-flto) builds with GCC.
See also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101766
For now the only extra symbols in liblzma_linux.map are the
compatibility symbols with the patch that spread from RHEL/CentOS 7.
These require the use of __symver__ attribute or __asm__(".symver ...")
in the C code. Compatibility with the patch from CentOS 7 doesn't
seem valuable on MicroBlaze so use liblzma_generic.map on MicroBlaze
instead. It doesn't require anything special in the C code and thus
no LTO issues either.
An alternative would be to detect support for __symver__
attribute in configure.ac and CMakeLists.txt and fall back
to __asm__(".symver ...") but then LTO would be silently broken
on MicroBlaze. It sounds likely that MicroBlaze is a special
case so let's treat it as a such because that is simpler. If
a similar issue exists on some other platform too then hopefully
someone will report it and this can be reconsidered.
(This doesn't do the same fix in CMakeLists.txt. Perhaps it should
but perhaps CMake build of liblzma doesn't matter much on MicroBlaze.
The problem breaks the build so it's easy to notice and can be fixed
later.)
Thanks to Vincent Fazio for reporting the problem and proposing
a patch (in the end that solution wasn't used):
https://github.com/tukaani-project/xz/pull/32
|
|
Use "member" to refer to struct members as that's the term used
by the C standard.
Use lzma_options_delta.dist and such in docs so that in Doxygen's
HTML output they will link to the doc of the struct member.
Clean up a few trailing white spaces too.
|
|
|
|
|
|
Also adjusted preset value => preset level.
|
|
It gives C4146 here since unary minus with unsigned integer
is still unsigned (which is the intention here). Doing it
with substraction makes it clearer and avoids the warning.
Thanks to Nathan Moinvaziri for reporting this.
|
|
Standardizing each function to always specify parameters and return
values. Also moved the parameters and return values to the end of each
function description.
A few small things were reworded and long sentences broken up.
|
|
All functions now explicitly specify parameter and return values.
|
|
All functions now explicitly specify parameter and return values.
Also moved the note about SHA-256 functions not being exported to the
top of the file.
|
|
All functions now explicitly specify parameter and return values.
|
|
|
|
Add \private above this field and its sub-fields since it is not meant
to be modified by users.
|
|
LZMA_MEMLIMIT_ERROR was missing the "<" character needed to put
documentation after a member.
|
|
Standardizing each function to always specify params and return values.
Also fixed a small grammar mistake.
|
|
|
|
Added [out] annotations to parameters that are pointers and can have
their value changed. Also added a clarification to lzma_vli_is_valid.
|
|
Document LZMA_DELTA_DIST_MIN and LZMA_DELTA_DIST_MAX for completeness
and to avoid Doxygen warnings.
|
|
All functions now explicitly specify parameter and return values.
Also reworded the description of lzma_index_hash_init() for readability.
|
|
start_time is relative to an arbitary point in time, it's not
time of day, so using it for anything else than time differences
wouldn't make sense.
|
|
Now, the LZMA_VERSION_MAJOR, LZMA_VERSION_MINOR, and LZMA_VERSION_PATCH
macros do not need to be on consecutive lines in version.h. They can be
separated by more whitespace, comments, or even other content, as long
as they appear in the proper order (major, minor, patch).
|
|
|
|
Specified parameter and return values for API functions and documented
a few more of the macros.
|
|
|
|
The bug is only a problem in applications that do not properly terminate
the filters[] array with LZMA_VLI_UNKNOWN or have more than
LZMA_FILTERS_MAX filters. This bug does not affect xz.
|
|
Tests lzma_str_to_filters(), lzma_str_from_filters(), and
lzma_str_list_filters() API functions.
|
|
|
|
Added a few sentences to the description for lzma_block_encoder() and
lzma_block_decoder() to highlight that the Block Header must be coded
before calling these functions.
|
|
|
|
|
|
|
|
|
|
Standardizing each function to always specify params and return values.
Output pointer parameters are also marked with doxygen style [out] to
make it clear. Any note sections were also moved above the parameter and
return sections for consistency.
|
|
The flag description for LZMA_STR_NO_VALIDATION was previously confusing
about the treatment for filters than cannot be used with .xz format
(lzma1) without using LZMA_STR_ALL_FILTERS. Now, it is clear that
LZMA_STR_NO_VALIDATION is not a super set of LZMA_STR_ALL_FILTERS.
|
|
The workflow action for our CI pipeline can only reference artifacts in
the source directory, so we should ignore these files if the ci_build.sh
is run locally.
|
|
|
|
|
|
mythread.h and thus liblzma already does it.
|
|
|
|
This way, if xz is stopped the elapsed time and estimated time
remaining won't get confused by the amount of time spent in
the stopped state.
This raises SIGSTOP. It's not clear to me if this is the correct way.
POSIX and glibc docs say that SIGTSTP shouldn't stop the process if
it is orphaned but this commit doesn't attempt to handle that.
Search for SIGTSTP in section 2.4.3:
https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
|
|
Thanks to Rafael Fontenelle.
|
|
|
|
Clang can be configured to fake a too high GCC version so
this way it's more robust.
|
|
The previous documentation for lzma_str_to_filters() was technically
correct, but misleading. lzma_str_to_filters() returns NULL on success,
which is in practice always defined to 0. This is the same value as
LZMA_OK, but lzma_str_to_filters() does not return lzma_ret so we should
be more clear.
|
|
This reverts commit 82e3c968bfa10e3ff13333bd9cbbadb5988d6766.
Macros in the reserved namespace (_foo or __foo) shouldn't be #defined
without a very good reason. Here the alternative would have been
to #define tuklib_has_warning(str) to an approriate value.
Also the tuklib_* files should stay namespace clean if possible.
|
|
__has_warning and other __has_foo macros are meant to become
compiler-agnostic so it's not good to check for __clang__ with it.
This also relied on tuklib_common.h for #defining __has_warning
which was confusing as #defining reserved macros is generally
not a good idea.
|
|
Also edit style to match the existing coding style in the project.
|
|
|
|
This prevents the reserved fields from being part of the generated
Doxygen documentation.
|
|
A few Doxygen tags were obsolete from 1.4.7. Version 1.8.17 released
in 2019, so this should be compatible with resonable modern distros.
The purpose of Doxygen these days is for docs on the website, so it
doesn't necessarily have to work for everyone. Just when the maintainers
want to update the docs.
|
|
Doxygen is now configurable in autotools only with
--enable-doxygen=[api|all]. The default is "api", which will only
generate HTML output for liblzma API functions. The LaTex documentation
output was also disabled.
|
|
This improves the generated Doxygen HTML files to better highlight
how to properly use the liblzma API header files.
|
|
tuklib_physmem depends on GetProcAddress() for both MSVC and MinGW-w64
to retrieve a function address. The proper way to do this is to cast the
return value to the type of function pointer retrieved. Unfortunately,
this causes a cast-function-type warning, so the best solution is to
simply ignore the warning.
|
|
clang supports the __has_warning macro to determine if the version of
clang compiling the code supports a given warning. If we do not define
it for other compilers, it may cause a preprocessor error.
|
|
The 32-bit build needs to be first so the configure cache only needs to
be reset one time. The 32-bit build sets the CFLAGS env variable, so any
build using that flag after will fail unless the cache is reset.
|
|
If CFLAGS are set in a build, the cache must be cleared with
"make distclean", or by deleting the cache file.
|
|
|
|
Calling coder_set_compression_settings() in list mode with verbose mode
on caused the filter chain and memory requirements to print. This was
unnecessary since the command results in an error and not consistent
with other formats like lzma and alone.
|
|
|
|
Disabling shared library generation and linking should help speed up the
runners. The shared library is still being tested in the 32 bit build
and the full feature.
Disabling nls is to check for any unexpected warnings or errors.
|
|
Run the 32 bit job sooner since this is a more interesting test than
some of the later jobs.
|
|
|
|
|
|
|
|
|
|
It's not that important. It can be annoying in builds that
disable many features since in those cases the tests programs
will correctly trigger this warning with Clang.
|
|
It makes no difference here as the return value fits into an int
too and it then gets ignored but this looks better.
|
|
|
|
It doesn't warn on a 64-bit system because truncating
a ptrdiff_t (signed long) to uint32_t is diagnosed under
-Wconversion by GCC and -Wshorten-64-to-32 by Clang.
|
|
|
|
-Wstrict-aliasing was removed from the list since it is enabled
by -Wall already.
A normal build is clean with these on GNU/Linux x86-64 with
GCC 12.2.0 and Clang 14.0.6.
|
|
Explicitly casting the integer to lzma_check silences the warning.
Since such an invalid value is needed in multiple tests, a constant
INVALID_LZMA_CHECK_ID was added to tests.h.
The use of 0x1000 for lzma_block.check wasn't optimal as if
the underlying type is a char then 0x1000 will be truncated to 0.
However, in these test cases the value is ignored, thus even with
such truncation the test would have passed.
|
|
Note that assigning an unsigned int to lzma_check doesn't warn
on GNU/Linux x86-64 since the enum type is unsigned on that
platform. The enum can be signed on some other platform though
so it's best to use enumeration type lzma_check in these situations.
|
|
This is similar to 2ce4f36f179a81d0c6e182a409f363df759d1ad0.
The actual initialization of the variables is done inside
mythread_sync() macro. Clang doesn't seem to see that
the initialization code inside the macro is always executed.
|
|
|
|
|
|
|
|
|
|
clang and gcc differ in how they handle -Wformat-nonliteral. gcc will
allow a non-literal format string as long as the function takes its
format arguments as a va_list.
|
|
|
|
This only occurs in test_filter_flags when the BCJ filters are not
configured and built. In this case, ARRAY_SIZE() returns 0 and causes a
type-limits warning with the loop variable since an unsigned number will
always be >= 0.
|
|
This affects only 32-bit x86 builds. x86-64 is OK as is.
I still cannot easily test this myself. The reporter has tested
this and it passes the tests included in the CMake build and
performance is good: raw CRC64 is 2-3 times faster than the
C version of the slice-by-four method. (Note that liblzma doesn't
include a MSVC-compatible version of the 32-bit x86 assembly code
for the slice-by-four method.)
Thanks to Iouri Kharon for figuring out a fix, testing, and
benchmarking.
|
|
One of the global arrays of filters was only used in a test that
required both encoders and decoders to be configured in the build.
|
|
test_index_hash does not use fill_index_hash() unless both encoders
and decoders are configured in the build.
|
|
If all goes well, Mac autotools and Linux and Mac CMake will be added
later for 32-bit builds.
|
|
This will help us catch warnings and potential bugs in builds that are
not often tested by us.
|
|
For now, the suggested option is for -m32 only, but this can be updated
later if other flags are deemed useful.
|
|
This reverts commit 36edc65ab4cf10a131f239acbd423b4510ba52d5.
It was reported that it wasn't a good enough fix and MSVC
still produced (different kind of) bad code when building
for 32-bit x86 if optimizations are enabled.
Thanks to Iouri Kharon.
|
|
On some platforms src/xz/suffix.c may need <strings.h> for
strcasecmp() but suffix.c includes the header when it needs it.
Unless there is an old system that otherwise supports enough C99
to build XZ Utils but doesn't have C89/C90-compatible <string.h>,
there should be no need to include <strings.h> in sysdefs.h.
|
|
SUSv2 and POSIX.1‐2017 declare only a few functions in <strings.h>.
Of these, strcasecmp() is used on some platforms in suffix.c.
Nothing else in the project needs <strings.h> (at least if
building on a modern system).
sysdefs.h currently includes <strings.h> if HAVE_STRINGS_H is
defined and suffix.c relied on this.
Note that dos/config.h doesn't #define HAVE_STRINGS_H even though
DJGPP does have strings.h. It isn't needed with DJGPP as strcasecmp()
is also in <string.h> in DJGPP.
|
|
|
|
It quite probably was never needed, that is, any system where memory.h
was required likely couldn't compile XZ Utils for other reasons anyway.
XZ Utils 5.2.6 and later source packages were generated using
Autoconf 2.71 which no longer defines HAVE_MEMORY_H. So the code
being removed is no longer used anyway.
|
|
It's a string, not a list. It only worked when the variable was empty.
Thanks to Iouri Kharon.
|
|
|
|
At least on some systems, GNU windres needs --use-temp-file
in addition to the \x20 hack to avoid spaces in the command line
argument. Hovever, that \x20 syntax is broken with llvm-windres
version 15.0.0 (results in "XZx20Utils") but luckily it works
with a regular space. Thus it is best to limit the workarounds
to GNU toolchain on Windows.
|
|
Here are the list of the most significant issues addressed:
- Avoid using internal common.h header. It's not good to copy the
constants like this but common.h cannot be included for use outside
of liblzma. This is the quickest thing to do that could be fixed later.
- Omit the INIT_FILTER macro. Initialization should be done with just
regular designated initializers.
- Use start_offset = 257 for BCJ tests. It demonstrates that Filter
Flags encoder and decoder don't validate the options thoroughly.
257 is valid only for the x86 filter. This is a bit silly but
not a significant problem in practice because the encoder and
decoder initialization functions will catch bad alignment still.
Perhaps this should be fixed but it's not urgent and doesn't need
to be in 5.4.x.
- Various tweaks to comments such as filter id -> Filter ID
|
|
Converts the existing filter flags tests into tuktests.
|
|
I haven't tested with MSVC myself and there doesn't seem to be
information about the problem online, so I'm relying on the bug report.
Thanks to Iouri Kharon for the bug report and the patch.
|
|
It was my mistake. Thanks to Iouri Kharon for the bug report.
|
|
It's not needed in XZ Utils at least for now. It's good to support
it still because if such use is needed later, it wouldn't be
caught on GNU/Linux since malloc(0) from glibc returns non-NULL.
|
|
|
|
The changes listed on cmake-policies(7) for versions 3.17 to 3.25
shouldn't affect this project.
|
|
|
|
The command line tools cannot be built with MSVC for now but
they can be built with MinGW-w64.
Thanks to Iouri Kharon for the bug report and the original patch.
|
|
Thanks to Iouri Kharon for the bug report and the original patch.
|
|
VS2013 doesn't have _mm_set_epi64x() so this way CLMUL gets
disabled with VS2013.
Thanks to Iouri Kharon for the bug report.
|
|
The phase split was only done for Autotools before, so should also
apply to CMake.
|
|
The old version used too many runners that resulted in unnecessary
dependency downloads. Now, the runners are reused for the different
configurations for each OS and build system.
|
|
The new PHASE argument can be build, test, or all. all is the default.
This way, the CI/CD script can differentiate between the build and test
phases to make it easier to track down errors when they happen.
|
|
Tuktest index hash
|
|
|
|
|
|
|
|
|
|
It can trigger warnings from -Wshadow on some systems.
|
|
|
|
The words defined in the .xz file format specification
begin with capital letter to emphasize that they have
a specific meaning.
|
|
|
|
|
|
The line in the .vcxproj files for building with was missing in 5.4.0.
Thank to Hajin Jang for reporting the issue.
|
|
common/index.h is needed by liblzma internally and tests. common.h will
include and define many things that are not needed by the tests. Also,
this prevents include order problems because common.h will redefine
LZMA_API resulting in a warning.
|
|
|
|
|
|
The shell parameter expansion using # and ## is not supported in
Solaris 10 Bourne shell (/bin/sh). Even though this is POSIX, it is not fully
portable, so we should avoid it.
|
|
Thanks to Seong-ho Cho
|
|
|
|
5.5.0alpha won't be released, it's just to mark that
the branch is not for stable 5.4.x.
Once again there is no API/ABI stability for new features
in devel versions. The major soname won't be bumped even
if API/ABI of new features breaks between devel releases.
|
|
|
|
HAVE_DECL_PROGRAM_INVOCATION_NAME is renamed to
HAVE_PROGRAM_INVOCATION_NAME. Previously,
HAVE_DECL_PROGRAM_INVOCATION_NAME was always set when
building with autotools. CMake would only set this when it was 1, and the
dos/config.h did not define it. The new macro definition is consistent
across build systems.
|
|
|
|
Tests all API functions exported from index_hash.h. Does not have a
dedicated test for lzma_index_hash_end.
|
|
This is for consistency with lzma_index_append.
|
|
|
|
|
|
|
|
Previously, mytime.c depended on mythread.h for <time.h> to be included.
|
|
Previously, <sys/time.h> was always included, even if mythread only used
clock_gettime. <time.h> is still needed even if clock_gettime is not used
though because struct timespec is needed for mythread_condtime.
|
|
Previously, if threading was enabled HAVE_DECL_CLOCK_MONOTONIC would always
be set to 0 or 1. However, this macro was needed in xz so if xz was not
built with threading and HAVE_DECL_CLOCK_MONOTONIC was not defined but
HAVE_CLOCK_GETTIME was, it caused a warning during build. Now,
HAVE_DECL_CLOCK_MONOTONIC has been renamed to HAVE_CLOCK_MONOTONIC and
will only be set if it is 1.
|
|
Thanks to Yuri Chornoivan
|
|
The CI/CD workflow will only execute on Ubuntu and MacOS latest version.
The workflow will attempt to build with autotools and CMake and execute
the tests. The workflow will run for all pull requests and pushes done
to the master branch.
|
|
|
|
|
|
In source builds are not recommended, but we should still ignore
the generated artifacts.
|
|
Using return_if_error on lzma_lzma_lclppb_encode was improper because
return_if_error is expecting an lzma_ret value, but
lzma_lzma_lclppb_encode returns a boolean. This could result in
lzma_microlzma_encoder, which would be misleading for applications.
|
|
In source builds are not recommended, but we can make it easier
by ignoring the generated artifacts from CMake.
|
|
|
|
|
|
Using CMake to build liblzma should work on a few other OSes
but building the command line tools is still subtly broken.
It is known that shared library versioning may differ between
CMake and Libtool builds on some OSes, most notably Darwin.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The TODO file outdated still.
|
|
|
|
|
|
|
|
|
|
The code that parses --memlimit options and --block-list modified
the argv[] when parsing the option string from optarg. This was
visible in "ps auxf" and such and could be confusing. I didn't
understand it back in the day when I wrote that code. Now a copy
is allocated when modifiable strings are needed.
|
|
|
|
|
|
|
|
|
|
Thanks to Remus-Gabriel Chelu.
|
|
|
|
The API docs gave an impression that such checks are done
but they actually weren't done. In practice it made little
difference since the calling code has a bug if these are NULL.
Thanks to Jia Tan for the original patch that checked for
block->filters == NULL.
|
|
This also sorts the symbol names alphabetically in liblzma_*.map.
|
|
|
|
|
|
If someone sets up Clang to define __GNUC__ to 10 or greater
then symvers broke. __has_attribute is supported by such GCC
and Clang versions that don't support __symver__ so this should
be much better and simpler way to detect if __symver__ is
actually supported.
Thanks to Tomasz Gajc for the bug report.
|
