| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
While the backdoor was inactive (and thus harmless) without inserting
a small trigger code into the build system when the source package was
created, it's good to remove this anyway:
- The executable payloads were embedded as binary blobs in
the test files. This was a blatant violation of the
Debian Free Software Guidelines.
- On machines that see lots bots poking at the SSH port, the backdoor
noticeably increased CPU load, resulting in degraded user experience
and thus overwhelmingly negative user feedback.
- The maintainer who added the backdoor has disappeared.
- Backdoors are bad for security.
This reverts the following without making any other changes:
6e636819 Tests: Update two test files.
a3a29bbd Tests: Test --single-stream can decompress bad-3-corrupt_lzma2.xz.
0b4ccc91 Tests: Update RISC-V test files.
8c9b8b20 liblzma: Fix typos in crc32_fast.c and crc64_fast.c.
82ecc538 liblzma: Fix false Valgrind error report with GCC.
cf44e4b7 Tests: Add a few test files.
3060e107 Tests: Use smaller dictionary size in RISC-V test files.
e2870db5 Tests: Add two RISC-V Filter test files.
The RISC-V test files also have real content that tests the filter
but the real content would fit into much smaller files. A generator
program would need to be available as well.
Thanks to Andres Freund for finding and reporting it and making
it public quickly so others could act without a delay.
See: https://www.openwall.com/lists/oss-security/2024/03/29/4
|
|
|
|
|
|
|
|
The original files were generated with random local to my machine.
To better reproduce these files in the future, a constant seed was used
to recreate these files.
|
|
The first stream in this file is valid, so this tests that xz properly
stops after decompressing it.
|
|
This increases code coverage and tests for possible shifting bugs.
|
|
|
|
This reverts commit adaacafde6661496ca2814b1e94a3ba5186428cb.
|
|
With GCC and a certain combination of flags, Valgrind will falsely
trigger an invalid write. This appears to be due to the omission of
instructions to properly save, set up, and restore the frame pointer.
The IFUNC resolver is a leaf function since it only calls a function
that is inlined. So sometimes GCC omits the frame pointer instructions
in the resolver unless this optimization is explictly disabled.
This fixes https://bugzilla.redhat.com/show_bug.cgi?id=2267598.
|
|
|
|
Thanks to Sam James for determining this was the attribute needed to
workaround the GCC bug and for his version of the patch in Gentoo.
|
|
Using __attribute__((__no_profile_instrument_function__)) on the ifunc
resolver works around a bug in GCC -fprofile-generate:
it adds profiling code even to ifunc resolvers which can make
the ifunc resolver crash at program startup. This attribute
was not introduced until GCC 7 and Clang 13, so ifunc won't
be used with prior versions of these compilers.
This bug was brought to our attention by:
https://bugs.gentoo.org/925415
And was reported to upstream GCC by:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411
|
|
|
|
|
|
It's only done with CMake >= 3.20 and if library support
for translation was already found.
Sort of fixes: https://github.com/tukaani-project/xz/issues/82
|
|
|
|
Now that multi threaded encoding is the default, users do not need to
see a warning message everytime the number of threads is reduced. On
some machines, this could happen very often. It is not unreasonable for
users to need to set double verbose mode to see this kind of
information.
To see these warning messages -vv or --verbose --verbose must be passed
to set xz into the highest possible verbosity mode.
These warnings had caused automated testing frameworks to fail when they
expected no output to stderr.
Thanks to Sebastian Andrzej Siewior for reporting this and for the
initial version of the patch.
|
|
|
|
|
|
Signed-off-by: Chien Wong <m@xv97.com>
|
|
The previous Linux Landlock feature test assumed that having the
linux/landlock.h header file was enough. The new feature tests also
requires that prctl() and the required Landlock system calls are
supported.
|
|
|
|
|
|
|
|
|
|
|
|
The French and Brazilian Portuguese man page translations have not been
updated since the switch from public domain to 0BSD. The old GPLv2
strings have now been removed from these files.
|
|
This will be fixed in the next round of translations, but this avoids
having a fuzzy match or not fixing the English version.
|
|
Thanks to Yuri Chornoivan.
|
|
|
|
|
|
|
|
|
|
|
|
user_abort_pipe[] was still being used instead of the parameters.
|
|
Old versions of Clang reported the unsupported function attribute and
__crc32d() function as warnings instead of errors, so the feature test
passed when it shouldn't have, causing a compile error at build time.
-Werror was added to this feature test to fix this. The change is not
needed for CMake because check_c_source_compiles() also performs
linking and the error is caught then.
Thanks to Sebastian Andrzej Siewior for reporting this.
|
|
LOCALEDIR may contain spaces like in "C:\Program Files".
|
|
If xz is given a directory, it should look like this:
$ xz /usr/bin
xz: /usr/bin: Is a directory, skipping
The Landlock rules didn't allow opening directories for reading:
$ xz /usr/bin
xz: /usr/bin: Permission denied
The simplest fix was to allow opening directories for reading.
While it's a bit silly to allow it solely for the error message,
it shouldn't make the sandbox significantly weaker.
The single-file use case (like when called from GNU tar) is
still as strict as possible: all Landlock restrictions are
enabled before (de)compression starts.
|
|
Thanks to Sebastian Andrzej Siewior and Sam James for
benchmarking on various systems.
|
|
|
|
|
|
|
|
|
|
It's for binary packages built with windows/build.bash.
|
|
Support for the old MinGW was dropped. Only MinGW-w64 with GCC
is supported now.
The script now supports also cross-compilation from GNU/Linux
(tests are not run). MSYS2 and also the old MSYS 1.0.11 work
for building on Windows. The i686 and x86_64 toolchains must
be in PATH to build both 32-bit and 64-bit versions.
Parallel builds are done if "nproc" from GNU coreutils is available.
MinGW-w64 runtime copyright information file was renamed from
COPYING-Windows.txt to COPYING.MinGW-w64-runtime.txt which
is the filename used by MinGW-w64 itself. Its existence
is now mandatory, it's checked at the beginning of the script.
The file TODO is no longer copied to the package.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The original code was good enough for supporting GNU/Linux
and a few others but it wasn't very portable.
CMake doesn't support Solaris Studio's -xldscope=hidden.
If it ever does, things should still work with this commit
as Solaris Studio supports not only its own __global but also
the GNU C __attribute__((visibility("default"))). Support for the
attribute was added in 2007 to Sun Studio 12 compiler version 5.9.
|
|
|
|
|
|
-O3 doesn't seem useful for speed but it makes the code bigger.
CMake makes is difficult for users to simply override the
optimization level: CFLAGS / CMAKE_C_FLAGS aren't helpful because
they go before CMAKE_C_FLAGS_RELEASE. Of course, users can override
CMAKE_C_FLAGS_RELEASE directly but then they have to remember to
add also -DNDEBUG to disable assertions.
This commit changes -O3 to -O2 in CMAKE_C_FLAGS_RELEASE if and only if
CMAKE_C_FLAGS_RELEASE cache variable doesn't already exist. So if
a custom value is passed on the command line (or reconfiguring an
already-configured build), the cache variable won't be modified.
|
|
This is to match configure.ac.
|
|
|
|
|
|
CMP0154 doesn't affect us since we don't use FILE_SET.
|
|
|
|
|
|
In contrast to Automake, skipping of this test when decoders
are disabled is handled at CMake side instead of test_scripts.sh
because CMake-build doesn't create config.h.
|
|
Compared to the Autotools-based build, this has simpler handling
for the shell (@POSIX_SHELL@) and extra PATH entry for the scripts
(configure has --enable-path-for-scripts=PREFIX). The simpler
metho should be enough for non-ancient systems and Solaris.
|
|
PACKAGE_VERSION was already used in liblzma.pc.in.
This way only one version @foo@ is used.
|
|
It helps that cmake_install.cmake doesn't parallelize installation
so symlinks can be created so that the target is always known to
exist (a requirement on Windows in some cases).
This bumps the minimum CMake version from 3.13 to 3.14 to use
file(CREATE_LINK ...). It could be made to work on 3.13 by
calling "cmake -E create_symlink" but it's uglier code and
slower in "make install". 3.14 should be a reasonable version
to require nowadays, especially since the Autotools build
is still the primary build system for most OSes.
|
|
If gettext tools are available, the .po files listed in po/LINGUAS
are converted using msgfmt. This allows building with translations
directly from xz.git without Autotools.
If gettext tools aren't available, the Autotools-created .gmo files
in the "po" directory will be used. This allows CMake-based build
to use translations from Autotools-generated tarball.
If translation support is found (Intl_FOUND) but both the
gettext tools and the pre-generated .gmo files are missing,
then "make" will fail.
|
|
|
|
|
|
This makes these sandboxing methods stricter when no files are
created or deleted. That is, it's a middle ground between the
initial sandbox and the strictest single-file-to-stdout sandbox:
this allows opening files for reading but output has to go to stdout.
|
|
Linux 6.7 added support for ABI version 4 which restricts
TCP connections which xz won't need and thus those can be
forbidden now. Since the ABI version is handled at runtime,
supporting version 4 won't cause any compatibility issues.
Note that new enough kernel headers are required to get
version 4 support enabled at build time.
|
|
Landlock is now always used just like pledge(2) is: first in more
permissive mode and later (under certain common conditions) in
a strict mode that doesn't allow opening more files.
I put pledge(2) first in sandbox.c because it's the simplest API
to use and still somewhat fine-grained for basic applications.
So it's the simplest thing to understand for anyone reading sandbox.c.
|
|
|
|
Also explicitly initialize progress_automatic to make it clear
that it can be read before message_init() sets it. Static variable
was initialized to false by default already so this is only for
clarity.
|
|
Dirs first, then files in case-sensitive ASCII order.
|
|
|
|
|
|
|
|
All other translated man pages were being installed but
lzmainfo had been forgotten.
|
|
GCC docs promise that it works and a few other compilers do
too. Clang/LLVM is documented source code only but unsurprisingly
it behaves the same as others on x86-64 at least. But the
certainly-portable way is good enough here so use that.
|
|
|
|
The x32 port has a x86-64 ABI in term of all registers but uses only
32bit pointer like x86-32. The assembly optimisation fails to compile on
x32. Given the state of x32 I suggest to exclude it from the
optimisation rather than trying to fix it.
Signed-off-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Don't assume that Clang defines __GNUC__ as the extensions
are available in clang-cl as well (and possibly in some other
Clang variants?).
|
|
|
|
|
|
|
|
Adding the SPDX license identifier changed the line numbers.
|
|
|
|
|
|
|
|
|
|
|
|
It was added in 2017 in c2e29f06a7d1e3ba242ac2fafc69f5d6e92f62cd
but it never got into any release tarballs because it was
forgotten to be added to Makefile.am.
|
|
|
|
|
|
This kind of fixes the problem reported here:
https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/1291020
|
|
|
|
|
|
|
|
|
|
It's compatible with GCC and Clang.
|
|
It's used only for basic bittrees and fixed-size reverse bittree
because those showed a clear benefit on x86-64 with GCC and Clang.
The other methods were more mixed and thus are commented out but
they should be tested on other archs.
|
|
|
|
But now it needs one more local variable.
|
|
|
|
|
|
|
|
Now extra buffer space is reserved so that repeating bytes for
any single match will never need to copy from two places (both
the beginning and the end of the buffer). This simplifies
dict_repeat() and helps a little with speed.
This seems to reduce .lzma decompression time about 2 %, so
with .xz and CRC it could be slightly less. The small things
add up still.
|
|
It's not completely obvious if this is better in the decoder.
It should be good if compiler can avoid creating a branch
(like using CMOV on x86).
This also makes lzma_encoder.c use the new macros.
|
|
This adds macros for bittree decoding which prepares the code
for alternative C versions and inline assembly.
|
|
The new decoder resumes the first decoder loop in the Resumable mode.
Then, the code executes in Non-resumable mode until it detects that it
cannot guarantee to have enough input/output to decode another symbol.
The Resumable mode is how the decoder has always worked. Before decoding
every input bit, it checks if there is enough space and will save its
location to be resumed later. When the decoder has more input/output,
it jumps back to the correct sequence in the Resumable mode code.
When the input/output buffers are large, the Resumable mode is much
slower than the Non-resumable because it has more branches and is harder
for the compiler to optimize since it is in a large switch block.
Early benchmarking shows significant time improvement (8-10% on gcc and
clang x86) by using the Non-resumable code as much as possible.
|
|
The new "safe" range decoder mode is the same as old range decoder, but
now the default behavior of the range decoder will not check if there is
enough input or output to complete the operation. When the buffers are
close to fully consumed, the "safe" operations must be used instead. This
will improve speed because it will reduce the number of branches needed
for most of the range decoder operations.
|
|
The footer template from Doxygen has the closing </body> </html>
as Doxygen doesn't add them otherwise.
target="_blank" was omitted as it's not useful here but
it can be slightly annoying as one cannot just go back
in the browser history.
Since the footer links to the license file in the same
directory and not to CC website, the rel attributes
can be omitted.
|
|
|
|
It looked odd to only have the original English authors listed
in the header comments of the translated files.
|
|
lzmainfo has had translation support since 2009 at least but
it was never added to po/POTFILES.in so the messages weren't
translated. It's a very rarely needed tool so it's not too bad.
This also adds src/xz/mytime.c to po/POTFILES.in although there
are no translatable strings. It's simpler this way so that it
won't be forgotten if strings were ever added to that file.
|
|
The same is used for both po/xz.pot and po4a/xz-man.pot.
|
|
All man pages are under 0BSD now so this is simple now.
|
|
|
|
The main reason is a kind of silly one:
xz-man.pot contains strings from all man pages in XZ Utils.
The man pages of xzdiff, xzgrep, and xzmore were under GPLv2
and the rest under 0BSD. Thus xz-man.pot contained strings
under two licences. po4a creates the translated man pages
from the combined 0BSD+GPLv2 xz-man.pot.
I haven't liked this mixing in xz-man.pot but the
Translation Project requires that all man pages must be
in the same .pot file. So a separate xz-man-gpl.pot
wasn't an option.
Since these man pages are short, rewriting them was quick enough.
Now xz-man.pot is entirely under 0BSD and marking the per-file
licenses is simpler.
As a bonus, some wording hopefully is now slightly better
although it's perhaps a matter of taste.
NOTE: In xzgrep.1, the EXIT STATUS section was written by me
in the commit d796b6d7fdb8b7238b277056cf9146cce25db604 so that's
why that section could be taken as is from the old xzgrep.1.
|
|
The xz tool can decompress three file formats and xzless
has always supported uncompressed files too.
|
|
|
|
Also add SPDX license identifier now that there is a known license.
|
|
Perhaps the generated files aren't even copyrightable but
using the same license for them as for the rest of the liblzma
keeps things more consistent for tools that look for license info.
|
|
It is built and run only manually so this didn't matter
unless one wanted to regenerate the price_table.c.
|
|
|
|
|
|
|
|
Translations and doc/xz-file-format.txt and doc/lzma-file-format.txt
were not touched.
COPYING.0BSD was added.
|
|
The initial commit 5d018dc03549c1ee4958364712fb0c94e1bf2741
in 2007 had a comment in sha256.c that the code is based on
Crypto++ Library 5.5.1. In 2009 the Authors list in sha256.c
and the AUTHORS file was updated with information that the
code had come from Crypto++ but via 7-Zip. I know I had viewed
7-Zip's SHA-256 code but back then the C code has been identical
enough with Crypto++, so I don't why I thought the author info
would need that extra step via 7-Zip for this single file.
Another error is that I had mixed sha.* and shacal2.* files
when checking for author info in Crypto++. The shacal2.* files
aren't related to liblzma's sha256.c and thus Kevin Springle's
code in Crypto++ isn't either.
|
|
It was last updated in 2013.
|
|
It was good to keep these around in parallel with the newer examples
but I think it's OK to remove the old ones at this point.
|
|
|
|
If any other BCJ filter was enabled for encoding or decoding, then this
was not a problem.
|
|
|
|
|
|
|
|
|
|
This makes "less" show a warning if a decompression error occurred.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This was split from the prior commit so it could be easily applied to
the 5.4 branch.
Closes: https://github.com/tukaani-project/xz/pull/77
|
|
If liblzma is configured with --disable-clmul-crc
CFLAGS="-msse4.1 -mpclmul", then it will fail to compile because the
generic version must be used but the CRC tables were not included.
|
|
The code was using HAVE_FUNC_ATTRIBUTE_IFUNC instead of CRC_USE_IFUNC.
With ARM64, ifunc is incompatible because it requires non-inline
function calls for runtime detection.
|
|
|
|
This is similar to the existing x86-64 CLMUL conditions to omit the
tables. They were slightly refactored to improve readability.
|
|
Even though the proper name for the architecture is aarch64, this
project uses ARM64 throughout. So the rename is for consistency.
Additionally, crc32_arm64.h was slightly refactored for the following
changes:
* Added MSVC, FreeBSD, and macOS support in
is_arch_extension_supported().
* crc32_arch_optimized() now checks the size when aligning the
buffer.
* crc32_arch_optimized() loop conditions were slightly modified to
avoid both decrementing the size and incrementing the buffer
pointer.
* Use the intrinsic wrappers defined in <arm_acle.h> because GCC and
Clang name them differently.
* Minor spacing and comment changes.
|
|
The CRC_GENERIC is now split into CRC32_GENERIC and CRC64_GENERIC, since
the ARM64 optimizations will be different between CRC32 and CRC64.
For the same reason, CRC_ARCH_OPTIMIZED is split into
CRC32_ARCH_OPTIMIZED and CRC64_ARCH_OPTIMIZED.
ifunc will only be used with x86-64 CLMUL because the runtime detection
methods needed with ARM64 are not compatible with ifunc.
|
|
|
|
This adds --enable-arm64-crc32/--disable-arm64-crc32 (enabled by
default) for using the ARM64 CRC32 instruction. This can be disabled if
one knows the binary will never need to run on an ARM64 machine
with this instruction extension.
|
|
The CRC32 instructions in ARM64 can calculate the CRC32 result
for 8 bytes in a single operation, making the use of ARM64
instructions much faster compared to the general CRC32 algorithm.
Optimized CRC32 will be enabled if ARM64 has CRC extension
running on Linux.
Signed-off-by: Chenxi Mao <chenxi.mao2013@gmail.com>
|
|
|
|
|
|
|
|
The footer isn't a complete HTML file so having it in the doxygen
directory is a tiny bit clearer.
|
|
The Overall documentation section (1.1) table spacing had to be adjusted
since the filename was very long.
|
|
|
|
|
|
The PROJECT_LOGO field is now used to include the XZ logo. The footer
of each page now lists the copyright information instead of the default
footer. The license is also copied to statisfy the copyright and so the
link in the documentation can be local.
|
|
This hopefully does more good than bad:
+ It's faster by default.
+ Only the threaded compressor creates files that
can be decompressed in threaded mode.
- Compression ratio is worse, usually not too much though.
When it matters, -T1 must be used.
- Memory usage increases.
- Scripts that assume single-threaded mode but don't use -T1 will
possibly use too much resources, for example, if they run
multiple xz processes in parallel to compress multiple files.
- Output from single-threaded and multi-threaded compressors
differ but such changes could happen for other reasons too
(they just haven't happened since 5.0.0).
|
|
|
|
|
|
|
|
|
|
Not all RISC-V processors support fast unaligned access so
it's better to read only one byte in the main loop. This can
be faster even on x86-64 when compared to reading 32 bits at
a time as half the time the address is only 16-bit aligned.
The downside is larger code size on archs that do support
fast unaligned access.
|
|
Version 5.6.0 will be shown, even though upcoming alphas and betas
will be able to support this filter. 5.6.0 looks nicer in the output and
people shouldn't be encouraged to use an unstable version in production
in any way.
|
|
These test files achieve 100% code coverage in
src/liblzma/simple/riscv.c. They contain all of the instructions that
should be filtered and a few cases that should not.
|
|
|
|
A special note was added to suggest using four-byte alignment when the
compressed instruction extension is not present in a RISC-V binary.
|
|
|
|
|
|
|
|
The new Filter ID is 0x0B.
Thanks to Chien Wong <m@xv97.com> for the initial version of the Filter,
the xz CLI updates, and the Autotools build system modifications.
Thanks to Igor Pavlov for his many contributions to the design of
the filter.
|
|
The new RISC-V filter was added to the specification, in addition to
updating the specification URL.
|
|
|
