aboutsummaryrefslogtreecommitdiff
path: root/src/xz/hardware.c (follow)
AgeCommit message (Collapse)AuthorFilesLines
2024-01-23xz: Use threaded mode by defaut (as if --threads=0 was used).Lasse Collin1-1/+5
This hopefully does more good than bad: + It's faster by default. + Only the threaded compressor creates files that can be decompressed in threaded mode. - Compression ratio is worse, usually not too much though. When it matters, -T1 must be used. - Memory usage increases. - Scripts that assume single-threaded mode but don't use -T1 will possibly use too much resources, for example, if they run multiple xz processes in parallel to compress multiple files. - Output from single-threaded and multi-threaded compressors differ but such changes could happen for other reasons too (they just haven't happened since 5.0.0).
2023-07-31Docs: Fix typos found by codespellDimitri Papadopoulos Orfanos1-2/+2
2022-11-19xz: Refactor duplicate code from hardware_memlimit_mtenc_get().Lasse Collin1-1/+1
2022-11-19xz: Add support --threads=+N so that -T+1 gives threaded mode.Lasse Collin1-2/+15
2022-11-07xz: Extend --robot --info-memory output.Lasse Collin1-9/+15
Now it includes everything that the human-readable --info-memory shows.
2022-08-19xz: Revise --info-memory output.Lasse Collin1-6/+26
The strings could be more descriptive but it's good to have some version of this committed now. --robot mode wasn't changed yet.
2022-04-14xz: Fix build with --disable-threads.Lasse Collin1-0/+4
2022-04-14xz: Change the cap of the default -T0 memlimit for 32-bit xz.Lasse Collin1-1/+3
The SIZE_MAX / 3 was 1365 MiB. 1400 MiB gives little more room and it looks like a round (artificial) number in --info-memory once --info-memory is made to display it. Also, using #if avoids useless code on 64-bit builds.
2022-04-14xz: Add a default soft memory usage limit for --threads=0.Lasse Collin1-9/+29
This is a soft limit in sense that it only affects the number of threads. It never makes xz fail and it never makes xz change settings that would affect the compressed output. The idea is to make -T0 have more reasonable behavior when the system has very many cores or when a memory-hungry compression options are used. This also helps with 32-bit xz, preventing it from running out of address space. The downside of this commit is that now the number of threads might become too low compared to what the user expected. I hope this to be an acceptable compromise as the old behavior has been a source of well-argued complaints for a long time.
2022-04-14xz: Make -T0 use multithreaded mode on single-core systems.Lasse Collin1-0/+14
The main problem withi the old behavior is that the compressed output is different on single-core systems vs. multicore systems. This commit fixes it by making -T0 one thread in multithreaded mode on single-core systems. The downside of this is that it uses more memory. However, if --memlimit-compress is used, xz can (thanks to the previous commit) drop to the single-threaded mode still.
2022-04-12xz: Add --memlimit-mt-decompress along with a default limit value.Lasse Collin1-5/+55
--memlimit-mt-decompress allows specifying the limit for multithreaded decompression. This matches memlimit_threading in liblzma. This limit can only affect the number of threads being used; it will never prevent xz from decompressing a file. The old --memlimit-decompress option is still used at the same time. If the value of --memlimit-decompress (the default value or one specified by the user) is less than the value of --memlimit-mt-decompress , then --memlimit-mt-decompress is reduced to match --memlimit-decompress. Man page wasn't updated yet.
2021-04-11Reduce maximum possible memory limit on MIPS32Ivan A. Melnikov1-0/+6
Due to architectural limitations, address space available to a single userspace process on MIPS32 is limited to 2 GiB, not 4, even on systems that have more physical RAM -- e.g. 64-bit systems with 32-bit userspace, or systems that use XPA (an extension similar to x86's PAE). So, for MIPS32, we have to impose stronger memory limits. I've chosen 2000MiB to give the process some headroom.
2020-02-01xz: Limit --memlimit-compress to at most 4020 MiB for 32-bit xz.Lasse Collin1-1/+31
See the code comment for reasoning. It's far from perfect but hopefully good enough for certain cases while hopefully doing nothing bad in other situations. At presets -5 ... -9, 4020 MiB vs. 4096 MiB makes no difference on how xz scales down the number of threads. The limit has to be a few MiB below 4096 MiB because otherwise things like "xz --lzma2=dict=500MiB" won't scale down the dict size enough and xz cannot allocate enough memory. With "ulimit -v $((4096 * 1024))" on x86-64, the limit in xz had to be no more than 4085 MiB. Some safety margin is good though. This is hack but it should be useful when running 32-bit xz on a 64-bit kernel that gives full 4 GiB address space to xz. Hopefully this is enough to solve this: https://bugzilla.redhat.com/show_bug.cgi?id=1196786 FreeBSD has a patch that limits the result in tuklib_physmem() to SIZE_MAX on 32-bit systems. While I think it's not the way to do it, the results on --memlimit-compress have been good. This commit should achieve practically identical results for compression while leaving decompression and tuklib_physmem() and thus lzma_physmem() unaffected.
2019-06-28xz: Automatically align the strings in --info-memory.Lasse Collin1-11/+34
This makes it easier to translate the strings. Also, the string for amount of RAM was shortened.
2014-06-18xz: Use lzma_cputhreads() instead of own copy of tuklib_cpucores().Lasse Collin1-3/+9
2011-04-09xz: Minor internal changes to handling of --threads.Lasse Collin1-12/+12
Now it always defaults to one thread. Maybe this will change again if a threading method is added that doesn't affect memory usage.
2010-09-07xz: Add a note to translators.Lasse Collin1-0/+2
2010-08-07Disable the memory usage limiter by default.Lasse Collin1-33/+63
For several people, the limiter causes bigger problems that it solves, so it is better to have it disabled by default. Those who want to have a limiter by default need to enable it via the environment variable XZ_DEFAULTS. Support for environment variable XZ_DEFAULTS was added. It is parsed before XZ_OPT and technically identical with it. The intended uses differ quite a bit though; see the man page. The memory usage limit can now be set separately for compression and decompression using --memlimit-compress and --memlimit-decompress. To set both at once, -M or --memlimit can be used. --memory was retained as a legacy alias for --memlimit for backwards compatibility. The semantics of --info-memory were changed in backwards incompatible way. Compatibility wasn't meaningful due to changes in the memory usage limiter functionality. The memory usage limiter info is no longer shown at the bottom of xz --long -help. The memory usage limiter support for removed completely from xzdec. xz's man page was updated to match the above changes. Various unrelated fixes were also made to the man page.
2010-03-07Increase the default memory usage limit on "low-memory" systems.Lasse Collin1-12/+31
Previously the default limit was always 40 % of RAM. The new limit is a little bit more complex: - If 40 % of RAM is at least 80 MiB, 40 % of RAM is used as the limit. - If 80 % of RAM is over 80 MiB, 80 MiB is used as the limit. - Otherwise 80 % of RAM is used as the limit. This should make it possible to decompress files created with "xz -9" on more systems. Swapping is generally more expected on systems with less RAM, so higher default limit on them shouldn't cause too bad surprises in terms of heavy swapping. Instead, the higher default limit should reduce the number of bad surprises when it used to prevent decompression of files created with "xz -9". The DoS prevention system shouldn't be a DoS itself. Note that even with the new default limit, a system with 64 MiB RAM cannot decompress files created with "xz -9" without user overriding the limit. This should be OK, because if xz is going to need more memory than the system has RAM, it will run very very slowly and thus it's good that user has to override the limit in that case.
2009-11-15Add lzma_physmem().Lasse Collin1-2/+1
I had hoped to keep liblzma as purely a compression library as possible (e.g. file I/O will go into a different library), but it seems that applications linking agaisnt liblzma need some way to determine the memory usage limit, and knowing the amount of RAM is one reasonable way to help making such decisions. Thanks to Jonathan Nieder for the original patch.
2009-10-02Add support for --enable-assume-ram=SIZE.Lasse Collin1-4/+3
2009-09-19Various changes.Lasse Collin1-4/+4
Separate a few reusable components from XZ Utils specific code. The reusable code is now in "tuklib" modules. A few more could be separated still, e.g. bswap.h. Fix some bugs in lzmainfo. Fix physmem and cpucores code on OS/2. Thanks to Elbert Pol for help. Add OpenVMS support into physmem. Add a few #ifdefs to ease building XZ Utils on OpenVMS. Thanks to Jouk Jansen for the original patch.
2009-05-22Make the default memory usage limit 40 % of RAM for bothLasse Collin1-50/+36
compressing and decompressing. This should be OK now that xz automatically scales down the compression settings if they would exceed the memory usage limit (earlier, the limit for compression was increased to 90 % because low limit broke scripts that used "xz -9" on systems with low RAM). Support spcifying the memory usage limit as a percentage of RAM (e.g. --memory=50%). Support --threads=0 to reset the thread limit to the default value (number of available CPU cores). Use UINT32_MAX instead of SIZE_MAX as the maximum in args.c. hardware.c was already expecting uint32_t value. Cleaned up the output of --help and --long-help.
2009-04-13Put the interesting parts of XZ Utils into the public domain.Lasse Collin1-10/+3
Some minor documentation cleanups were made at the same time.
2009-02-14Cleanups to the code that detects the amount of RAM andLasse Collin1-28/+22
the number of CPU cores. Added support for using sysinfo() on Linux systems whose libc lacks appropriate sysconf() support (at least dietlibc). The Autoconf macros were split into separate files, and CPU core count detection was moved from hardware.c to cpucores.h. The core count isn't used for anything real for now, so a problematic part in process.c was commented out.
2009-02-07Assume 32 MiB of RAM on unsupported operating systems likeLasse Collin1-1/+1
the comment in hardware.c already said.
2008-11-19Renamed lzma to xz and lzmadec to xzdec. We create symlinksLasse Collin1-0/+122
lzma, unlzma, and lzcat in "make install" for backwards compatibility with LZMA Utils 4.32.x; I'm not sure if this should be the default though.
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-20 12:27:07 +0000