1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
|
LZMA Utils FAQ
--------------
Copyright (C) 2007 Lasse Collin
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.
Q: What are LZMA, LZMA Utils, lzma, .lzma, liblzma, LZMA SDK, LZMA_Alone,
7-Zip and p7zip?
A: LZMA stands for Lempel-Ziv-Markov chain-Algorithm. LZMA is the name
of the compression algorithm designed by Igor Pavlov. He is the author
of 7-Zip, which is a great LGPL'd compression tool for Microsoft
Windows operating systems. In addition to 7-Zip itself, also LZMA SDK
is available on the website of 7-Zip. LZMA SDK contains LZMA
implementations in C++, Java and C#. The C++ version is the original
implementation which is used also in 7-Zip itself.
Excluding the unrar plugin, 7-Zip is free software (free as in
freedom). Thanks to this, it was possible to port it to POSIX
platforms. The port was done and is maintained by myspace (TODO:
myspace's real name?). p7zip is a port of 7-Zip's command line version;
p7zip doesn't include the 7-Zip's GUI.
In POSIX world, users are used to gzip and bzip2 command line tools.
Developers know APIs of zlib and libbzip2. LZMA Utils try to ease
adoption of LZMA on free operating systems by providing a compression
library and a set of command line tools. The library is called liblzma.
It provides a zlib-like API making it easy to adapt LZMA compression in
existing applications. The main command line tool is known as lzma,
whose command line syntax is very similar to that of gzip and bzip2.
The original command line tool from LZMA SDK (lzma.exe) was found from
a directory called LZMA_Alone in the LZMA SDK. It used a simple header
format in .lzma files. This format was also used by LZMA Utils up to
and including 4.32.x. In LZMA Utils documentation, LZMA_Alone refers
to both the file format and the command line tool from LZMA SDK.
Because of various limitations of the LZMA_Alone file format, a new
file format was developed. Extending some existing format such as .gz
used by gzip was considered, but these formats were found to be too
limited. The filename suffix for the new .lzma format is `.lzma'. The
same suffix is also used for files in the LZMA_Alone format. To make
the transition to the new format as transparent as possible, LZMA Utils
support both the new and old formats transparently.
7-Zip and LZMA SDK: <http://7-zip.org/>
p7zip: <http://p7zip.sourceforge.net/>
LZMA Utils: <http://tukaani.org/lzma/>
Q: What LZMA implementations there are available?
A: LZMA SDK contains implementations in C++, Java and C#. The C++ version
is the original implementation which is part of 7-Zip. LZMA SDK
contains also a small LZMA decoder in C.
A port of LZMA SDK to Pascal was made by Alan Birtles
<http://www.birtles.org.uk/programming/>. It should work with
multiple Pascal programming language implementations.
LZMA Utils includes liblzma, which is directly based on LZMA SDK.
liblzma is written in C (C99, not C89). In contrast to C++ callback
API used by LZMA SDK, liblzma uses zlib-like stateful C API. I do not
want to comment whether both/former/latter/neither API(s) are good or
bad. The only reason to implement a zlib-like API was, that many
developers are already familiar with zlib, and very many applications
already use zlib. Having a similar API makes it easier to include LZMA
support in existing applications.
See also <http://en.wikipedia.org/wiki/LZMA#External_links>.
Q: Which file formats are supported by LZMA Utils?
A: Even when the raw LZMA stream is always the same, it can be wrapped
in different container formats. The preferred format is the new .lzma
format. It has magic bytes (the first six bytes: 0xFF 'L' 'Z' 'M'
'A' 0x00). The format supports chaining up to seven filters filters,
splitting data to multiple blocks for easier multi-threading and rough
random-access reading. The file integrity is verified using CRC32,
CRC64, or SHA256, and by verifying the uncompressed size of the file.
LZMA SDK includes a tool called LZMA_Alone. It supports uses a
primitive header which includes only the mandatory stream information
required by the LZMA decoder. This format can be both read and
written by liblzma and the command line tool (use --format=alone to
create such files).
.7z is the native archive format used by 7-Zip. This format is not
supported by liblzma, and probably will never be supported. You
should use e.g. p7zip to extract .7z files.
It is possible to implement custom file formats by using raw filter
mode in liblzma. In this mode the application needs to store the filter
properties and provide them to liblzma before starting to uncompress
the data.
Q: How can I identify files containing LZMA compressed data?
A: The preferred filename suffix for .lzma files is `.lzma'. `.tar.lzma'
may be abbreviated to `.tlz'. The same suffixes are used for files in
LZMA_Alone format. In practice this should be no problem since tools
included in LZMA Utils support both formats transparently.
Checking the magic bytes is easy way to detect files in the new .lzma
format (the first six bytes: 0xFF 'L' 'Z' 'M' 'A' 0x00). The "file"
command version FIXME contains magic strings for this format.
The old LZMA_Alone format has no magic bytes. Its header cannot contain
arbitrary bytes, thus it is possible to make a guess. Unfortunately the
guessing is usually too hard to be reliable, so don't try it unless you
are desperate.
Q: Does the lzma command line tool support sparse files?
A: Sparse files can (of course) be compressed like normal files, but
uncompression will not restore sparseness of the file. Use an archiver
tool to take care of sparseness before compressing the data with lzma.
The reason for this is that archiver tools handle files, while
compression tools handle streams or buffers. Being a sparse file is
a property of the file on the disk, not a property of the stream or
buffer.
Q: Can I recover parts of a broken LZMA file (e.g. corrupted CD-R)?
A: With LZMA_Alone and single-block .lzma files, you can uncompress the
file until you hit the first broken byte. The data after the broken
position is lost. LZMA relies on the uncompression history, and if
bytes are missing in the middle of the file, it is impossible to
reliably continue after the broken section.
With multi-block .lzma files it may be possible to locale the next
block in the file and continue decoding there. A limited recovery
tool for this kind of situations is planned.
Q: Is LZMA patented?
A: No, the authors are not aware of any patents that could affect LZMA.
However, due to nature of software patents, the authors cannot
guarantee, that LZMA isn't affected by any third party patent.
Q: Where can I find documentation about how LZMA works as an algorithm?
A: Read the source code, Luke. There is no documentation about LZMA
internals. It is possible that Igor Pavlov is the only person on
the Earth that completely knows and understands the algorithm.
You could begin by downloading LZMA SDK, and start reading from
the LZMA decoder to get some idea about the bitstream format.
Before you begin, you should know the basics of LZ77 and
range coding algorithms. LZMA is based on LZ77, but LZMA is
*a lot* more complex. Range coding is used to compress the
final bitstream like Huffman coding is used in Deflate.
Q: What are filters?
A: In context of .lzma files, a filter means an implementation of a
compression algorithm. The primary filter is LZMA, which is why
the names of the tools contain the letters LZMA.
liblzma and the new .lzma format support also other filters than LZMA.
There are different types of filters, which are suitable for different
types of data. Thus, to select the optimal filter and settings, the
type of the input data being compressed needs to be known.
Some filters are most useful when combined with another filter like
LZMA. These filters increase redundancy in the data, without changing
the size of the data, by taking advantage of properties specific to
the data being compressed.
So far, all the filters are always reversible. That is, no matter what
data you pass to a filter encoder, it can be always defiltered back to
the original form. Because of this, it is safe to compress for example
a software package that contains other file types than executables
using a filter specific to the architechture of the package being
compressed.
The old LZMA_Alone format supports only the LZMA filter.
Q: I cannot find BCJ and BCJ2 filters. Don't they exist in liblzma?
A: BCJ filter is called "x86" in liblzma. BCJ2 is not included,
because it requires using more than one encoded output stream.
Q: Can I use LZMA in proprietary, non-free applications?
A: liblzma is under the GNU LGPL version 2.1 or (at your opinion) any
later version. To summarise (*NOTE* This summary is not legally
binding, that is, it doesn't give you any extra permissions compared
to the LGPL. Read the GNU LGPL carefully for the exact license
conditions.):
* All the changes made into the library itself must be published
under the same license.
* End users must be able to replace the used liblzma. Easiest way
to assure this is to link dynamically against liblzma so users
can replace the shared library file if they want.
* You must make it clear to your users, that your application uses
liblzma, and that liblzma is free software under the GNU LGPL.
A copy of GNU LGPL must be included.
LZMA SDK contains a special exception which allows linking *unmodified*
code statically with a non-free application. This exception does *not*
apply to liblzma.
As an alternative, you can support the development of LZMA and 7-Zip
by buying a proprietary license from Igor Pavlov. See homepage of
LZMA SDK <http://7-zip.org/sdk.html> for more information. Note that
having a proprietary license from Igor Pavlov doesn't allow you to use
liblzma in a way that contradicts with the GNU LGPL, because liblzma
contains code that is not copyrighted by Igor Pavlov. Please contact
both Lasse Collin and Igor Pavlov if the license conditions of liblzma
are not suitable for you.
Q: I would like to help. What can I do?
A: See the TODO file. Please contact Lasse Collin before starting to do
anything, because it is possible that someone else is already working
on the same thing.
Q: How can I contact the authors?
A: Lasse Collin is the maintainer of LZMA Utils. You can contact him
either via IRC (Larhzu on #tukaani at Freenode or IRCnet). Email
should work too, <lasse.collin@tukaani.org>.
Igor Pavlov is the father of LZMA. He is the author of 7-Zip
and LZMA SDK. <http://7-zip.org/>
NOTE: Please don't bother Igor Pavlov with questions specific
to LZMA Utils.
|
