aboutsummaryrefslogtreecommitdiff
path: root/doc/liblzma-advanced.txt
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--doc/liblzma-advanced.txt324
1 files changed, 0 insertions, 324 deletions
diff --git a/doc/liblzma-advanced.txt b/doc/liblzma-advanced.txt
deleted file mode 100644
index 6e1c9834..00000000
--- a/doc/liblzma-advanced.txt
+++ /dev/null
@@ -1,324 +0,0 @@
-
-Advanced features of liblzma
-----------------------------
-
-0. Introduction
-
- Most developers need only the basic features of liblzma. These
- features allow single-threaded encoding and decoding of .lzma files
- in streamed mode.
-
- In some cases developers want more. The .lzma file format is
- designed to allow multi-threaded encoding and decoding and limited
- random-access reading. These features are possible in non-streamed
- mode and limitedly also in streamed mode.
-
- To take advange of these features, the application needs a custom
- .lzma file format handler. liblzma provides a set of tools to ease
- this task, but it's still quite a bit of work to get a good custom
- .lzma handler done.
-
-
-1. Where to begin
-
- Start by reading the .lzma file format specification. Understanding
- the basics of the .lzma file structure is required to implement a
- custom .lzma file handler and to understand the rest of this document.
-
-
-2. The basic components
-
-2.1. Stream Header and tail
-
- Stream Header begins the .lzma Stream and Stream tail ends it. Stream
- Header is defined in the file format specification, but Stream tail
- isn't (thus I write "tail" with a lower-case letter). Stream tail is
- simply the Stream Flags and the Footer Magic Bytes fields together.
- It was done this way in liblzma, because the Block coders take care
- of the rest of the stuff in the Stream Footer.
-
- For now, the size of Stream Header is fixed to 11 bytes. The header
- <lzma/stream_flags.h> defines LZMA_STREAM_HEADER_SIZE, which you
- should use instead of a hardcoded number. Similarly, Stream tail
- is fixed to 3 bytes, and there is a constant LZMA_STREAM_TAIL_SIZE.
-
- It is possible, that a future version of the .lzma format will have
- variable-sized Stream Header and tail. As of writing, this seems so
- unlikely though, that it was considered simplest to just use a
- constant instead of providing a functions to get and store the sizes
- of the Stream Header and tail.
-
-
-2.x. Stream tail
-
- For now, the size of Stream tail is fixed to 3 bytes. The header
- <lzma/stream_flags.h> defines LZMA_STREAM_TAIL_SIZE, which you
- should use instead of a hardcoded number.
-
-
-3. Keeping track of size information
-
- The lzma_info_* functions found from <lzma/info.h> should ease the
- task of keeping track of sizes of the Blocks and also the Stream
- as a whole. Using these functions is strongly recommended, because
- there are surprisingly many situations where an error can occur,
- and these functions check for possible errors every time some new
- information becomes available.
-
- If you find lzma_info_* functions lacking something that you would
- find useful, please contact the author.
-
-
-3.1. Start offset of the Stream
-
- If you are storing the .lzma Stream inside anothe file format, or
- for some other reason are placing the .lzma Stream to somewhere
- else than to the beginning of the file, you should tell the starting
- offset of the Stream using lzma_info_start_offset_set().
-
- The start offset of the Stream is used for two distinct purporses.
- First, knowing the start offset of the Stream allows
- lzma_info_alignment_get() to correctly calculate the alignment of
- every Block. This information is given to the Block encoder, which
- will calculate the size of Header Padding so that Compressed Data
- is alignment at an optimal offset.
-
- Another use for start offset of the Stream is in random-access
- reading. If you set the start offset of the Stream, lzma_info_locate()
- will be able to calculate the offset relative to the beginning of the
- file containing the Stream (instead of offset relative to the
- beginning of the Stream).
-
-
-3.2. Size of Stream Header
-
- While the size of Stream Header is constant (11 bytes) in the current
- version of the .lzma file format, this may change in future.
-
-
-3.3. Size of Header Metadata Block
-
- This information is needed when doing random-access reading, and
- to verify the value of this field stored in Footer Metadata Block.
-
-
-3.4. Total Size of the Data Blocks
-
-
-3.5. Uncompressed Size of Data Blocks
-
-
-3.6. Index
-
-
-
-
-x. Alignment
-
- There are a few slightly different types of alignment issues when
- working with .lzma files.
-
- The .lzma format doesn't strictly require any kind of alignment.
- However, if the encoder carefully optimizes the alignment in all
- situations, it can improve compression ratio, speed of the encoder
- and decoder, and slightly help if the files get damaged and need
- recovery.
-
- Alignment has the most significant effect compression ratio FIXME
-
-
-x.1. Compression ratio
-
- Some filters take advantage of the alignment of the input data.
- To get the best compression ratio, make sure that you feed these
- filters correctly aligned data.
-
- Some filters (e.g. LZMA) don't necessarily mind too much if the
- input doesn't match the preferred alignment. With these filters
- the penalty in compression ratio depends on the specific type of
- data being compressed.
-
- Other filters (e.g. PowerPC executable filter) won't work at all
- with data that is improperly aligned. While the data can still
- be de-filtered back to its original form, the benefit of the
- filtering (better compression ratio) is completely lost, because
- these filters expect certain patterns at properly aligned offsets.
- The compression ratio may even worse with incorrectly aligned input
- than without the filter.
-
-
-x.1.1. Inter-filter alignment
-
- When there are multiple filters chained, checking the alignment can
- be useful not only with the input of the first filter and output of
- the last filter, but also between the filters.
-
- Inter-filter alignment important especially with the Subblock filter.
-
-
-x.1.2. Further compression with external tools
-
- This is relatively rare situation in practice, but still worth
- understanding.
-
- Let's say that there are several SPARC executables, which are each
- filtered to separate .lzma files using only the SPARC filter. If
- Uncompressed Size is written to the Block Header, the size of Block
- Header may vary between the .lzma files. If no Padding is used in
- the Block Header to correct the alignment, the starting offset of
- the Compressed Data field will be differently aligned in different
- .lzma files.
-
- All these .lzma files are archived into a single .tar archive. Due
- to nature of the .tar format, every file is aligned inside the
- archive to an offset that is a multiple of 512 bytes.
-
- The .tar archive is compressed into a new .lzma file using the LZMA
- filter with options, that prefer input alignment of four bytes. Now
- if the independent .lzma files don't have the same alignment of
- the Compressed Data fields, the LZMA filter will be unable to take
- advantage of the input alignment between the files in the .tar
- archive, which reduces compression ratio.
-
- Thus, even if you have only single Block per file, it can be good for
- compression ratio to align the Compressed Data to optimal offset.
-
-
-x.2. Speed
-
- Most modern computers are faster when multi-byte data is located
- at aligned offsets in RAM. Proper alignment of the Compressed Data
- fields can slightly increase the speed of some filters.
-
-
-x.3. Recovery
-
- Aligning every Block Header to start at an offset with big enough
- alignment may ease or at least speed up recovery of broken files.
-
-
-y. Typical usage cases
-
-y.x. Parsing the Stream backwards
-
- You may need to parse the Stream backwards if you need to get
- information such as the sizes of the Stream, Index, or Extra.
- The basic procedure to do this follows.
-
- Locate the end of the Stream. If the Stream is stored as is in a
- standalone .lzma file, simply seek to the end of the file and start
- reading backwards using appropriate buffer size. The file format
- specification allows arbitrary amount of Footer Padding (zero or more
- NUL bytes), which you skip before trying to decode the Stream tail.
-
- Once you have located the end of the Stream (a non-NULL byte), make
- sure you have at least the last LZMA_STREAM_TAIL_SIZE bytes of the
- Stream in a buffer. If there isn't enough bytes left from the file,
- the file is too small to contain a valid Stream. Decode the Stream
- tail using lzma_stream_tail_decoder(). Store the offset of the first
- byte of the Stream tail; you will need it later.
-
- You may now want to do some internal verifications e.g. if the Check
- type is supported by the liblzma build you are using.
-
- Decode the Backward Size field with lzma_vli_reverse_decode(). The
- field is at maximum of LZMA_VLI_BYTES_MAX bytes long. Check that
- Backward Size is not zero. Store the offset of the first byte of
- the Backward Size; you will need it later.
-
- Now you know the Total Size of the last Block of the Stream. It's the
- value of Backward Size plus the size of the Backward Size field. Note
- that you cannot use lzma_vli_size() to calculate the size since there
- might be padding; you need to use the real observed size of the
- Backward Size field.
-
- At this point, the operation continues differently for Single-Block
- and Multi-Block Streams.
-
-
-y.x.1. Single-Block Stream
-
- There might be Uncompressed Size field present in the Stream Footer.
- You cannot know it for sure unless you have already parsed the Block
- Header earlier. For security reasons, you probably want to try to
- decode the Uncompressed Size field, but you must not indicate any
- error if decoding fails. Later you can give the decoded Uncompressed
- Size to Block decoder if Uncopmressed Size isn't otherwise known;
- this prevents it from producing too much output in case of (possibly
- intentionally) corrupt file.
-
- Calculate the start offset of the Stream:
-
- backward_offset - backward_size - LZMA_STREAM_HEADER_SIZE
-
- backward_offset is the offset of the first byte of the Backward Size
- field. Remember to check for integer overflows, which can occur with
- invalid input files.
-
- Seek to the beginning of the Stream. Decode the Stream Header using
- lzma_stream_header_decoder(). Verify that the decoded Stream Flags
- match the values found from Stream tail. You can use the
- lzma_stream_flags_is_equal() macro for this.
-
- Decode the Block Header. Verify that it isn't a Metadata Block, since
- Single-Block Streams cannot have Metadata. If Uncompressed Size is
- present in the Block Header, the value you tried to decode from the
- Stream Footer must be ignored, since Uncompressed Size wasn't actually
- present there. If Block Header doesn't have Uncompressed Size, and
- decoding the Uncompressed Size field from the Stream Footer failed,
- the file is corrupt.
-
- If you were only looking for the Uncompressed Size of the Stream,
- you now got that information, and you can stop processing the Stream.
-
- To decode the Block, the same instructions apply as described in
- FIXME. However, because you have some extra known information decoded
- from the Stream Footer, you should give this information to the Block
- decoder so that it can verify it while decoding:
- - If Uncompressed Size is not present in the Block Header, set
- lzma_options_block.uncompressed_size to the value you decoded
- from the Stream Footer.
- - Always set lzma_options_block.total_size to backward_size +
- size_of_backward_size (you calculated this sum earlier already).
-
-
-y.x.2. Multi-Block Stream
-
- Calculate the start offset of the Footer Metadata Block:
-
- backward_offset - backward_size
-
- backward_offset is the offset of the first byte of the Backward Size
- field. Remember to check for integer overflows, which can occur with
- broken input files.
-
- Decode the Block Header. Verify that it is a Metadata Block. Set
- lzma_options_block.total_size to backward_size + size_of_backward_size
- (you calculated this sum earlier already). Then decode the Footer
- Metadata Block.
-
- Store the decoded Footer Metadata to lzma_info structure using
- lzma_info_set_metadata(). Set also the offset of the Backward Size
- field using lzma_info_size_set(). Then you can get the start offset
- of the Stream using lzma_info_size_get(). Note that any of these steps
- may fail so don't omit error checking.
-
- Seek to the beginning of the Stream. Decode the Stream Header using
- lzma_stream_header_decoder(). Verify that the decoded Stream Flags
- match the values found from Stream tail. You can use the
- lzma_stream_flags_is_equal() macro for this.
-
- If you were only looking for the Uncompressed Size of the Stream,
- it's possible that you already have it now. If Uncompressed Size (or
- whatever information you were looking for) isn't available yet,
- continue by decoding also the Header Metadata Block. (If some
- information is missing, the Header Metadata Block has to be present.)
-
- Decoding the Data Blocks goes the same way as described in FIXME.
-
-
-y.x.3. Variations
-
- If you know the offset of the beginning of the Stream, you may want
- to parse the Stream Header before parsing the Stream tail.
-