xz: Fix decompressor behavior if input uses an unsupported check type.

Now files with unsupported check will make xz display
a warning, set the exit status to 2 (unless --no-warn is used),
and then decompress the file normally. This is how it was
supposed to work since the beginning but this was broken by
the commit 231c3c7098f1099a56abb8afece76fc9b8699f05, that is,
a little before 5.0.0 was released. The buggy behavior displayed
a message, set exit status 1 (error), and xz didn't attempt to
to decompress the file.

This doesn't matter today except for special builds that disable
CRC64 or SHA-256 at build time (but such builds should be used
in special situations only). The bug matters if new check type
is added in the future and an old xz version is used to decompress
such a file; however, it's likely that such files would use a new
filter too and an old xz wouldn't be able to decompress the file
anyway.

The first hunk in the commit is the actual fix. The second hunk
is a cleanup since LZMA_TELL_ANY_CHECK isn't used in xz.

There is a test file for unsupported check type but it wasn't
used by test_files.sh, perhaps due to different behavior between
xz and the simpler xzdec.

1 files changed, 15 insertions, 4 deletions

diff --git a/src/xz/coder.c b/src/xz/coder.c
index 224c2d39..98e9783a 100644
--- a/src/xz/coder.c
+++ b/src/xz/coder.c
@@ -611,10 +611,21 @@ coder_init(file_pair *pair)
 		// memory usage limit in case it happens in the first
 		// Block of the first Stream, which is where it very
 		// probably will happen if it is going to happen.
+		//
+		// This will also catch unsupported check type which
+		// we treat as a warning only. If there are empty
+		// concatenated Streams with unsupported check type then
+		// the message can be shown more than once here. The loop
+		// is used in case there is first a warning about
+		// unsupported check type and then the first Block
+		// would exceed the memlimit.
 		if (ret == LZMA_OK && init_format != FORMAT_RAW) {
 			strm.next_out = NULL;
 			strm.avail_out = 0;
-			ret = lzma_code(&strm, LZMA_RUN);
+			while ((ret = lzma_code(&strm, LZMA_RUN))
+					== LZMA_UNSUPPORTED_CHECK)
+				message_warning("%s: %s", pair->src_name,
+						message_strm(ret));
 		}
 #endif
 	}
@@ -825,9 +836,9 @@ coder_normal(file_pair *pair)
 
 		} else if (ret != LZMA_OK) {
 			// Determine if the return value indicates that we
-			// won't continue coding.
-			const bool stop = ret != LZMA_NO_CHECK
-					&& ret != LZMA_UNSUPPORTED_CHECK;
+			// won't continue coding. LZMA_NO_CHECK would be
+			// here too if LZMA_TELL_ANY_CHECK was used.
+			const bool stop = ret != LZMA_UNSUPPORTED_CHECK;
 
 			if (stop) {
 				// Write the remaining bytes even if something