xar: clamp the input count before the 32 bit avail_in fields
The XAR reader gets its byte count from __archive_read_ahead(). With archive_read_open_memory() that count can exceed 4 GiB, because the whole buffer arrives as one block. The gzip and bzip2 branches cast the count into avail_in, which is 32 bits wide. A count of exactly 4 GiB truncates to zero. inflate() then returns Z_BUF_ERROR, and the read fails with "File decompression failed (-5)". The bzip2 branch truncates the same way into bzstream.avail_in. Clamp both counts. This is the same guard that the 7-Zip reader already applies to the same fields. The clamp changes the local avail_in and avail_out, so the *used and *outbytes counts that both branches report from those locals stay correct. The lzma branch needs no guard, because lzma_stream holds both counts in a size_t. To reproduce: build a XAR archive with one gzip encoded entry whose declared compressed length is 2^32, put a short zlib stream at the head of the heap, pad the heap out to that length, then call archive_read_open_memory() and archive_read_data_block(). Before this change the read fails with "File decompression failed (-5)".
A
abhinavmir committed
502a52a7d4d2f551c8481da33a9b2a7c791fed98
Parent: 17b5db0