diff --git a/AUTHORS b/AUTHORS new file mode 100644 index 0000000000..f805a204ec --- /dev/null +++ b/AUTHORS @@ -0,0 +1,58 @@ + +Authors of XZ Utils +=================== + + XZ Utils is developed and maintained by + Lasse Collin . + + Major parts of liblzma are based on code written by Igor Pavlov, + specifically the LZMA SDK . Without + this code, XZ Utils wouldn't exist. + + The SHA-256 implementation in liblzma is based on code written by + Wei Dai in Crypto++ Library . + + A few scripts have been adapted from GNU gzip. The original + versions were written by Jean-loup Gailly, Charles Levert, and + Paul Eggert. Andrew Dudman helped adapting the scripts and their + man pages for XZ Utils. + + The initial version of the threaded .xz decompressor was written + by Sebastian Andrzej Siewior. + + The initial version of the .lz (lzip) decoder was written + by Michał Górny. + + Architecture-specific CRC optimizations were contributed by + Ilya Kurdyukov, Chenxi Mao, and Xi Ruoyao. + + Other authors: + - Jonathan Nieder + - Joachim Henke + + Special author: Jia Tan was a co-maintainer in 2022-2024. He and + the team behind him inserted a backdoor (CVE-2024-3094) into + XZ Utils 5.6.0 and 5.6.1 releases. He suddenly disappeared when + this was discovered. + + Many people have contributed improvements or reported bugs. + Most of these people are mentioned in the file THANKS. + + The translations of the command line tools and man pages have been + contributed by many people via the Translation Project: + + - https://translationproject.org/domain/xz.html + - https://translationproject.org/domain/xz-man.html + + The authors of the translated man pages are in the header comments + of the man page files. In the source package, the authors of the + translations are in po/*.po and po4a/*.po files. + + Third-party code whose authors aren't listed here: + + - GNU getopt_long() in the 'lib' directory is included for + platforms that don't have a usable getopt_long(). + + - The build system files from GNU Autoconf, GNU Automake, + GNU Libtool, GNU Gettext, Autoconf Archive, and related files. + diff --git a/COPYING b/COPYING new file mode 100644 index 0000000000..ef3371389d --- /dev/null +++ b/COPYING @@ -0,0 +1,70 @@ + +XZ Utils Licensing +================== + + Different licenses apply to different files in this package. Here + is a summary of which licenses apply to which parts of this package: + + - liblzma is under the BSD Zero Clause License (0BSD). + + - The command line tools xz, xzdec, lzmadec, and lzmainfo are + under 0BSD except that, on systems that don't have a usable + getopt_long, GNU getopt_long is compiled and linked in from the + 'lib' directory. The getopt_long code is under GNU LGPLv2.1+. + + - The scripts to grep, diff, and view compressed files have been + adapted from GNU gzip. These scripts (xzgrep, xzdiff, xzless, + and xzmore) are under GNU GPLv2+. The man pages of the scripts + are under 0BSD; they aren't based on the man pages of GNU gzip. + + - Most of the XZ Utils specific documentation that is in + plain text files (like README, INSTALL, PACKAGERS, NEWS, + and ChangeLog) are under 0BSD unless stated otherwise in + the file itself. The files xz-file-format.txt and + lzma-file-format.xt are in the public domain but may + be distributed under the terms of 0BSD too. + + - Translated messages and man pages are under 0BSD except that + some old translations are in the public domain. + + - Test files and test code in the 'tests' directory, and + debugging utilities in the 'debug' directory are under + the BSD Zero Clause License (0BSD). + + - The GNU Autotools based build system contains files that are + under GNU GPLv2+, GNU GPLv3+, and a few permissive licenses. + These files don't affect the licensing of the binaries being + built. + + - The 'extra' directory contains files that are under various + free software licenses. These aren't built or installed as + part of XZ Utils. + + The following command may be helpful in finding per-file license + information. It works on xz.git and on a clean file tree extracted + from a release tarball. + + sh build-aux/license-check.sh -v + + For the files under the BSD Zero Clause License (0BSD), if + a copyright notice is needed, the following is sufficient: + + Copyright (C) The XZ Utils authors and contributors + + If you copy significant amounts of 0BSD-licensed code from XZ Utils + into your project, acknowledging this somewhere in your software is + polite (especially if it is proprietary, non-free software), but + it is not legally required by the license terms. Here is an example + of a good notice to put into "about box" or into documentation: + + This software includes code from XZ Utils . + + The following license texts are included in the following files: + - COPYING.0BSD: BSD Zero Clause License + - COPYING.LGPLv2.1: GNU Lesser General Public License version 2.1 + - COPYING.GPLv2: GNU General Public License version 2 + - COPYING.GPLv3: GNU General Public License version 3 + + If you have questions, don't hesitate to ask for more information. + The contact information is in the README file. + diff --git a/COPYING.0BSD b/COPYING.0BSD new file mode 100644 index 0000000000..4a9be38928 --- /dev/null +++ b/COPYING.0BSD @@ -0,0 +1,11 @@ +Permission to use, copy, modify, and/or distribute this +software for any purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL +WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL +THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR +CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, +NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/ChangeLog b/ChangeLog new file mode 100644 index 0000000000..577dce5e12 --- /dev/null +++ b/ChangeLog @@ -0,0 +1,17389 @@ +commit a522a226545730551f7e7c2685fab27cf567746c +Author: Lasse Collin +Date: 2025-04-03 14:34:43 +0300 + + Bump version and soname for 5.8.1 + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit 1c462c2ad86ff85766928638431029cd0b0dc995 +Author: Lasse Collin +Date: 2025-04-03 14:34:43 +0300 + + Add NEWS for 5.8.1 + + NEWS | 30 ++++++++++++++++++++++++++++++ + 1 file changed, 30 insertions(+) + +commit 513cabcf7f5ce1c3ed0619e791393fc53d1dbbd0 +Author: Lasse Collin +Date: 2025-04-03 14:34:43 +0300 + + Tests: Call lzma_code() in smaller chunks in fuzz_common.h + + This makes it easy to crash fuzz_decode_stream_mt when tested + against the code from 5.8.0. + + Obviously this might make it harder to reach some other code path now. + The previous code has been in use since 2018 when fuzzing was added + in 106d1a663d4b ("Tests: Add a fuzz test program and a config file + for OSS-Fuzz."). + + tests/ossfuzz/fuzz_common.h | 31 ++++++++++++++++++++++++------- + 1 file changed, 24 insertions(+), 7 deletions(-) + +commit 48440e24a25911ae59e8518b67a1e0f6f1c293bf +Author: Lasse Collin +Date: 2025-04-03 14:34:43 +0300 + + Tests: Add a fuzzing target for the multithreaded .xz decoder + + It doesn't seem possible to trigger the CVE-2025-31115 bug with this + fuzzing target at the moment. It's because the code in fuzz_common.h + passes the whole input buffer to lzma_code() at once. + + tests/ossfuzz/fuzz_decode_stream_mt.c | 47 +++++++++++++++++++++++++++++++++++ + 1 file changed, 47 insertions(+) + +commit 0c80045ab82c406858d9d5bcea9f48ebc3d0a81d +Author: Lasse Collin +Date: 2025-04-03 14:34:42 +0300 + + liblzma: mt dec: Fix lack of parallelization in single-shot decoding + + Single-shot decoding means calling lzma_code() by giving it the whole + input at once and enough output buffer space to store the uncompressed + data, and combining this with LZMA_FINISH and no timeout + (lzma_mt.timeout = 0). This way the file is decoded with a single + lzma_code() call if possible. + + The bug prevented the decoder from starting more than one worker thread + in single-shot mode. The issue was noticed when reviewing the code; + there are no bug reports. Thus maybe few have tried this mode. + + Fixes: 64b6d496dc81 ("liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.") + + src/liblzma/common/stream_decoder_mt.c | 11 +++++++++-- + 1 file changed, 9 insertions(+), 2 deletions(-) + +commit 8188048854e8d11071b8a50d093c74f4c030acc9 +Author: Lasse Collin +Date: 2025-04-03 14:34:42 +0300 + + liblzma: mt dec: Don't modify thr->in_size in the worker thread + + Don't set thr->in_size = 0 when returning the thread to the stack of + available threads. Not only is it useless, but the main thread may + read the value in SEQ_BLOCK_THR_RUN. With valid inputs, it made + no difference if the main thread saw the original value or 0. With + invalid inputs (when worker thread stops early), thr->in_size was + no longer modified after the previous commit with the security fix + ("Don't free the input buffer too early"). + + So while the bug appears harmless now, it's important to fix it because + the variable was being modified without proper locking. It's trivial + to fix because there is no need to change the value. Only main thread + needs to set the value in (in SEQ_BLOCK_THR_INIT) when starting a new + Block before the worker thread is activated. + + Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.") + Reviewed-by: Sebastian Andrzej Siewior + Thanks-to: Sam James + + src/liblzma/common/stream_decoder_mt.c | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit d5a2ffe41bb77b918a8c96084885d4dbe4bf6480 +Author: Lasse Collin +Date: 2025-04-03 14:34:42 +0300 + + liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115) + + The input buffer must be valid as long as the main thread is writing + to the worker-specific input buffer. Fix it by making the worker + thread not free the buffer on errors and not return the worker thread to + the pool. The input buffer will be freed when threads_end() is called. + + With invalid input, the bug could at least result in a crash. The + effects include heap use after free and writing to an address based + on the null pointer plus an offset. + + The bug has been there since the first committed version of the threaded + decoder and thus affects versions from 5.3.3alpha to 5.8.0. + + As the commit message in 4cce3e27f529 says, I had made significant + changes on top of Sebastian's patch. This bug was indeed introduced + by my changes; it wasn't in Sebastian's version. + + Thanks to Harri K. Koskinen for discovering and reporting this issue. + + Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.") + Reported-by: Harri K. Koskinen + Reviewed-by: Sebastian Andrzej Siewior + Thanks-to: Sam James + + src/liblzma/common/stream_decoder_mt.c | 31 ++++++++++++++++++++++--------- + 1 file changed, 22 insertions(+), 9 deletions(-) + +commit c0c835964dfaeb2513a3c0bdb642105152fe9f34 +Author: Lasse Collin +Date: 2025-04-03 14:34:42 +0300 + + liblzma: mt dec: Simplify by removing the THR_STOP state + + The main thread can directly set THR_IDLE in threads_stop() which is + called when errors are detected. threads_stop() won't return the stopped + threads to the pool or free the memory pointed by thr->in anymore, but + it doesn't matter because the existing workers won't be reused after + an error. The resources will be cleaned up when threads_end() is + called (reinitializing the decoder always calls threads_end()). + + Reviewed-by: Sebastian Andrzej Siewior + Thanks-to: Sam James + + src/liblzma/common/stream_decoder_mt.c | 75 +++++++++++++--------------------- + 1 file changed, 29 insertions(+), 46 deletions(-) + +commit 831b55b971cf579ee16a854f177c36b20d3c6999 +Author: Lasse Collin +Date: 2025-04-03 14:34:42 +0300 + + liblzma: mt dec: Fix a comment + + Reviewed-by: Sebastian Andrzej Siewior + Thanks-to: Sam James + + src/liblzma/common/stream_decoder_mt.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b9d168eee4fb6393b4fe207c0aeb5faee316ca1a +Author: Lasse Collin +Date: 2025-04-03 14:34:30 +0300 + + liblzma: Add assertions to lzma_bufcpy() + + src/liblzma/common/common.c | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit c8e0a4897b4d0f906966f5d4d4f662221d64f3ae +Author: Lasse Collin +Date: 2025-04-02 16:40:22 +0300 + + DOS: Update Makefile to fix the build + + dos/Makefile | 2 ++ + 1 file changed, 2 insertions(+) + +commit 307c02ed698a69763ef1c9c0df4ff24727442118 +Author: Lasse Collin +Date: 2025-03-29 12:41:32 +0200 + + sysdefs.h: Avoid even with C11 compilers + + Oracle Developer Studio 12.6 on Solaris 10 claims C11 support in + __STDC_VERSION__ and supports _Alignas. However, is missing. + We only need alignas, so define it to _Alignas with C11/C17 compilers. + If something included later, it shouldn't cause problems. + + Thanks to Ihsan Dogan for reporting the issue and testing the fix. + + Fixes: c0e7eaae8d6eef1e313c9d0da20ccf126ec61f38 + + src/common/sysdefs.h | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 7ce38b318339d6c01378a77585e08169ca3a604e +Author: Lasse Collin +Date: 2025-03-29 12:32:05 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 688e51bde4c987589717b2be1a1fde9576c604fc +Author: Lasse Collin +Date: 2025-03-29 12:21:51 +0200 + + Translations: Update the Croatian translation + + po/hr.po | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +commit 173fb5c68b08a8c1369550267be258132b7760c6 +Author: Lasse Collin +Date: 2025-03-25 18:23:57 +0200 + + doc/SHA256SUMS: Add 5.8.0 + + doc/SHA256SUMS | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit db9258e828bc2cd96e3954f1ddcc9d3530589025 +Author: Lasse Collin +Date: 2025-03-25 15:18:32 +0200 + + Bump version and soname for 5.8.0 + + Also remove the LZMA_UNSTABLE macro. + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/bcj.h | 2 -- + src/liblzma/api/lzma/version.h | 6 +++--- + src/liblzma/common/common.h | 2 -- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 6 files changed, 6 insertions(+), 10 deletions(-) + +commit bfb752a38f89ed03fc93d54f11c09f43fda64bc2 +Author: Lasse Collin +Date: 2025-03-25 15:18:32 +0200 + + Add NEWS for 5.8.0 + + NEWS | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 62 insertions(+) + +commit 6ccbb904da851eb0c174c8dbd43e84da31739720 +Author: Lasse Collin +Date: 2025-03-25 15:18:31 +0200 + + Translations: Run "make -C po update-po" + + POT-Creation-Date is set to match the timestamp in 5.7.2beta which + in the Translation Project is known as 5.8.0-pre1. The strings + haven't changed since 5.7.1alpha but a few comments have. + + This is a very noisy commit, but this helps keeping the PO files + similar between the Git repository and stable release tarballs. + + po/ca.po | 964 ++++++++++++++++++++++++++++++++++++++++++++--------------- + po/cs.po | 935 ++++++++++++++++++++++++++++++++++++++++++---------------- + po/da.po | 663 ++++++++++++++++++++++++++++++----------- + po/de.po | 7 +- + po/eo.po | 966 +++++++++++++++++++++++++++++++++++++++++++++--------------- + po/es.po | 7 +- + po/fi.po | 2 +- + po/fr.po | 916 +++++++++++++++++++++++++++++++++++++++++--------------- + po/hu.po | 966 +++++++++++++++++++++++++++++++++++++++++++++--------------- + po/ka.po | 7 +- + po/ko.po | 7 +- + po/nl.po | 7 +- + po/pl.po | 7 +- + po/pt_BR.po | 962 ++++++++++++++++++++++++++++++++++++++++++++--------------- + po/sr.po | 2 +- + po/sv.po | 7 +- + po/tr.po | 7 +- + po/uk.po | 7 +- + po/vi.po | 948 +++++++++++++++++++++++++++++++++++++++++++--------------- + po/zh_CN.po | 940 ++++++++++++++++++++++++++++++++++++++++++++-------------- + po/zh_TW.po | 2 +- + 21 files changed, 6209 insertions(+), 2120 deletions(-) + +commit 891a5f057a6bb2dd2e3ce5e3bdd7a1f1ee03b800 +Author: Lasse Collin +Date: 2025-03-25 15:18:31 +0200 + + Translations: Run po4a/update-po + + Also remove the trivial obsolete messages like man page dates. + + This is a noisy commit, but this helps keeping the PO files similar + between the Git repository and stable release tarballs. + + po4a/fr.po | 82 +++++++++++++++++++++++++++++++++++++------------------ + po4a/pt_BR.po | 88 +++++++++++++++++++++++++++++++++++++++++------------------ + po4a/sr.po | 79 ++++++++++++++++++++++++++++++++++------------------- + 3 files changed, 167 insertions(+), 82 deletions(-) + +commit 4f52e7387012cb3510b01c937dd9b3a0c6a3ac6c +Author: Lasse Collin +Date: 2025-03-25 15:18:31 +0200 + + Translations: Partially fix overtranslation in Serbian man pages + + Names of environment variables and some other strings must be present + in the original form. The translator couldn't be reached so I'm + changing some of the strings myself. In the "Robot mode" section, + occurrences in the middle of sentences weren't changed to reduce + the chance of grammar breakage, but I kept the translated strings in + parenthesis in the headings. It's not ideal, but now people shouldn't + need to look at the English man page to find the English strings. + + po4a/sr.po | 66 ++++++++++++++++++++++++++++++++++++++++++-------------------- + 1 file changed, 45 insertions(+), 21 deletions(-) + +commit ff5d944749b99eb5ab35e2ebaf01d05a59e7169b +Author: Lasse Collin +Date: 2025-03-25 15:18:31 +0200 + + liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage + + src/liblzma/lz/lz_decoder.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 943b012d09f717f7b44284c4e4976ea41264c731 +Author: Lasse Collin +Date: 2025-03-25 15:18:31 +0200 + + liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat() + + SSE2 is supported on every x86-64 processor. The SSE2 code is used on + 32-bit x86 if compiler options permit unconditional use of SSE2. + + dict_repeat() copies short random-sized unaligned buffers. At least + on glibc, FreeBSD, and Windows (MSYS2, UCRT, MSVCRT), memcpy() is + clearly faster than byte-by-byte copying in this use case. Compared + to the memcpy() version, the new SSE2 version reduces decompression + time by 0-5 % depending on the machine and libc. It should never be + slower than the memcpy() version. + + However, on musl 1.2.5 on x86-64, the memcpy() version is the slowest. + Compared to the memcpy() version: + + - The byte-by-version takes 6-7 % less time to decompress. + - The SSE2 version takes 16-18 % less time to decompress. + + The numbers are from decompressing a Linux kernel source tarball in + single-threaded mode on older AMD and Intel systems. The tarball + compresses well, and thus dict_repeat() performance matters more + than with some other files. + + src/liblzma/lz/lz_decoder.c | 14 ++++++-- + src/liblzma/lz/lz_decoder.h | 87 ++++++++++++++++++++++++++++++++++++++++----- + 2 files changed, 90 insertions(+), 11 deletions(-) + +commit bc14e4c94e788d42eeab984298391fc0ca46f969 +Author: Lasse Collin +Date: 2025-03-25 15:18:31 +0200 + + liblzma: Add "restrict" to a few functions in lz_decoder.h + + This doesn't make any difference in practice because compilers can + already see that writing through the dict->buf pointer cannot modify + the contents of *dict itself: The LZMA decoder makes a local copy of + the lzma_dict structure, and even if it didn't, the pointer to + lzma_dict in the LZMA decoder is already "restrict". + + It's nice to add "restrict" anyway. uint8_t is typically unsigned char + which can alias anything. Without the above conditions or "restrict", + compilers could need to assume that writing through dict->buf might + modify *dict. This would matter in dict_repeat() because the loops + refer to dict->buf and dict->pos instead of making local copies of + those members for the duration of the loops. If compilers had to + assume that writing through dict->buf can affect *dict, then compilers + would need to emit code that reloads dict->buf and dict->pos after + every write through dict->buf. + + src/liblzma/lz/lz_decoder.h | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +commit e82ee090c567e560f51a056775a17f534d159d65 +Author: Lasse Collin +Date: 2025-03-25 15:18:30 +0200 + + liblzma: Define LZ_DICT_INIT_POS for initial dictionary position + + It's more readable. + + src/liblzma/lz/lz_decoder.c | 4 ++-- + src/liblzma/lz/lz_decoder.h | 9 ++++++--- + 2 files changed, 8 insertions(+), 5 deletions(-) + +commit 8e7cd0091e5239334437decbe1989662d45a2f47 +Author: Lasse Collin +Date: 2025-03-25 15:18:30 +0200 + + Windows: Update README-Windows.txt about UCRT + + windows/README-Windows.txt | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 2c24292d341e505e5579fccac3bce5bc71d839ef +Author: Lasse Collin +Date: 2025-03-25 15:18:15 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 48053c90898fa191a216aefca01626520a7413f4 +Author: Lasse Collin +Date: 2025-03-17 15:33:25 +0200 + + Translations: Update the Italian translation + + po/it.po | 32 ++++++++++++++++---------------- + 1 file changed, 16 insertions(+), 16 deletions(-) + +commit 8d6f06a65f50358fad13567f5dd8af41ef1d2b58 +Author: Lasse Collin +Date: 2025-03-17 15:28:56 +0200 + + Translations: Update the Portuguese translation + + The language tag in the Translation Project is pt, not pt_PT, + thus I changed the "Language:" line to pt. + + po/pt.po | 1045 +++++++++++++++++++++++++++++++------------------------------- + 1 file changed, 526 insertions(+), 519 deletions(-) + +commit c3439b039f46fe547ad603e16dc3bd63c1ca9b0c +Author: Lasse Collin +Date: 2025-03-14 13:02:21 +0200 + + Translations: Update the Italian translation + + po/it.po | 1020 +++++++++++++++++++++++++++++++------------------------------- + 1 file changed, 516 insertions(+), 504 deletions(-) + +commit 79b4ab8d79528dd633a84df2d29e63f5d13ccbdf +Author: Lasse Collin +Date: 2025-03-12 20:48:39 +0200 + + Translations: Update the Italian man page translations + + Only trivial additions but this keeps the file in sync with the TP. + + po4a/it.po | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 515b6fc8557825e1335012b3b1c8cf71e2c38775 +Author: Lasse Collin +Date: 2025-03-12 19:38:54 +0200 + + Translations: Update the Italian man page translations + + po4a/it.po | 129 ++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 77 insertions(+), 52 deletions(-) + +commit 333b7c0b776295f0941269b4e6cdb1a0ba5f6218 +Author: Lasse Collin +Date: 2025-03-10 21:00:31 +0200 + + Translations: Update the Korean man page translations + + po4a/ko.po | 139 +++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 80 insertions(+), 59 deletions(-) + +commit ae52ebd27dc0be5e1ba62fb0c45255d8563fcd88 +Author: Lasse Collin +Date: 2025-03-10 20:56:57 +0200 + + Translations: Update the German man page translations + + po4a/de.po | 102 ++++++++++++++++++++++++++++++++++++++----------------------- + 1 file changed, 63 insertions(+), 39 deletions(-) + +commit 1028e52c93d2292b44ff7bae8e721025d2f2c94d +Author: Lasse Collin +Date: 2025-03-10 13:13:30 +0200 + + CMake: Fix tuklib_use_system_extensions + + Revert back to a macro so that list(APPEND CMAKE_REQUIRED_DEFINITIONS) + will affect the calling scope. I had forgotten that while CMake + functions inherit the variables from the parent scope, the changes + to them are local unless using set(... PARENT_SCOPE). + + This also means that the commit message in 5bb77d0920dc is wrong. The + commit itself is still fine, making it clearer that -DHAVE_SYS_PARAM_H + is only needed for specific check_c_source_compiles() calls. + + Fixes: c1ea7bd0b60eed6ebcdf9a713ca69034f6f07179 + + cmake/tuklib_common.cmake | 7 +++++-- + 1 file changed, 5 insertions(+), 2 deletions(-) + +commit 80e48836024ec2d7cbd557575be6da3d1f055cba +Author: Lasse Collin +Date: 2025-03-10 11:38:55 +0200 + + INSTALL: Document -bmaxdata on AIX + + This is based on a pull request and AIX docs. I haven't tested the + instructions myself. + + Closes: https://github.com/tukaani-project/xz/pull/137 + + INSTALL | 5 +++++ + 1 file changed, 5 insertions(+) + +commit ab319186b6d0454285ff4941a777ac95e580f60f +Author: Lasse Collin +Date: 2025-03-10 11:37:19 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 4434671a04436038f88ab0feaa251cc8d7abb683 +Author: Collin Funk +Date: 2025-03-09 19:14:31 -0700 + + tuklib_physmem: Silence -Wsign-conversion on AIX + + Closes: https://github.com/tukaani-project/xz/pull/168 + + src/common/tuklib_physmem.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 18bcaa4fafc935d89ffde94301fa6427907306bf +Author: Lasse Collin +Date: 2025-03-09 22:10:38 +0200 + + Translations: Update the Romanian man page translations + + po4a/ro.po | 110 ++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 66 insertions(+), 44 deletions(-) + +commit 1e17b7f42fe2f9df279f44ad7043d3753cd00363 +Author: Lasse Collin +Date: 2025-03-09 21:28:15 +0200 + + Translations: Update the Croatian translation + + po/hr.po | 19 +++++++++++-------- + 1 file changed, 11 insertions(+), 8 deletions(-) + +commit ff85e6130d5940896915cdbb99aa9ece9d41240b +Author: Lasse Collin +Date: 2025-03-09 21:23:34 +0200 + + Translations: Update the Romanian translation + + po/ro.po | 24 +++++++++++++----------- + 1 file changed, 13 insertions(+), 11 deletions(-) + +commit a5bfb33f30f77e656723d365db8b06e089d3de61 +Author: Lasse Collin +Date: 2025-03-09 21:11:34 +0200 + + Translations: Update the Ukrainian man page translations + + po4a/uk.po | 107 ++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 64 insertions(+), 43 deletions(-) + +commit 5bb77d0920dcf949d8eb04eb19204b7b199e42df +Author: Lasse Collin +Date: 2025-03-09 14:43:07 +0200 + + CMake: Use cmake_push_check_state in tuklib_cpucores and tuklib_physmem + + Now the changes to CMAKE_REQUIRED_DEFINITIONS are temporary and don't + leak to the calling code. + + cmake/tuklib_cpucores.cmake | 3 +++ + cmake/tuklib_physmem.cmake | 4 +++- + 2 files changed, 6 insertions(+), 1 deletion(-) + +commit c1ea7bd0b60eed6ebcdf9a713ca69034f6f07179 +Author: Lasse Collin +Date: 2025-03-09 14:06:35 +0200 + + CMake: Revise tuklib_use_system_extensions + + Define NetBSD and Darwin/macOS feature test macros. Autoconf defines + these too (and a few others). + + Define the macros on Windows except with MSVC. The _GNU_SOURCE macro + makes a difference with mingw-w64. + + Use a function instead of a macro. Don't take the TARGET_OR_ALL argument + because there's always global effect because the global variable + CMAKE_REQUIRED_DEFINITIONS is modified. + + CMakeLists.txt | 2 +- + cmake/tuklib_common.cmake | 27 +++++++++++++++------------ + 2 files changed, 16 insertions(+), 13 deletions(-) + +commit 4243c45a48ef8c103d77b75d9f93d48adcb631db +Author: Lasse Collin +Date: 2025-03-08 14:54:29 +0200 + + doc/SHA256SUMS: Add 5.7.2beta + + doc/SHA256SUMS | 3 +++ + 1 file changed, 3 insertions(+) + +commit cc7f2fc1cf9f3c63cbce90ee92bfbb004f98140b +Author: Lasse Collin +Date: 2025-03-08 14:29:57 +0200 + + Bump version and soname for 5.7.2beta + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 4 ++-- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 4 files changed, 5 insertions(+), 5 deletions(-) + +commit 62e44b36167de27541776dcf677ed04077c9fd19 +Author: Lasse Collin +Date: 2025-03-08 14:24:38 +0200 + + Add NEWS for 5.7.2beta + + NEWS | 35 +++++++++++++++++++++++++++++++++++ + 1 file changed, 35 insertions(+) + +commit 70f1f203789433b5d7b8b22e1655abc465d659f7 +Author: Lasse Collin +Date: 2025-03-08 14:23:00 +0200 + + COPYING: Remove the note about old releases + + COPYING | 19 ------------------- + 1 file changed, 19 deletions(-) + +commit db9827dc38ff79de747a6fc7a99619e961dbc5e6 +Author: Lasse Collin +Date: 2025-03-08 14:22:28 +0200 + + xz: Update the man page about the environment variables again + + src/xz/xz.1 | 22 +++++++++++----------- + 1 file changed, 11 insertions(+), 11 deletions(-) + +commit 99c584891bd1d946561cebded2226df9b83f1efb +Author: Lasse Collin +Date: 2025-03-06 19:26:09 +0200 + + liblzma: Edit spelling in a comment + + It was found with codespell. + + src/liblzma/api/lzma/container.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 7a234c8c05a8f64efde013cd6a6d31a90b7d0d28 +Author: Lasse Collin +Date: 2025-03-06 19:14:23 +0200 + + xz: Update the man page about the environment variables + + src/xz/xz.1 | 26 ++++++++++++++++++++++++-- + 1 file changed, 24 insertions(+), 2 deletions(-) + +commit 808f05af3ef40730d40b3798666757bd866484f1 +Author: Lasse Collin +Date: 2025-03-06 17:37:39 +0200 + + Docs: Add a few TRANSLATORS comments to man pages + + All translators know that --command-line-options must not be translated. + With some other strings it's not obvious when the untranslated string + must be preserved. These comments hopefully help. + + src/scripts/xzmore.1 | 2 ++ + src/xz/xz.1 | 22 ++++++++++++++++++++++ + 2 files changed, 24 insertions(+) + +commit 051de255f00dda331e2a6fa189a6e7fe56a7c69b +Author: Lasse Collin +Date: 2025-03-06 16:34:32 +0200 + + Scripts: Mark the LZMA Utils script aliases as deprecated + + The deprecated aliases are lzcmp, lzdiff, lzless, lzmore, + lzgrep, lzegrep, and lzfgrep. The commands that start with + the xz prefix have identical behavior, for example, both + lzgrep and xzgrep handle all supported file formats. + + This doesn't affect lzma, unlzma, lzcat, lzmadec, or lzmainfo. + The last release of LZMA Utils was made in 2008, but the lzma + compatibility alias for the gzip-like tool is still in common use. + Deprecating it would cause unnecessary breakage. + + src/scripts/xzdiff.1 | 5 ++++- + src/scripts/xzgrep.1 | 6 +++++- + src/scripts/xzless.1 | 4 +++- + src/scripts/xzmore.1 | 4 +++- + 4 files changed, 15 insertions(+), 4 deletions(-) + +commit 4941ea454c02cf15a64d6434a0778fc2a81282fc +Author: Lasse Collin +Date: 2025-03-02 21:13:04 +0200 + + Translations: Add Serbian man page translations + + po4a/po4a.conf | 2 +- + po4a/sr.po | 3892 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 3893 insertions(+), 1 deletion(-) + +commit d142d96f24daa451edaabfca8594e202932b3c0b +Author: Lasse Collin +Date: 2025-03-02 20:42:14 +0200 + + Translations: Update Georgian translation + + po/ka.po | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 9b7e45d841195c8fd8d286e26f810df28c53dd16 +Author: Lasse Collin +Date: 2025-02-28 21:07:21 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 9351592710e0df3238b09d39c545a643c50ac88f +Author: Lasse Collin +Date: 2025-02-22 16:04:58 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 9023be7831faca2f28def55e16c39e3a42e1e262 +Author: Lasse Collin +Date: 2025-02-19 16:33:52 +0200 + + Translations: Update the Croatian translation + + po/hr.po | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 2eaf242c56e8c65db83d48b018fa44aeafeb33a5 +Author: Lasse Collin +Date: 2025-02-17 21:46:15 +0200 + + Build: Fix out-of-tree builds when using the replacement getopt_long + + Nowaways $(top_builddir)/lib/getopt.h depends on headers in + $(top_srcdir)/lib, so both have to be in the include path. + CMake-based build already did this. + + Fixes: 7e884c00d0093c38339f17fb1d280eec493f42ca + + src/lzmainfo/Makefile.am | 6 ++++-- + src/xz/Makefile.am | 6 ++++-- + src/xzdec/Makefile.am | 6 ++++-- + 3 files changed, 12 insertions(+), 6 deletions(-) + +commit 41322b2c60cd2c67a1053cb40d27e573420185b7 +Author: Lasse Collin +Date: 2025-02-17 18:25:52 +0200 + + m4/getopt.m4: Remove an outdated comment + + m4/getopt.m4 | 3 --- + 1 file changed, 3 deletions(-) + +commit 03c23a4952bce1b50a1d213ca2d1c15acd76a489 +Author: Lasse Collin +Date: 2025-02-17 18:11:58 +0200 + + Build: Allow forcing the use of the replacement getopt_long + + Now one can pass gl_replace_getopt=yes to configure to force the use + of GNU getopt_long from the lib directory. This only checks that the + value of gl_replace_getopt is non-empty, so one cannot force the + replacement to be disabled. + + Closes: https://github.com/tukaani-project/xz/pull/166 + + m4/getopt.m4 | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit c23b837d15960ecc0d537f0260f389904e1e7f02 +Author: Lasse Collin +Date: 2025-02-17 18:11:42 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 2672a38f1159babf9ba3cca429f644bb823a8bdd +Author: Lasse Collin +Date: 2025-02-12 19:23:31 +0200 + + Update THANKS + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit 4fdcbfaf3f222299747c6a815762a74eeb1b0b23 +Author: Lasse Collin +Date: 2025-02-11 12:13:41 +0200 + + Update THANKS + + THANKS | 3 +++ + 1 file changed, 3 insertions(+) + +commit 0d553568f1af9a35779ecac41392a6c871786930 +Author: Lasse Collin +Date: 2025-02-08 11:39:08 +0200 + + Translations: Update the Polish translation + + po/pl.po | 802 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 464 insertions(+), 338 deletions(-) + +commit 9f165076aebb3b5115d2b6520529db8fa11a6bdd +Author: Lasse Collin +Date: 2025-02-07 19:12:03 +0200 + + Docs: Update TODO a little + + TODO | 22 ++++------------------ + 1 file changed, 4 insertions(+), 18 deletions(-) + +commit f5aa292c534f87b9dd588e667d1c65ed31e5f289 +Author: Lasse Collin +Date: 2025-02-07 18:50:56 +0200 + + Add researcher credits of CVE-2022-1271 and CVE-2024-47611 to THANKS + + These are specific phrases that were included in the advisories and + NEWS. It's nice to have them in THANKS as well. + + THANKS | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 7cf463b5add70e3fb48a10de3965c8beb6c01ad9 +Author: Lasse Collin +Date: 2025-02-07 18:43:00 +0200 + + Update THANKS + + THANKS | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 6b7fe7e27b77038592e2c2e31df955059dda7d1d +Author: Lasse Collin +Date: 2025-02-04 14:12:46 +0200 + + Docs: Update the "Translations" section in README + + Make it clearer that translations cannot be accepted if they don't + come via the Translation Project. + + Column headings have been handled automatically for years and now --help + is autowrapped too, so the related instructions can be removed. + + README | 107 ++++++++++++++++++++++++----------------------------------------- + 1 file changed, 39 insertions(+), 68 deletions(-) + +commit 2c7aee94936babf84b61b55420e503a0b2629ec1 +Author: Lasse Collin +Date: 2025-02-04 13:23:53 +0200 + + debug/translations.bash: Revise a little + + Make it work for out-of-tree builds without requiring one to specify + the location of the xz executable. + + Add xz --filters-help. + + Make the output shorter by reducing the number of xz -lvv test files. + + Show the value of LANGUAGE environment variable. + + Show the xz.git version using git describe --abbrev=8 instead of =4. + + debug/translation.bash | 24 +++++++++++------------- + 1 file changed, 11 insertions(+), 13 deletions(-) + +commit c6b15e7045209002bbbf4979c48072af01c20d8d +Author: Lasse Collin +Date: 2025-02-04 13:20:52 +0200 + + Build: Use "git describe --abbrev=8" in snapshot tarball names + + 8 is more likely to be reproducible than the old 4 without being + excessively long for a small repository like this. + + Makefile.am | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 0ce97987c5b27cfb6f98984e5fd7477880e0cf33 +Author: Lasse Collin +Date: 2025-02-04 19:37:17 +0200 + + Update THANKS + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit 353c33355cb12e5016d49052fd1e90d15568aa37 +Author: Lasse Collin +Date: 2025-02-03 16:29:31 +0200 + + Translations: Update the Serbian translation + + po/sr.po | 805 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 458 insertions(+), 347 deletions(-) + +commit 887dc281885052bced32b3aa309506ea58a2e78e +Author: Lasse Collin +Date: 2025-02-03 16:15:38 +0200 + + Translations: Update Chinese (traditional) translation + + Since there are no spaces between words, the unsophisticated automatic + word wrapping code needs some help. Compared to the version in the + Translation Project, I added a few \t characters which the word + wrapping code interprets as zero width spaces (hopefully they are + placed correctly). These edits can be seen with this command: + + grep -v ^# po/zh_TW.po | grep --color -F '\t' + + po/zh_TW.po | 843 +++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 471 insertions(+), 372 deletions(-) + +commit 0f1454cf5f460a4095f47f8f73f5a290e9777d7f +Author: Lasse Collin +Date: 2025-02-03 16:12:44 +0200 + + Update THANKS + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit 23ea031820086d302a213be005a091df763b8a7b +Author: Lasse Collin +Date: 2025-02-02 14:15:07 +0200 + + Build: Update posix-shell.m4 from Gnulib + + Tabs have been converted to spaces and a "serial" number has been + added. The previous version was from 2008/2009. There are no functional + changes since then but now it's clearer that the copy in XZ Utils + isn't outdated. + + The new file was picked from the Gnulib commit + 81a4c1e3b7692e95c0806d948cbab9148ad85ef2. A later commit adds + a warranty disclaimer to the license, which obviously is fine, + but I didn't find a SPDX license identifier for the new license, + so for simplicity I used the earlier commit. + + m4/posix-shell.m4 | 31 ++++++++++++++++--------------- + 1 file changed, 16 insertions(+), 15 deletions(-) + +commit 84c33c0384aa4604ff7956f2fae6f83ea60ba96b +Author: Lasse Collin +Date: 2025-02-02 12:51:03 +0200 + + Build: Check for -fsanitize= also in $CC + + People may put -fsanitize in CC instead of CFLAGS so check both. + Landlock sandbox isn't compatible with sanitizers so it's nice + to catch the incompatible options at configure time. + + Don't attempt to do the same in CMakeLists.txt; the check for + CMAKE_C_FLAGS / CFLAGS shall be enough there. The extra flags from + the CC environment variable go into the undocumented internal variable + CMAKE_C_COMPILER_ARG1 (all flags from CC go into that same variable). + Peeking the internal variable merely for improved diagnostics isn't + worth it. + + Fixes: 88588b1246d8c26ffbc138b3e5c413c5f14c3179 + + configure.ac | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit a7304ea4a7daede9789a8fe422b714e372737120 +Author: Lasse Collin +Date: 2023-09-26 19:11:20 +0300 + + Build: Remove the FIXME about -Werror checks + + configure.ac | 7 ------- + 1 file changed, 7 deletions(-) + +commit 1780bba74075da5e7764615bd323e95e19057dee +Author: Lasse Collin +Date: 2023-09-26 19:10:51 +0300 + + Build: If using a GCC compatible compiler, ensure that -Werror works + + The check can be skipped by passing SKIP_WERROR_CHECK=yes to configure. + It won't be documented anywhere else than in the error message. + + Ways to test: + + ./configure CC=gcc CFLAGS=-Wunused-macros + ./configure CC=clang CFLAGS=-Weverything + ./configure CC=clang CFLAGS=-Weverything SKIP_WERROR_CHECK=yes + + configure.ac | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +commit 3aca2daefbdedd7cc0fb75ddde6b714273b1cc1d +Author: Lasse Collin +Date: 2025-02-02 14:30:15 +0200 + + Update THANKS + + THANKS | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 186ff78ab40ceb07cde139506cab42a927ca99d2 +Author: Lasse Collin +Date: 2025-02-01 12:49:09 +0200 + + Translations: Update Romanian translation + + po/ro.po | 12 ++++++------ + 1 file changed, 6 insertions(+), 6 deletions(-) + +commit 40a8ce3e10747ca5233610cc2cb704fc303c48e4 +Author: Lasse Collin +Date: 2025-01-30 18:16:43 +0200 + + Translations: Update Korean man page translations + + po4a/ko.po | 146 ++++++++++++++++++++++++------------------------------------- + 1 file changed, 56 insertions(+), 90 deletions(-) + +commit 1787f9bd18ea8798d64b636cdefe6d0fda9b8f72 +Author: Lasse Collin +Date: 2025-01-30 18:15:52 +0200 + + Translations: Add Italian man page translations + + po4a/it.po | 3876 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + po4a/po4a.conf | 2 +- + 2 files changed, 3877 insertions(+), 1 deletion(-) + +commit 9b9182e561787a811fc0178489589f28c3e0174c +Author: Lasse Collin +Date: 2025-01-29 22:18:29 +0200 + + Translations: Update the Finnish translation + + po/fi.po | 13 +++++++------ + 1 file changed, 7 insertions(+), 6 deletions(-) + +commit 7d73ff7a9d8eab6270f0b1ff7d10c0aa6f5ba53f +Author: Lasse Collin +Date: 2025-01-29 20:50:03 +0200 + + lzmainfo: Use tuklib_mbstr_wrap for --help text + + Some languages have so long strings that they need to be wrapped. + + CMakeLists.txt | 4 ++++ + src/lzmainfo/Makefile.am | 2 ++ + src/lzmainfo/lzmainfo.c | 36 ++++++++++++++++++++++++++---------- + 3 files changed, 32 insertions(+), 10 deletions(-) + +commit c56eb4707627d700695813fccdddd1483eac4f21 +Author: Lasse Collin +Date: 2025-01-29 20:00:06 +0200 + + Translations: Update the Croatian translation + + po/hr.po | 926 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 529 insertions(+), 397 deletions(-) + +commit 69f4aec0a2442ab81f9ab66e5871a6546aefb0fc +Author: Lasse Collin +Date: 2025-01-29 19:56:01 +0200 + + Translations: Update the Finnish translation + + po/fi.po | 911 +++++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 483 insertions(+), 428 deletions(-) + +commit d49dde33cf5f488bb38b1f57e172c4e3343fb383 +Author: Lasse Collin +Date: 2025-01-29 19:55:27 +0200 + + Translations: Update the German man page translations + + po4a/de.po | 147 +++++++++++++++++++++++-------------------------------------- + 1 file changed, 55 insertions(+), 92 deletions(-) + +commit 23b99fc4a1f35bec5d63ffd02b14cacbdce9fe3c +Author: Lasse Collin +Date: 2025-01-29 19:55:17 +0200 + + Translations: Update the German translation + + po/de.po | 825 +++++++++++++++++++++++++++++++++++---------------------------- + 1 file changed, 460 insertions(+), 365 deletions(-) + +commit 7edab2bde0606b42229d9c04fe664069e38de3fb +Author: Lasse Collin +Date: 2025-01-29 19:55:05 +0200 + + Translations: Update the Turkish translation + + po/tr.po | 892 +++++++++++++++++++++++++++++++++++---------------------------- + 1 file changed, 490 insertions(+), 402 deletions(-) + +commit fac4d0fa5277d7a1f621707621ee9516f0bdbac5 +Author: Lasse Collin +Date: 2025-01-29 19:54:36 +0200 + + Translations: Add the Dutch translation + + po/LINGUAS | 1 + + po/nl.po | 1268 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 1269 insertions(+) + +commit abe5092f24b55dde9f7f78fac1bf810bce173273 +Author: Lasse Collin +Date: 2025-01-29 19:53:50 +0200 + + Translations: Update the Georgian translation + + po/ka.po | 153 +++++++++++++++++++++++++++++++++++++++++++++++---------------- + 1 file changed, 115 insertions(+), 38 deletions(-) + +commit b97b23c78d8100eec363c3e999c511560366d347 +Author: Lasse Collin +Date: 2025-01-29 19:53:21 +0200 + + Translations: Update the Spanish translation + + po/es.po | 824 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 450 insertions(+), 374 deletions(-) + +commit c68318cb49e0562bd22e88724ce85e76c6789a3a +Author: Lasse Collin +Date: 2025-01-29 19:53:06 +0200 + + Translations: Update the Korean translation + + po/ko.po | 785 +++++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 460 insertions(+), 325 deletions(-) + +commit 153ee17f635962a474499f786ea1de1e1a2bb276 +Author: Lasse Collin +Date: 2025-01-29 19:52:42 +0200 + + Translations: Update the Romanian man page translations + + po4a/ro.po | 141 +++++++++++++++++++++++-------------------------------------- + 1 file changed, 54 insertions(+), 87 deletions(-) + +commit 6ed308197e1f9d6c7a5cfe5aae301e75544017c4 +Author: Lasse Collin +Date: 2025-01-29 19:51:59 +0200 + + Translations: Update the Romanian translation + + po/ro.po | 818 +++++++++++++++++++++++++++++++++++---------------------------- + 1 file changed, 461 insertions(+), 357 deletions(-) + +commit 06028803e19219f642aa9abddd3525c43594ec6c +Author: Lasse Collin +Date: 2025-01-29 19:50:50 +0200 + + Translations: Update the Ukrainian man page translations + + po4a/uk.po | 142 +++++++++++++++++++++++-------------------------------------- + 1 file changed, 54 insertions(+), 88 deletions(-) + +commit 8cbaf896a65a53c1d1e7e2ffc80d6ea216b1e8df +Author: Lasse Collin +Date: 2025-01-29 19:50:26 +0200 + + Translations: Update the Ukrainian translation + + po/uk.po | 813 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 460 insertions(+), 353 deletions(-) + +commit 81c352907b8048b97d9868947026701a49f377ef +Author: Lasse Collin +Date: 2025-01-29 19:48:43 +0200 + + Translations: Update the Swedish translation + + po/sv.po | 847 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 462 insertions(+), 385 deletions(-) + +commit 999ce263718a52ba74245c3e2a416ab11494d1b1 +Author: Lasse Collin +Date: 2025-01-28 16:33:32 +0200 + + tuklib_physmem: Clean up disabled code + + src/common/tuklib_physmem.c | 9 +-------- + 1 file changed, 1 insertion(+), 8 deletions(-) + +commit 4d7e7c9d94f7a5ad4931a5bbd6ed9d00173fa1ab +Author: Lasse Collin +Date: 2025-01-28 16:28:18 +0200 + + Windows: Avoid an error message on broken pipe + + Also make xz not process more input files after a broken pipe has + been detected. This matches the behavior on POSIX. If all files + are being written to standard output, trying with the next file is + pointless when it's known that standard output won't accept more data. + + xzdec already stopped after the first error. It does so with all + errors, so it differs from xz: + + $ xz -dc not_found_1 not_found_2 + xz: not_found_1: No such file or directory + xz: not_found_2: No such file or directory + + $ xzdec not_found_1 not_found_2 + xzdec: not_found_1: No such file or directory + + Reported-by: Vincent Torri + + src/xz/file_io.c | 13 +++++++++++++ + src/xzdec/xzdec.c | 11 ++++++++++- + 2 files changed, 23 insertions(+), 1 deletion(-) + +commit 95b638480aa8203e547c709c651f421c22db1718 +Author: Lasse Collin +Date: 2025-01-23 19:59:17 +0200 + + doc/SHA256SUMS: Add 5.6.4 and 5.7.1alpha + + doc/SHA256SUMS | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit cdae0df31e4c2dfb1e885941cd1998e5a2b6e39d +Author: Lasse Collin +Date: 2025-01-23 11:50:42 +0200 + + Bump version and soname for 5.7.1alpha + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 4 files changed, 4 insertions(+), 4 deletions(-) + +commit 4d2af2c43bae25ef4ef9cd88304471d4859aa322 +Author: Lasse Collin +Date: 2025-01-23 11:48:43 +0200 + + Translations: Run po4a/update-po + + po4a/de.po | 64 +++++++++++++++++++++++++++++++++++++++++++++++++---------- + po4a/fr.po | 57 +++++++++++++++++++++++++++++++++++++++++++++++----- + po4a/ko.po | 64 +++++++++++++++++++++++++++++++++++++++++++++++++---------- + po4a/pt_BR.po | 57 +++++++++++++++++++++++++++++++++++++++++++++++----- + po4a/ro.po | 64 +++++++++++++++++++++++++++++++++++++++++++++++++---------- + po4a/uk.po | 64 +++++++++++++++++++++++++++++++++++++++++++++++++---------- + 6 files changed, 320 insertions(+), 50 deletions(-) + +commit ff0b825505e60e21b32e33c42f551c8f34ba393f +Author: Lasse Collin +Date: 2025-01-23 11:40:46 +0200 + + Add NEWS for 5.7.1alpha + + NEWS | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 107 insertions(+) + +commit f6cd3e3bfc8d1f5a76dd55170968bf4582b95baf +Author: Lasse Collin +Date: 2025-01-23 11:40:46 +0200 + + Add NEWS for 5.6.4 + + NEWS | 45 +++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 45 insertions(+) + +commit b3af3297e4d6cf0eafb48155aa97bb06c82a9228 +Author: Lasse Collin +Date: 2025-01-23 11:40:46 +0200 + + NEWS: The security fix in 5.6.3 is known as CVE-2024-47611 + + NEWS | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit a04b9dd0c7c74fabd8c393d2dc68a221276d6e29 +Author: Lasse Collin +Date: 2025-01-22 16:55:09 +0200 + + windows/build.bash: Fix error message + + Fixes: 1ee716f74085223c8fbcae1d5a384e6bf53c0f6a + + windows/build.bash | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 4eae859ae8ad7072eaa74aeaee79a2c3c12c55cb +Author: Lasse Collin +Date: 2025-01-22 15:03:55 +0200 + + Windows: Disable MinGW-w64's stdio functions in size-optimized builds + + This only affects builds with UCRT. With legacy MSVCRT, the replacement + functions are always enabled. + + Omitting the MinGW-w64 replacements saves over 20 KiB per executable. + The downside is that --enable-small or XZ_SMALL=ON disables thousand + separator support in xz messages. If someone is OK with the slower + speed of slightly smaller builds, lack of thousand separators won't + matter. + + Don't override __USE_MINGW_ANSI_STDIO if it is already defined (via + CPPFLAGS or such method). + + src/common/sysdefs.h | 30 +++++++++++++++++++++--------- + src/xz/util.c | 6 +++++- + 2 files changed, 26 insertions(+), 10 deletions(-) + +commit a831bc185bdd44c06847eae8df2d35cc281f65da +Author: Lasse Collin +Date: 2025-01-20 16:44:27 +0200 + + liblzma: Add raw ARM64, RISC-V, and x86 BCJ filter APIs + + Put them behind the LZMA_UNSTABLE macro for now. + + These low-level special APIs might become useful in erofs-utils. + + src/liblzma/api/lzma/bcj.h | 99 +++++++++++++++++++++++++++++++++++++++++ + src/liblzma/common/common.h | 2 + + src/liblzma/liblzma_generic.map | 10 +++++ + src/liblzma/liblzma_linux.map | 10 +++++ + src/liblzma/simple/arm64.c | 18 ++++++++ + src/liblzma/simple/riscv.c | 18 ++++++++ + src/liblzma/simple/x86.c | 24 ++++++++++ + 7 files changed, 181 insertions(+) + +commit 6f5cdd4534faf7db4b6c123651d6a606bc59b98c +Author: Lasse Collin +Date: 2025-01-20 16:31:49 +0200 + + xz: Unify a few strings with liblzma + + Avoid having both "%s: foo" and "foo" as translatable strings + so that translators don't need to handle it twice. + + src/xz/options.c | 11 ++++++----- + src/xz/util.c | 4 ++-- + 2 files changed, 8 insertions(+), 7 deletions(-) + +commit 713fdaa8b06a83f18b06811aba7b9bd7b7cbf1cb +Author: Lasse Collin +Date: 2025-01-20 16:31:49 +0200 + + xz: Translate error messages from lzma_str_to_filters() + + liblzma doesn't use gettext but the messages are included in xz.pot, + so xz can translate the messages. + + src/xz/coder.c | 9 +++------ + 1 file changed, 3 insertions(+), 6 deletions(-) + +commit f2e2b267cab8d7aa0b0a58c325546ee5070c0028 +Author: Lasse Collin +Date: 2025-01-20 16:31:49 +0200 + + liblzma: Mark string conversion messages as translatable + + po/POTFILES.in | 1 + + src/liblzma/common/string_conversion.c | 96 ++++++++++++++++++++-------------- + 2 files changed, 59 insertions(+), 38 deletions(-) + +commit f49d7413d9a0d480ded6d448c1ef7475ae6cd1c9 +Author: Lasse Collin +Date: 2025-01-20 16:31:35 +0200 + + liblzma: Tweak a few error messages in lzma_str_to_filters() + + src/liblzma/common/string_conversion.c | 9 +++++---- + 1 file changed, 5 insertions(+), 4 deletions(-) + +commit da359c360e986b21cd8d7b888c6a80f56b9d49c7 +Author: Lasse Collin +Date: 2025-01-19 20:11:54 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit f032373561cefaf07f92ffe3fbc471ec6770456e +Author: Lasse Collin +Date: 2025-01-19 19:40:32 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 51f038f8cbd5d8a95954c05bfcbbc32f2a313615 +Author: Lasse Collin +Date: 2025-01-13 08:44:58 +0200 + + liblzma: memcmplen.h: Use 8-byte method on 64-bit unaligned archs + + Previously it was enabled only on x86-64 and ARM64 when also support + for unaligned access was detected or manually enabled at built time. + + In the default build configuration, the 8-byte method is now enabled + also on 64-bit RISC-V and 64-bit PowerPC (both endiannesses). It was + reported that on big endian POWER9, encoding time may reduce 12-13 %. + + This change only affects builds with GCC and Clang because the code + uses __builtin_ctzll or __builtin_clzll. + + Thanks to Marcus Comstedt for testing on POWER9. + + src/liblzma/common/memcmplen.h | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + +commit 96336b0110d47756a9fd2a103fbf0a99e905fbed +Author: Lasse Collin +Date: 2025-01-12 13:06:17 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 150356207c8d6a3e0af465b676430d19d62f884c +Author: Lasse Collin +Date: 2025-01-12 12:59:20 +0200 + + liblzma: Fix the encoder breakage on big endian ARM64 + + When the 8-byte method was enabled for ARM64, a check for endianness + wasn't added. This broke the LZMA/LZMA2 encoder. Test suite caught it. + + Fixes: cd64dd70d5665b6048829c45772d08606f44672e + Co-authored-by: Marcus Comstedt + + src/liblzma/common/memcmplen.h | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +commit b01b0958025a2da284b53a583f313f8140636cb5 +Author: Lasse Collin +Date: 2025-01-12 11:04:27 +0200 + + Windows: Update manifest comments about long UTF-8 filenames + + src/common/w32_application.manifest.comments.txt | 23 +++++++++++++++-------- + 1 file changed, 15 insertions(+), 8 deletions(-) + +commit 0dfc67d37ebb038be8a9b17b536d1b561d52e81a +Author: Lasse Collin +Date: 2025-01-12 10:47:58 +0200 + + Windows: Update build.bash and its README-Windows.txt to UCRT + + While MSVCRT builds are possible, UCRT works better with UTF-8. + A 32-bit build is included still but hopefully it's not actually + needed anymore. + + windows/README-Windows.txt | 17 ++++++++--------- + windows/build.bash | 20 ++++++++++++++------ + 2 files changed, 22 insertions(+), 15 deletions(-) + +commit 7b3eb2db6c4ba24b5eb438e58ab1ca57e14e59c2 +Author: Lasse Collin +Date: 2025-01-10 13:11:40 +0200 + + Translations: Update Serbian translation + + I rewrapped a few overlong lines. Those edits aren't in the + Translation Project. Automatic wrapping in the master branch + means that these strings need to be updated soon anyway. + + po/sr.po | 346 ++++++++++++++++++++++----------------------------------------- + 1 file changed, 121 insertions(+), 225 deletions(-) + +commit 950da11ce09c90412dcbca29689575037640667a +Author: Lasse Collin +Date: 2025-01-08 19:26:29 +0200 + + Build: Use --sort=name in TAR_OPTIONS + + Use also LC_COLLATE=C to make the sorting locale-independent. + Sorting makes the file order reproducible. + + Makefile.am | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 75d91d6b39ea3e2fae8f027dcec01be2dca9594d +Author: Lasse Collin +Date: 2025-01-08 19:08:08 +0200 + + xz: Workaround broken O_SEARCH in musl + + Testing with musl 1.2.5 and Linux 6.12, O_SEARCH doesn't result + in a file descriptor that works with fsync() although it should work. + See the added comment. + + The same issue affected gzip --synchronous: + + https://bugs.gnu.org/75405 + + Thanks to Paul Eggert. + + src/xz/file_io.c | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit ea92eae122a3ccefa61087f84fd99b417fc9ee3c +Author: Lasse Collin +Date: 2025-01-07 21:34:33 +0200 + + Revert "xz: O_SEARCH cannot be used for fsync()" + + This reverts commit 4014e2479c7b0273f15bd0c9c017c5fe859b0d8f. + + POSIX-conforming O_SEARCH should allow fsync(). + + src/xz/file_io.c | 21 +++++++++++---------- + 1 file changed, 11 insertions(+), 10 deletions(-) + +commit 4014e2479c7b0273f15bd0c9c017c5fe859b0d8f +Author: Lasse Collin +Date: 2025-01-05 21:43:11 +0200 + + xz: O_SEARCH cannot be used for fsync() + + Opening a directory with O_SEARCH results in a file descriptor that can + be used with functions like openat(). Such a file descriptor cannot be + used with fsync(). Use O_RDONLY instead. + + In musl, O_SEARCH becomes Linux-specific O_PATH. A file descriptor + from O_PATH doesn't allow fsync(). + + Seems that it's not possible to fsync() a directory that has write + and search permissions but not read permission. + + Fixes: 2a9e91d796d091740489d951fa7780525e4275f1 + + src/xz/file_io.c | 21 ++++++++++----------- + 1 file changed, 10 insertions(+), 11 deletions(-) + +commit ad2b57cb477b753293c25a01fc24c7f84ee523c2 +Author: Lasse Collin +Date: 2025-01-05 20:48:28 +0200 + + CI: Make ctest show errors from failed tests + + build-aux/ci_build.bash | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit c405264c031aceaf68dfd1546d6337afcebd48e5 +Author: Lasse Collin +Date: 2025-01-05 20:14:49 +0200 + + tuklib_mbstr_nonprint: Preserve the value of errno + + A typical use case is like this: + + printf("%s: %s\n", tuklib_mask_nonprint(filename), strerror(errno)); + + tuklib_mask_nonprint() may call mbrtowc() and malloc() which may modify + errno. If errno isn't preserved, the error message might be wrong if + a compiler decides to call tuklib_mask_nonprint() before strerror(). + + Fixes: 40e573305535960574404d2eae848b248c95ea7e + + src/common/tuklib_mbstr_nonprint.c | 17 ++++++++++++++--- + src/common/tuklib_mbstr_nonprint.h | 4 +++- + 2 files changed, 17 insertions(+), 4 deletions(-) + +commit 2a9e91d796d091740489d951fa7780525e4275f1 +Author: Lasse Collin +Date: 2025-01-05 20:14:49 +0200 + + xz: Use fsync() before deleting the input file, and add --no-sync + + xz's default behavior is to delete the input file after successful + compression or decompression (unless writing to standard output). + If the system crashes soon after the deletion, it is possible that + the newly written file has not yet hit the disk while the previous + delete operation might have. In that case neither the original file + nor the written file is available. + + Call fsync() on the file. On POSIX systems, sync also the directory + where the file was created. + + Add a new option --no-sync which disables fsync() usage. It can avoid + a (possibly significant) performance penalty when processing many + small files. It's fine to use --no-sync when one knows that the files + are easy to recreate or restore after a system crash. + + Using fsync() after every flush initiated by --flush-timeout was + considered. It wasn't implemented at least for now. + + - --flush-timeout is typically used when writing to stdout. If stdout + is a file, xz cannot (portably) sync the directory of the file. + One would need to create the output file first, sync the directory, + and then run xz with fsync() enabled. + + - If xz --flush-timeout output goes to a file, it's possible to use + a separate script to sync the file, for example, once per minute + while telling xz to flush more frequently. + + - Not supporting syncing with --flush-timeout was simpler. + + Portability notes: + + - On systems that lack O_SEARCH (like Linux), "xz dir/file" will now + fail if "dir" cannot be opened for reading. If "dir" still has + write and search permissions (like d-wx------ in "ls -l"), + previously xz would have been able to compress "dir/file" still. + Now it only works if using --no-sync (or --keep or --stdout). + + - and dirname() should be available on all POSIX systems, + and aren't needed on non-POSIX systems. + + - fsync() is available on all POSIX systems. The directory syncing + could be changed to fdatasync() although at least on ext4 it + doesn't seem to make a performance difference in xz's usage. + fdatasync() would need a build system check to support (old) + special cases, for example, MINIX 3.3.0 doesn't have fdatasync() + and Solaris 10 needs -lrt. + + - On native Windows, _commit() is used to replace fsync(). Directory + syncing isn't done and shouldn't be needed. (In Cygwin, fsync() on + directories is a no-op.) + + - DJGPP has fsync() for files. ;-) + + Using fsync() was considered somewhere around 2009 and again in 2016 but + those times the idea was rejected. For comparison, GNU gzip 1.7 (2016) + added the option --synchronous which enables fsync(). + + Co-authored-by: Sebastian Andrzej Siewior + Fixes: https://bugs.debian.org/814089 + Link: https://www.mail-archive.com/xz-devel@tukaani.org/msg00282.html + Closes: https://github.com/tukaani-project/xz/pull/151 + + src/xz/args.c | 14 ++++++ + src/xz/args.h | 2 +- + src/xz/file_io.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- + src/xz/file_io.h | 6 +++ + src/xz/message.c | 3 ++ + src/xz/sandbox.c | 5 ++- + src/xz/xz.1 | 24 ++++++++++- + 7 files changed, 177 insertions(+), 6 deletions(-) + +commit 2e28c7145747b3287283f13c9d2becd73a7c4a1f +Author: Lasse Collin +Date: 2024-12-27 09:15:50 +0200 + + xz: Use "goto" for error handling in io_open_dest_real() + + src/xz/file_io.c | 20 +++++++++----------- + 1 file changed, 9 insertions(+), 11 deletions(-) + +commit 75107217670a97b7b772833669d88c3c2f188e37 +Author: Lasse Collin +Date: 2025-01-05 12:10:05 +0200 + + liblzma: Always validate the first digit of a preset string + + lzma_str_to_filters() may call parse_lzma12_preset() in two ways. The + call from str_to_filters() detects the string type from the first + character(s) and as a side-effect it validates the first digit of + the preset string. So this change makes no difference there. + + However, the call from parse_options() doesn't pre-validate the string. + parse_lzma12_preset() will return an invalid value which is passed to + lzma_lzma_preset() which safely rejects it. The bug still affects the + the error message: + + $ xz --filters=lzma2:preset=X + xz: Error in --filters=FILTERS option: + xz: lzma2:preset=X + xz: ^ + xz: Unsupported preset + + After the fix: + + $ xz --filters=lzma2:preset=X + xz: Error in --filters=FILTERS option: + xz: lzma2:preset=X + xz: ^ + xz: Unsupported preset + + The ^ now correctly points to the X and not past it because the X itself + is the problematic character. + + Fixes: cedeeca2ea6ada5b0411b2ae10d7a859e837f203 + + src/liblzma/common/string_conversion.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 52ff32433734d03befd85a5bf00fba77d6501455 +Author: Lasse Collin +Date: 2025-01-05 11:40:34 +0200 + + xz: Fix getopt_long argument type in --filters* + + Forgetting the argument (or not using = to separate the option from + the argument) resulted in lzma_str_to_filters() being called with NULL + as input string argument. The function handles it fine but xz passes + the NULL to printf() too: + + $ xz --filters + xz: Error in --filters=FILTERS option: + xz: (null) + xz: ^ + xz: Unexpected NULL pointer argument(s) to lzma_str_to_filters() + + Now it's correct: + + $ xz --filters + xz: option '--filters' requires an argument + + The --filters-help option doesn't take any arguments. + + Fixes: 9ded880a0221f4d1256845fc4ab957ffd377c760 + Fixes: d6af7f347077b22403133239592e478931307759 + Fixes: a165d7df1964121eb9df715e6f836a31c865beef + + src/xz/args.c | 22 +++++++++++----------- + 1 file changed, 11 insertions(+), 11 deletions(-) + +commit 2655c81b5e92278b0fd51f6537c1116f8349b02a +Author: Lasse Collin +Date: 2025-01-04 20:04:56 +0200 + + xzdec: Don't leave Landlock file descriptor open for no reason + + This fix is similar to 48ff3f06521ca326996ab9a04d1b342098960427. + + Fixes: d74fb5f060b76db709b50f5fd37490394e52f975 + + src/xzdec/xzdec.c | 2 ++ + 1 file changed, 2 insertions(+) + +commit 35df4c2bc0500e60ba9d0d163d37a6d110d6841e +Author: Lasse Collin +Date: 2025-01-04 20:02:18 +0200 + + xz: Make --single-stream imply --keep + + Suggested by xx on #tukaani on 2024-04-12. + + src/xz/args.c | 3 +++ + src/xz/xz.1 | 9 ++++++++- + 2 files changed, 11 insertions(+), 1 deletion(-) + +commit 6f412814a8019700248229ce972530159a0d9872 +Author: Lasse Collin +Date: 2025-01-04 19:57:07 +0200 + + Update AUTHORS + + The contributions have been rewritten. + + AUTHORS | 2 +- + src/liblzma/check/crc32_arm64.h | 1 - + src/liblzma/check/crc32_fast.c | 1 - + src/liblzma/check/crc_common.h | 1 - + 4 files changed, 1 insertion(+), 4 deletions(-) + +commit 5651d153031a7ee2581cdba9bff658031826cb50 +Author: Lasse Collin +Date: 2025-01-04 15:02:16 +0200 + + xz: Avoid printf formats like %2$s + + It's a POSIX feature that isn't in standard C. It's not available on + Windows. Even MinGW-w64 with __USE_MINGW_ANSI_STDIO doesn't support + it even though it supports POSIX %'d for thousand separators. + + Gettext's provides overrides for printf and other functions + which do support the %2$s formats. Translations use them. But xz should + work on Windows without too. + + Fixes: 3e9177fd206d20d6d8acc7d203c25a9ae0549229 + + src/xz/message.c | 51 ++++++++++++++++++++++++++++++++------------------- + 1 file changed, 32 insertions(+), 19 deletions(-) + +commit 63b246c90e7677c617faab1d3f6fc5c643b5e7cf +Author: Lasse Collin +Date: 2025-01-04 14:41:37 +0200 + + tuklib_mbstr_wrap: Add printf format attribute + + It's supported by GCC 3.x already. + + src/common/tuklib_common.h | 7 +++++++ + src/common/tuklib_mbstr_wrap.h | 1 + + 2 files changed, 8 insertions(+) + +commit a7313c01d9b8db71ffb61dc1dd7c4ea928824b4b +Author: Lasse Collin +Date: 2025-01-04 13:44:12 +0200 + + xz: Translate a Windows-specific string + + Originally I thought that native Windows builds wouldn't be translated + but nowadays at least MSYS2 ships such binaries. + + src/xz/file_io.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 00eb6073c088be9e7516dfc00a13ef520827b57c +Author: Lasse Collin +Date: 2025-01-02 15:32:10 +0200 + + xz: Use my_landlock.h + + A slightly silly thing is that xz may now query the ABI version up to + three times. We could call my_landlock_ruleset_attr_forbid_all() only + once and cache the result but it didn't seem worth doing. + + CMakeLists.txt | 1 + + src/xz/sandbox.c | 72 ++++++++++---------------------------------------------- + 2 files changed, 13 insertions(+), 60 deletions(-) + +commit 0fc5a625d7cc4ad51fde9367de088b9ad3bd40f6 +Author: Lasse Collin +Date: 2025-01-02 15:32:10 +0200 + + xzdec: Use my_landlock.h + + CMakeLists.txt | 1 + + src/xzdec/xzdec.c | 34 ++++++---------------------------- + 2 files changed, 7 insertions(+), 28 deletions(-) + +commit 38cb8ec9fd70d25fca6b473de44cf61586238552 +Author: Lasse Collin +Date: 2025-01-02 15:32:10 +0200 + + Add my_landlock.h with helper functions to use Linux Landlock + + This supports up to Landlock ABI version 6. The current code in + xz and xzdec only support up to ABI version 4. + + src/Makefile.am | 1 + + src/common/my_landlock.h | 141 +++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 142 insertions(+) + +commit 672da29bb3a209a727ae46c0df948d7eea69f2e2 +Author: Lasse Collin +Date: 2025-01-01 18:46:50 +0200 + + liblzma: Silence warnings from "clang -Wimplicit-fallthrough" + + src/liblzma/lzma/lzma_decoder.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 1a8a1ad9a1e3179ce267baa551fb17b30624b4dd +Author: Lasse Collin +Date: 2025-01-01 15:34:51 +0200 + + Build: Use -Wimplicit-fallthrough=5 when supported + + Now that we have the FALLTHROUGH macro, use the strictest mode with + GCC so that comment-based fallthrough markings are no longer accepted. + + In GCC, -Wextra includes -Wimplicit-fallthrough=3 and + -Wimplicit-fallthrough is the same as -Wimplicit-fallthrough=3. + Thus, the strict mode requires specifying -Wimplicit-fallthrough=5. + + Clang has -Wimplicit-fallthrough which is *not* enabled by -Wextra. + Clang doesn't have a variant that takes an argument. Thus we need + to check for -Wimplicit-fallthrough. Do it before checking for + -Wimplicit-fallthrough=5 so that the latter overrides the former + when using GCC. + + CMakeLists.txt | 2 ++ + configure.ac | 2 ++ + 2 files changed, 4 insertions(+) + +commit 94adc996e45cc5cad9352cc3271d3a1a2f5c4c22 +Author: Lasse Collin +Date: 2025-01-01 15:30:50 +0200 + + Replace "Fall through" comments with FALLTHROUGH + + src/liblzma/common/alone_decoder.c | 3 +-- + src/liblzma/common/auto_decoder.c | 5 ++--- + src/liblzma/common/block_decoder.c | 6 ++---- + src/liblzma/common/block_encoder.c | 6 ++---- + src/liblzma/common/common.c | 2 +- + src/liblzma/common/file_info.c | 22 +++++++++------------- + src/liblzma/common/index_decoder.c | 9 +++------ + src/liblzma/common/index_encoder.c | 6 ++---- + src/liblzma/common/index_hash.c | 7 +++---- + src/liblzma/common/lzip_decoder.c | 14 +++++--------- + src/liblzma/common/stream_decoder.c | 16 ++++++---------- + src/liblzma/common/stream_decoder_mt.c | 25 +++++++++---------------- + src/liblzma/common/stream_encoder_mt.c | 10 ++++------ + src/liblzma/lzma/lzma2_encoder.c | 9 +++------ + src/xz/args.c | 2 +- + src/xz/list.c | 3 +-- + 16 files changed, 54 insertions(+), 91 deletions(-) + +commit f31c3a6647b5a5d056324a9c83e6b2c940ebec22 +Author: Lasse Collin +Date: 2025-01-01 15:08:51 +0200 + + sysdefs.h: Add FALLTHROUGH macro + + src/common/sysdefs.h | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit e34dbd6a0ae7a560a5508d51fc0bd142c5a320dc +Author: Lasse Collin +Date: 2025-01-01 15:06:15 +0200 + + xzdec: Fix language in a comment + + src/xzdec/xzdec.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 16821252c504071f5c2012e415e59cbf5fb79820 +Author: Lasse Collin +Date: 2025-01-02 13:35:48 +0200 + + Windows: Make NLS require UCRT and gettext-runtime >= 0.23.1 + + Also remove the recently-added workaround from tuklib_gettext.h. + Requiring a new enough gettext-runtime is cleaner. I guess it's + mostly MSYS2 where xz is built with translation support, so once + MSYS2 has Gettext >= 0.23.1, this requirement shouldn't be a problem + in practice. + + CMakeLists.txt | 29 ++++++++++++++++++++++++++ + configure.ac | 29 ++++++++++++++++++++++++++ + src/common/tuklib_gettext.h | 51 --------------------------------------------- + 3 files changed, 58 insertions(+), 51 deletions(-) + +commit aa1807ed942579f700a08ab091b796cf04e31aec +Author: Lasse Collin +Date: 2025-01-02 11:52:17 +0200 + + windows/build-with-cmake.bat: Fix ENABLE_NLS to XZ_NLS + + Fixes: 29f77c7b707f2458fb047e77497354b195e05b14 + + windows/build-with-cmake.bat | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ea21c76aa2406ba06ac154fe57741734c04f260f +Author: Lasse Collin +Date: 2024-12-30 11:21:57 +0200 + + Build: Use git log --pretty=medium when creating ChangeLog + + It's the default in git-log. Specifying it explicitly is good in case + a user has set format.pretty to a different value. + + Makefile.am | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 08050c0788ce5bac0ffd572e9784a2749c4a13df +Author: Lasse Collin +Date: 2024-12-30 10:51:33 +0200 + + Windows: Update MinGW-w64 + CMake instructions to recommend UCRT + + windows/INSTALL-MinGW-w64_with_CMake.txt | 38 +++++++++++++++++++------------- + 1 file changed, 23 insertions(+), 15 deletions(-) + +commit 653732bd6f06d8f465bf353bf6e1c16f1405b906 +Author: Lasse Collin +Date: 2024-12-30 10:51:26 +0200 + + xz man page: Describe the source file deletion in -z and -d options + + The DESCRIPTION section always explained it, and the OPTIONS section + only described the differences to the default behavior. However, new + users in a hurry may skip reading DESCRIPTION. The default behavior + is a bit dangerous, thus it's good to repeat in --compress and + --decompress docs that source file is removed after successful operation. + + Fixes: https://github.com/tukaani-project/xz/issues/150 + + src/xz/xz.1 | 17 ++++++++++++++++- + 1 file changed, 16 insertions(+), 1 deletion(-) + +commit bb79f79b278fd4fb06a0bcd5ab3445c468f9baaf +Author: Lasse Collin +Date: 2024-12-27 21:52:28 +0200 + + Build: Set libtool -version-info so that it matches with CMake + + In the past, they haven't been in sync in development versions + although they (of course) have been in stable releases. + + src/liblzma/Makefile.am | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit cf54f70e14c218faf5019ffa2fa769ed73772ee8 +Author: Lasse Collin +Date: 2024-12-28 18:28:56 +0200 + + CMake/macOS: Use GNU Libtool compatible shared library versioning + + Because this increases the Mach-O compatibility_version, this commit + shouldn't cause any ABI compatibility trouble for existing CMake users + on macOS. This is assuming that they won't later downgrade to an older + liblzma version that was built with CMake before this commit. + + Meson allows customising the Mach-O versioning too. So the three + build systems can be configured to be compatible. + + CMakeLists.txt | 51 ++++++++++++++++++++++++++++++++++++++++++++++++--- + 1 file changed, 48 insertions(+), 3 deletions(-) + +commit 94e17916689d38bc09bf35e602ed6f6276034b59 +Author: Lasse Collin +Date: 2024-12-28 14:49:45 +0200 + + CMake: Edit a comment + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 6b50590725aeae8a2aed06faa3238cb9f8771c1b +Author: Lasse Collin +Date: 2024-12-28 20:39:49 +0200 + + version.sh: Omit an unwanted dot from development versions + + It printed 5.7.0.alpha instead of 5.7.0alpha. + + Fixes: e7a42cda7c827e016619e8cab15e2faf5d4181ae + + build-aux/version.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f7a248f56e94310a080051c4a709c08514fa48b1 +Author: Lasse Collin +Date: 2024-12-27 16:25:07 +0200 + + CMake: Remove a duplicate word from a comment + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 8b7c55d148f4a9b3702207164e862437ddffad33 +Author: Lasse Collin +Date: 2024-12-27 16:23:12 +0200 + + INSTALL: Document CMAKE_DLL_NAME_WITH_SOVERSION + + INSTALL | 19 +++++++++++++++++++ + 1 file changed, 19 insertions(+) + +commit 260d5d36203955a7148ae1ab05d0931c942028d5 +Author: Lasse Collin +Date: 2024-12-26 21:27:18 +0200 + + xz: Fix comments + + src/xz/file_io.c | 4 ++-- + src/xz/file_io.h | 4 ++-- + 2 files changed, 4 insertions(+), 4 deletions(-) + +commit bf6da9a573a780cd1a7fb1728ef55d09e58dad11 +Author: Dexter Castor Döpping +Date: 2024-12-22 13:44:03 +0100 + + CMake: Disable unity builds project-wide + + liblzma and xz can't be compiled as a unity/jumbo build because of + redeclarations and type name reuse. The CMake documentation recommends + setting UNITY_BUILD to false in this case. + + This is especially important if we're compiled as a subproject and the + consumer wants to use CMAKE_UNITY_BUILD=ON for the rest of their code + base. + + Closes: https://github.com/tukaani-project/xz/pull/158 + + CMakeLists.txt | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit f8c328eed1bf0a0168132025a52116b7735f894c +Author: Lasse Collin +Date: 2024-12-20 08:51:18 +0200 + + Windows: Workaround a UTF-8 issue in Gettext's libintl_setlocale() + + See the comment. In this package, locale is set at program startup and + not changed later, so the point (2) in the comment isn't a problem. + + Fixes: 46ee0061629fb075d61d83839e14dd193337af59 + + src/common/tuklib_gettext.h | 51 +++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 51 insertions(+) + +commit 03533906093529701ba91081907d8977991997de +Author: Lasse Collin +Date: 2024-12-20 06:50:36 +0200 + + Revert "Windows: Use UTF-8 locale when active code page is UTF-8" + + This reverts commit 0d0b574cc45045d6150d397776340c068df59e2a. + + src/common/tuklib_gettext.h | 32 ++------------------------------ + 1 file changed, 2 insertions(+), 30 deletions(-) + +commit 4b319e05afef4eab2fbafb6223f25d128ec99fce +Author: Lasse Collin +Date: 2024-12-19 18:31:09 +0200 + + xzdec: Use setlocale() instead of tuklib_gettext_setlocale() + + xzdec isn't translated and doesn't need libintl on Windows even + when NLS is enabled, thus libintl_setlocale() cannot interfere + with the locale settings. Thus, standard setlocale() works perfectly. + + In the commit 78868b6e, the explanation in the commit message is wrong. + + Fixes: 78868b6ed63fa4c89f73e3dfed27abfb8b0d46db + + src/xzdec/xzdec.c | 9 +++------ + 1 file changed, 3 insertions(+), 6 deletions(-) + +commit 34b80e282ea76ec793eaedaef58a36c3913dec78 +Author: Lasse Collin +Date: 2024-12-19 19:36:15 +0200 + + Windows: Revert the setlocale(LC_ALL, ".UTF8") documentation + + Only leave the FindFileFirstA() notes from 20dfca81, reverting + the incorrect setlocale() notes. On Windows, Gettext's + overrides setlocale() with libintl_setlocale() wrapper. I hadn't + noticed this, and thus my conclusions were wrong. + + Fixes: 20dfca8171dad4c64785ac61d5b68972c444877b + + src/common/w32_application.manifest.comments.txt | 21 +-------------------- + 1 file changed, 1 insertion(+), 20 deletions(-) + +commit 5794cda064ce980450eaa5a4e2c71bd317168ce4 +Author: Lasse Collin +Date: 2024-12-18 17:49:05 +0200 + + tuklib_mbstr_wrap: Silence a warning from Clang + + Fixes: ca529c3f41a4a19a59e2e252e6dd9255f130c634 + + src/common/tuklib_mbstr_wrap.c | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit 16c9796ef970ae349c54fef9a346e394d7cc4c75 +Author: Lasse Collin +Date: 2024-12-18 14:00:09 +0200 + + Update THANKS + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit 3b5c8a1fcab385eed9cc95684223fddd7cf5a053 +Author: Lasse Collin +Date: 2024-12-18 14:00:09 +0200 + + Update TODO + + Fixes: 5f6dddc6c911df02ba660564e78e6de80947c947 + + TODO | 3 --- + 1 file changed, 3 deletions(-) + +commit 22a35e64ce3d331b668f15f858a7bb3da3acc78e +Author: Lasse Collin +Date: 2024-12-18 14:00:09 +0200 + + lzmainfo: Use tuklib_mbstr_nonprint + + CMakeLists.txt | 3 +++ + src/lzmainfo/Makefile.am | 1 + + src/lzmainfo/lzmainfo.c | 16 ++++++++++------ + 3 files changed, 14 insertions(+), 6 deletions(-) + +commit 03111595ee713e0f94fb4f4a19a15594d5149347 +Author: Lasse Collin +Date: 2024-12-18 14:00:09 +0200 + + xzdec: Use tuklib_mbstr_nonprint + + CMakeLists.txt | 3 +++ + src/xzdec/Makefile.am | 2 ++ + src/xzdec/xzdec.c | 15 +++++++++++---- + 3 files changed, 16 insertions(+), 4 deletions(-) + +commit d22f96921fd2f94d842f3cc2e5f729cb3cca5122 +Author: Lasse Collin +Date: 2024-12-18 14:00:09 +0200 + + xz: Use tuklib_mbstr_nonprint + + Call tuklib_mask_nonprint() on filenames and also on a few other + strings from the command line too. + + The filename printed by "xz --robot --list" (in list.c) is also masked. + It's good to get rid of tabs and newlines which would desync the output + but masking other chars wouldn't be strictly necessary. It might matter + with sensible filenames if LC_CTYPE is "C" (when iswprint() might reject + non-ASCII chars) and a script wants to read a filename from xz's output. + Hopefully it's an unusual enough corner case to not be a real problem. + + CMakeLists.txt | 2 ++ + src/xz/Makefile.am | 1 + + src/xz/coder.c | 19 ++++++++----- + src/xz/file_io.c | 81 ++++++++++++++++++++++++++++++++++-------------------- + src/xz/list.c | 32 +++++++++++++-------- + src/xz/main.c | 10 +++++-- + src/xz/message.c | 8 ++++-- + src/xz/options.c | 10 ++++--- + src/xz/private.h | 1 + + src/xz/suffix.c | 12 ++++---- + 10 files changed, 113 insertions(+), 63 deletions(-) + +commit 40e573305535960574404d2eae848b248c95ea7e +Author: Lasse Collin +Date: 2024-12-18 14:00:09 +0200 + + Add tuklib_mbstr_nonprint to mask non-printable characters + + Malicious filenames or other untrusted strings may affect the state of + the terminal when such strings are printed as part of (error) messages. + Add functions that mask such characters. + + It's not enough to handle only single-byte control characters. + In multibyte locales, some control characters are multibyte too, for + example, terminals interpret C1 control characters (U+0080 to U+009F) + that are two bytes as UTF-8. + + Instead of checking for control characters with iswcntrl(), this + uses iswprint() to detect printable characters. This is much stricter. + On Windows it's actually too strict as it rejects some characters that + definitely are printable. + + Gnulib's quotearg would do a lot more but I hope this simpler method + is good enough here. + + Thanks to Ryan Colyer for the discussion about the problems of + the earlier single-byte-only method. + + Thanks to Christian Weisgerber for reporting a bug in an earlier + version of this code. + + Thanks to Jeroen Roovers for a typo fix. + + Closes: https://github.com/tukaani-project/xz/pull/118 + + src/Makefile.am | 2 + + src/common/tuklib_mbstr_nonprint.c | 151 +++++++++++++++++++++++++++++++++++++ + src/common/tuklib_mbstr_nonprint.h | 69 +++++++++++++++++ + 3 files changed, 222 insertions(+) + +commit 36190c8c4bb13d1eab84a30f3650a5ec5ff0e402 +Author: Lasse Collin +Date: 2024-12-18 11:33:09 +0200 + + Translations: Add preliminary Georgian translation + + Most of the auto-wrapped strings are translated already. A few + strings have changed since this was created though. This file + isn't in the Translation Project *yet* because these strings + are still very new. + + Closes: https://github.com/tukaani-project/xz/pull/145 + + po/LINGUAS | 1 + + po/ka.po | 1186 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 1187 insertions(+) + +commit 4a0c4f92b820b84ace625a95305a9d56cb662f4e +Author: Lasse Collin +Date: 2024-10-30 20:50:20 +0200 + + xz: Make one string simpler for translators + + Leading spaces in the string can get miscounted by translators. + + src/xz/list.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 3fcf547e926f6c0414b23459f7b43164f7e8c378 +Author: Lasse Collin +Date: 2024-12-17 10:26:10 +0200 + + lzmainfo: Sync the translatable strings with xz + + src/lzmainfo/lzmainfo.c | 20 ++++++++++++-------- + 1 file changed, 12 insertions(+), 8 deletions(-) + +commit 3e9177fd206d20d6d8acc7d203c25a9ae0549229 +Author: Lasse Collin +Date: 2024-12-17 10:26:10 +0200 + + xz: Use automatic word wrapping for help texts + + --long-help is now one line longer because --lzma1 is now on its + own line. + + CMakeLists.txt | 2 + + src/xz/Makefile.am | 3 +- + src/xz/message.c | 482 ++++++++++++++++++++++++++++++++++------------------- + 3 files changed, 313 insertions(+), 174 deletions(-) + +commit a0eecc9eb23ac583ccf442de3f5c106d4b09482d +Author: Lasse Collin +Date: 2024-12-16 18:46:45 +0200 + + po/Makevars: Add --keyword=W_:... to XGETTEXT_OPTIONS + + The text was copied from tuklib_gettext.h. + + Also rearrange the --keyword options to be last on the line. + + po/Makevars | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ca529c3f41a4a19a59e2e252e6dd9255f130c634 +Author: Lasse Collin +Date: 2024-12-16 18:43:52 +0200 + + Add tuklib_mbstr_wrap for automatic word wrapping + + Automatic word wrapping makes translators' work easier and reduces + errors like misaligned columns or overlong lines. Right-to-left + languages and languages that don't use spaces between words will + still need extra effort. (xz hasn't been translated to any RTL + language so far.) + + cmake/tuklib_mbstr.cmake | 4 + + m4/tuklib_mbstr.m4 | 2 +- + src/Makefile.am | 2 + + src/common/tuklib_gettext.h | 11 ++ + src/common/tuklib_mbstr_wrap.c | 285 +++++++++++++++++++++++++++++++++++++++++ + src/common/tuklib_mbstr_wrap.h | 203 +++++++++++++++++++++++++++++ + 6 files changed, 506 insertions(+), 1 deletion(-) + +commit 314b83cebad0244a0015a8abc6d8d086b581c215 +Author: Lasse Collin +Date: 2024-12-17 17:57:18 +0200 + + Build: Sort filenames to ASCII order in Makefile.am + + src/Makefile.am | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit df399c52554dfdf60259ca2cce97adbcfff39dc0 +Author: Lasse Collin +Date: 2024-10-21 18:51:24 +0300 + + tuklib_mbstr_width: Add tuklib_mbstr_width_mem() + + It's a new function split from tuklib_mbstr_width(). + It's useful with partial strings that aren't terminated with \0. + + src/common/tuklib_mbstr.h | 17 +++++++++++++++++ + src/common/tuklib_mbstr_width.c | 8 ++++++++ + 2 files changed, 25 insertions(+) + +commit 51081efae4c52c226e96da95313916eba99f885f +Author: Lasse Collin +Date: 2024-12-16 20:08:27 +0200 + + tuklib_mbstr_width: Update a comment about shift states + + src/common/tuklib_mbstr_width.c | 11 ++++++++--- + 1 file changed, 8 insertions(+), 3 deletions(-) + +commit 7ff1b0ac53866877bdfd79acf5fee0269058c58b +Author: Lasse Collin +Date: 2024-10-21 18:47:56 +0300 + + tuklib_mbstr_width: Don't mention shift states in the API docs + + It is assumed that this code won't be used with charsets that use + locking shift states. + + src/common/tuklib_mbstr.h | 8 ++------ + 1 file changed, 2 insertions(+), 6 deletions(-) + +commit 3c16105936320e4095dbe84fa9a33a4a6d46a597 +Author: Lasse Collin +Date: 2024-10-21 18:41:41 +0300 + + tuklib_mbstr_width: Use stricter return value checking + + This should make no difference in practice (at least if mbrtowc() + isn't broken). + + src/common/tuklib_mbstr_width.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b797c44c42ea54fe1c52722a2fca0c9618575598 +Author: Lasse Collin +Date: 2024-12-16 20:06:07 +0200 + + tuklib_mbstr_width: Change the behavior when wcwidth() is not available + + If wcwidth() isn't available (Windows), previously it was assumed + that one byte == one column in the terminal. Now it is assumed that + one multibyte character == one column. This works better with UTF-8. + Languages that only use single-width characters without any combining + characters should work correctly with this. + + In xz, none of po/*.po contain combining characters and only ko.po, + zh_CN.po, and zh_TW.po contain fullwidth characters. Thus, "only" + those three translations in xz are broken on Windows with the + UTF-8 code page. Broken means that column headings in xz -lvv and + (only in the master branch) strings in --long-help are misaligned, + so it's not a huge problem. I don't know if those three languages + displayed perfectly before the UTF-8 change because I hadn't tested + translations with native Windows builds before. + + Fixes: 46ee0061629fb075d61d83839e14dd193337af59 + + src/common/tuklib_mbstr_width.c | 13 +++++++++++-- + 1 file changed, 11 insertions(+), 2 deletions(-) + +commit 78868b6ed63fa4c89f73e3dfed27abfb8b0d46db +Author: Lasse Collin +Date: 2024-12-18 14:23:13 +0200 + + xzdec: Use setlocale() via tuklib_gettext_setlocale() + + xzdec isn't translated and didn't have locale-specific behavior + in the past. On Windows with UTF-8 in the application manifest, + setting the locale makes a difference though: + + - Without any setlocale() call, non-ASCII filenames don't display + properly in Command Prompt unless one first uses "chcp 65001" + to set the console code page to UTF-8. + + - setlocale(LC_ALL, "") is enough to make non-ASCII filenames + print correctly in Command Prompt without using "chcp 65001", + assuming that the non-UTF-8 code page (like 850) supports + those non-ASCII characters. + + - setlocale(LC_ALL, ".UTF8") is even better because then mbrtowc() and + such functions use an UTF-8 locale instead of a legacy code page. + The tuklib_gettext_setlocale() macro takes care of this (without + enabling any translations). + + Fixes: 46ee0061629fb075d61d83839e14dd193337af59 + + src/xzdec/xzdec.c | 12 ++++++++++++ + 1 file changed, 12 insertions(+) + +commit 0d0b574cc45045d6150d397776340c068df59e2a +Author: Lasse Collin +Date: 2024-12-17 14:59:37 +0200 + + Windows: Use UTF-8 locale when active code page is UTF-8 + + XZ Utils 5.6.3 set the active code page to UTF-8 to fix CVE-2024-47611. + This wasn't paired with UCRT-specific setlocale(LC_ALL, ".UTF8"), thus + non-ASCII characters from translations became mojibake. + + Fixes: 46ee0061629fb075d61d83839e14dd193337af59 + + src/common/tuklib_gettext.h | 32 ++++++++++++++++++++++++++++++-- + 1 file changed, 30 insertions(+), 2 deletions(-) + +commit 20dfca8171dad4c64785ac61d5b68972c444877b +Author: Lasse Collin +Date: 2024-12-17 15:01:29 +0200 + + Windows: Document the need for setlocale(LC_ALL, ".UTF8") + + Also warn about unpaired surrogates and (somewhat UTF-8-specific) + MAX_PATH issue in FindFirstFileA(). + + Fixes: 46ee0061629fb075d61d83839e14dd193337af59 + + src/common/w32_application.manifest.comments.txt | 28 +++++++++++++++++++++++- + 1 file changed, 27 insertions(+), 1 deletion(-) + +commit 4e936f234056e5831013ed922145b666b04bb1e3 +Author: Lasse Collin +Date: 2024-12-18 14:12:22 +0200 + + xzdec: Call tuklib_progname_init() early enough + + If the early pledge() call on OpenBSD fails, it calls my_errorf() + which requires the "progname" variable. + + Fixes: d74fb5f060b76db709b50f5fd37490394e52f975 + + src/xzdec/xzdec.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 61feaf681bd793dc5c919732b44bca7dcf2ed1b8 +Author: Lasse Collin +Date: 2024-12-15 19:08:32 +0200 + + CMake: Bump maximum policy version to 3.31 + + With CMake 3.31, there were a few warnings from + CMP0177 "install() DESTINATION paths are normalized". + These occurred because the install(FILES) command in + my_install_man_lang() is called with a DESTINATION path + that contains two consecutive slashes, for example, + "share/man//man1". Such a path is for the English man pages. + With translated man pages, the language code goes between + the slashes. The warning was probably triggered because the + extra slash gets removed by the normalization. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b0bb84dd7bbdcc85243386a0051c7b2cb5fc6a18 +Author: Lasse Collin +Date: 2024-12-15 18:35:27 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit bee0c044d30a6ad3b3d94901c27e7519f6f46e27 +Author: Dexter Castor Döpping +Date: 2024-12-08 18:24:29 +0100 + + liblzma: Fix incorrect macro name in a comment + + Fixes: 33b8a24b6646a9dbfd8358405aec466b13078559 + Closes: https://github.com/tukaani-project/xz/pull/155 + + src/liblzma/api/lzma/lzma12.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2cfa1ad0a9eb62b1847cf13f9aee290158978a3a +Author: Lasse Collin +Date: 2024-12-17 10:36:43 +0200 + + license-check.sh: Add an exception for doc/SHA256SUMS + + Fixes: 36b531022f24a2ab57a2dfb9e5052f1c176e9d9a + + build-aux/license-check.sh | 1 + + 1 file changed, 1 insertion(+) + +commit 36b531022f24a2ab57a2dfb9e5052f1c176e9d9a +Author: Lasse Collin +Date: 2024-12-01 21:38:17 +0200 + + doc/SHA256SUMS: Add the list of SHA-256 hashes of release files + + The release files are signed but verifying the signatures cannot + catch certain types of attacks: + + 1. A malicious maintainer could make more than one variant of + a package. One could be for general distribution. Another + with malicious content could be targeted to specific users, + for example, distributing the malicious version on a mirror + controlled by the attacker. + + 2. If the signing key of an honest maintainer was compromised + without being detected, a similar situation as described + above could occur. + + SHA256SUMS could be put on the project website but having it in + the Git repository makes it obvious that old lines aren't modified + when the file is updated. + + Hashes of uncompressed files are included too. This way tarballs + can be recompressed and the hashes can still be verified. + + .gitattributes | 1 + + doc/SHA256SUMS | 218 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 219 insertions(+) + +commit fe9e66993fdbcc2981c7361b9b034a451eb0fc42 +Author: Lasse Collin +Date: 2024-11-30 12:05:59 +0200 + + Docs: Remove .github/SECURITY.md + + One of the reasons to have this file in the xz repository was to + show vulnerability reporting info in the Security section on GitHub. + On 2024-11-25, I added SECURITY.md to the tukaani-project organization + on GitHub: + + https://github.com/tukaani-project/.github/blob/main/SECURITY.md + + GitHub shows that file in all projects in the organization unless + overridden by a project-specific SECURITY.md. Thus, removing + the file from the xz repo makes GitHub show the organization-wide + text instead. + + Maintaining a single copy for the whole GitHub organization makes + things simpler. It's also nicer to have fewer GitHub-specific files + in the xz repo. Information how to report bugs (including security + issues) is available in README and on the home page too. + + The OpenSSF Scorecard tool didn't find .github/SECURITY.md from the + xz repository. There was a suggestion to move the file to the top-level + directory where Scorecard should find it. However, Scorecard does find + the organization-wide SECURITY.md. Thus, the file isn't needed in the + xz repository to score points in the Scorecard game: + + https://scorecard.dev/viewer/?uri=github.com/tukaani-project/xz + + Closes: https://github.com/tukaani-project/xz/issues/148 + Closes: https://github.com/tukaani-project/xz/pull/149 + + .github/SECURITY.md | 14 -------------- + 1 file changed, 14 deletions(-) + +commit b36177273602ebc83e9cc58517f63a7b6af33f70 +Author: Lasse Collin +Date: 2024-11-30 10:27:14 +0200 + + Translations: Update the Chinese (traditional) translation + + po/zh_TW.po | 201 +++++++++++++++++++++++++----------------------------------- + 1 file changed, 84 insertions(+), 117 deletions(-) + +commit c15115f7ede492f20c91b08ba485f9426f60233f +Author: Lasse Collin +Date: 2024-10-30 19:54:34 +0200 + + liblzma: Optimize the loop conditions in BCJ filters + + Compilers cannot optimize the addition "i + 4" away since theoretically + it could overflow. + + src/liblzma/simple/arm.c | 4 +++- + src/liblzma/simple/arm64.c | 4 +++- + src/liblzma/simple/armthumb.c | 7 ++++++- + src/liblzma/simple/ia64.c | 4 +++- + src/liblzma/simple/powerpc.c | 4 +++- + src/liblzma/simple/sparc.c | 5 +++-- + 6 files changed, 21 insertions(+), 7 deletions(-) + +commit 9f69e71e78621fd056f5eaaad7cdcd9279310fb5 +Author: Lasse Collin +Date: 2024-11-25 16:26:54 +0200 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 48ff3f06521ca326996ab9a04d1b342098960427 +Author: Mark Wielaard +Date: 2024-11-25 12:28:44 +0200 + + xz: Landlock: Fix a file descriptor leak + + src/xz/sandbox.c | 1 + + 1 file changed, 1 insertion(+) + +commit dbca3d078ec581600600abebbb18769d3d713914 +Author: Sam James +Date: 2024-10-02 03:04:03 +0100 + + CI: update FreeBSD, NetBSD, OpenBSD, Solaris actions + + Checked the changes and they're all innocuous. This should hopefully + fix the "externally managed" pip error in these jobs that started + recently. + + .github/workflows/freebsd.yml | 2 +- + .github/workflows/netbsd.yml | 2 +- + .github/workflows/openbsd.yml | 2 +- + .github/workflows/solaris.yml | 2 +- + 4 files changed, 4 insertions(+), 4 deletions(-) + +commit a94b85bea3f04d8c1f4e2e6f648a9a15bc6ce58f +Author: Lasse Collin +Date: 2024-10-01 12:17:39 +0300 + + Add NEWS for 5.6.3 + + NEWS | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 125 insertions(+) + +commit be4bf94446b6286a5dffdde85fc1d21448f4edff +Author: Lasse Collin +Date: 2024-10-01 14:49:41 +0300 + + cmake/tuklib_large_file_support.cmake: Add a missing include + + v5.2 didn't build with CMake. Other branches had + include(CMakePushCheckState) in top-level CMakeLists.txt + which made the build work. + + Fixes: 597f49b61475438a43a417236989b2acc968a686 + + cmake/tuklib_large_file_support.cmake | 1 + + 1 file changed, 1 insertion(+) + +commit 1ebbe915d4e0d877154261b5f8103719a6722975 +Author: Lasse Collin +Date: 2024-10-01 12:10:23 +0300 + + Update THANKS + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit 74702ee00ecfd080d8ab11118cd25dbe6c437ec0 +Author: Lasse Collin +Date: 2024-10-01 12:10:23 +0300 + + Tests/Windows: Add the application manifest to the test programs + + This ensures that the test programs get executed the same way as + the binaries that are installed. + + CMakeLists.txt | 14 ++++++++++---- + tests/Makefile.am | 10 ++++++++++ + tests/tests.cmake | 33 ++++++++++++++++++++++++++++++++- + tests/tests_w32res.rc | 18 ++++++++++++++++++ + 4 files changed, 70 insertions(+), 5 deletions(-) + +commit 7ddf2273e0e4654582ee65db19d44431bfdb5791 +Author: Lasse Collin +Date: 2024-10-01 12:10:23 +0300 + + license-check.sh: Add an exception for w32_application.manifest + + The file gets embedded as is into executables, thus it cannot + hold a license identifier. + + build-aux/license-check.sh | 1 + + 1 file changed, 1 insertion(+) + +commit 46ee0061629fb075d61d83839e14dd193337af59 +Author: Lasse Collin +Date: 2024-10-01 12:10:23 +0300 + + Windows: Embed an application manifest in the EXE files + + IMPORTANT: This includes a security fix to command line tool + argument handling. + + Some toolchains embed an application manifest by default to declare + UAC-compliance. Some also declare compatibility with Vista/8/8.1/10/11 + to let the app access features newer than those of Vista. + + We want all the above but also two more things: + + - Declare that the app is long path aware to support paths longer + than 259 characters (this may also require a registry change). + + - Force the code page to UTF-8. This allows the command line tools + to access files whose names contain characters that don't exist + in the current legacy code page (except unpaired surrogates). + The UTF-8 code page also fixes security issues in command line + argument handling which can be exploited with malicious filenames. + See the new file w32_application.manifest.comments.txt. + + Thanks to Orange Tsai and splitline from DEVCORE Research Team + for discovering this issue. + + Thanks to Vijay Sarvepalli for reporting the issue to me. + + Thanks to Kelvin Lee for testing with MSVC and helping with + the required build system fixes. + + CMakeLists.txt | 18 +++ + src/Makefile.am | 4 +- + src/common/common_w32res.rc | 5 + + src/common/w32_application.manifest | 28 ++++ + src/common/w32_application.manifest.comments.txt | 178 +++++++++++++++++++++++ + 5 files changed, 232 insertions(+), 1 deletion(-) + +commit dad153091552b52a41b95ec4981c6951f1cae487 +Author: Lasse Collin +Date: 2024-09-29 14:46:52 +0300 + + Windows: Set DLL name accurately in StringFileInfo on Cygwin and MSYS2 + + Now the information in the "Details" tab in the file properties + dialog matches the naming convention of Cygwin and MSYS2. This + is only a cosmetic change. + + src/liblzma/liblzma_w32res.rc | 10 +++++++++- + 1 file changed, 9 insertions(+), 1 deletion(-) + +commit 8940ecb96fe9f0f2a9cfb8b66fe9ed31ffbea904 +Author: Lasse Collin +Date: 2024-09-25 15:47:55 +0300 + + common_w32res.rc: White space edits + + LANGUAGE and VS_VERSION_INFO begin new statements so put an empty line + between them. + + src/common/common_w32res.rc | 15 ++++++++------- + 1 file changed, 8 insertions(+), 7 deletions(-) + +commit c3b9dad07d3fd9319f88386b7095019bcea45ce1 +Author: Lasse Collin +Date: 2024-09-28 20:09:50 +0300 + + CMake: Add the resource files to the Cygwin and MSYS2 builds + + Autotools-based build has always done this so this is for consistency. + + However, the CMake build won't create the DEF file when building + for Cygwin or MSYS2 because in that context it should be useless. + (If Cygwin or MSYS2 is used to host building of normal Windows + binaries then the DEF file is still created.) + + CMakeLists.txt | 16 ++++++++++------ + 1 file changed, 10 insertions(+), 6 deletions(-) + +commit da4f275bd1c18b897e5c2dd0043546de3accce0a +Author: Lasse Collin +Date: 2024-09-28 15:19:14 +0300 + + CMake: Fix Windows resource file dependencies + + If common_w32res.rc is modified, the resource files need to be rebuilt. + In contrast, the liblzma*.map files truly are link dependencies. + + CMakeLists.txt | 17 +++++++++-------- + 1 file changed, 9 insertions(+), 8 deletions(-) + +commit 1c673c0aac7f7dee8dda2c1140351c8417a71e47 +Author: Lasse Collin +Date: 2024-09-29 01:20:03 +0300 + + CMake: Checking for CYGWIN covers MSYS2 too + + On MSYS2, both CYGWIN and MSYS are set. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 6aaa0173b839e28429d43a8b62d257ad2f3b4521 +Author: Lasse Collin +Date: 2024-09-28 09:37:30 +0300 + + Translations: Add the SPDX license identifier to pt_BR.po + + po/pt_BR.po | 2 ++ + 1 file changed, 2 insertions(+) + +commit dc7b9f24b737e4e55bcbbdde6754883f991c2cfb +Author: Lasse Collin +Date: 2024-09-25 16:41:37 +0300 + + Windows/CMake: Use the correct resource file for lzmadec.exe + + CMakeLists.txt was using xzdec_w32res.rc for both xzdec and lzmadec. + + Fixes: 998d0b29536094a89cf385a3b894e157db1ccefe + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b834ae5f80911a3819d6cdb484f61b257174c544 +Author: Lasse Collin +Date: 2024-09-25 21:29:59 +0300 + + Translations: Update the Brazilian Portuguese translation + + po/pt_BR.po | 144 ++++++++++++++++++++++-------------------------------------- + 1 file changed, 53 insertions(+), 91 deletions(-) + +commit eceb023d4c129fd63ee881a2d8696eaf52ad1532 +Author: Lasse Collin +Date: 2024-09-17 01:21:15 +0300 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 76cfd0a9bb33ae8e534b1f73f6359dc825589f2f +Author: Tobias Stoeckmann +Date: 2024-09-16 23:19:46 +0200 + + lzmainfo: Avoid integer overflow + + The MB output can overflow with huge numbers. Most likely these are + invalid .lzma files anyway, but let's avoid garbage output. + + lzmadec was adapted from LZMA Utils. The original code with this bug + was written in 2005, over 19 years ago. + + Co-authored-by: Lasse Collin + Closes: https://github.com/tukaani-project/xz/pull/144 + + src/lzmainfo/lzmainfo.c | 5 ++--- + 1 file changed, 2 insertions(+), 3 deletions(-) + +commit 78355aebb7fb654302e5e33692ba109909dacaff +Author: Tobias Stoeckmann +Date: 2024-09-16 22:04:40 +0200 + + xzdec: Remove unused short option -M + + "xzdec -M123" exited with exit status 1 without printing + any messages. The "M:" entry should have been removed when + the memory usage limiter support was removed from xzdec. + + Fixes: 792331bdee706aa852a78b171040ebf814c6f3ae + Closes: https://github.com/tukaani-project/xz/pull/143 + [ Lasse: Commit message edits ] + + src/xzdec/xzdec.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit e5758db7bd75587a2499e0771907521a4aa86908 +Author: Lasse Collin +Date: 2024-09-10 13:54:47 +0300 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 80ffa38f56657257ed4d90d76f6bd2f2bcb8163c +Author: Firas Khalil Khana +Date: 2024-09-10 12:30:32 +0300 + + Build: Fix a typo in autogen.sh + + Fixes: e9be74f5b129fe8a5388d588e68b1b7f5168a310 + Closes: https://github.com/tukaani-project/xz/pull/141 + + autogen.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 68c54e45d042add64a4cb44bfc87ca74d29b87e2 +Author: Lasse Collin +Date: 2024-09-02 20:08:40 +0300 + + Translations: Update Chinese (simplified) translation + + Differences to the zh_CN.po file from the Translation Project: + + - Two uses of \v were fixed. + + - Missing "OPTS" translation in --riscv[=OPTS] was copied from + previous lines. + + - "make update-po" was run to remove line numbers from comments. + + po/zh_CN.po | 102 ++++++++++++++++++++++++------------------------------------ + 1 file changed, 40 insertions(+), 62 deletions(-) + +commit 2230692aa1bcebb586100183831e3daf1714d60a +Author: Lasse Collin +Date: 2024-09-02 19:40:50 +0300 + + Translations: Update the Catalan translation + + Differences to the ca.po file from the Translation Project: + + - An overlong line translating --filters-help was wrapped. + + - "make update-po" was used to remove line numbers from the comments + to match the changes in fccebe2b4fd513488fc920e4dac32562ed3c7637 + and 093490b58271e9424ce38a7b1b38bcf61b9c86c6. xz.pot in the TP + is older than these commits. + + po/ca.po | 171 ++++++++++++++++++++++++++------------------------------------- + 1 file changed, 69 insertions(+), 102 deletions(-) + +commit 3e7723ce26f74c71919984a6180504b4548cbb7e +Author: Lasse Collin +Date: 2024-08-22 14:06:16 +0300 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit d3e0e679b2b8b428598bb8ba56a17715190814db +Author: Lasse Collin +Date: 2024-08-22 14:06:16 +0300 + + CMake: Don't install lzmadec.1 symlinks if XZ_TOOL_LZMADEC=OFF + + Thanks-to: 榆柳松 (ZhengSen Wang) + Fixes: fb50c6ba1d4c9405e5b12b5988b01a3002638c5d + Closes: https://github.com/tukaani-project/xz/pull/134 + + CMakeLists.txt | 12 ++++++++++-- + 1 file changed, 10 insertions(+), 2 deletions(-) + +commit acdf21033abe347d9a279e9fe757f90ed16c1dbb +Author: Lasse Collin +Date: 2024-08-22 14:06:16 +0300 + + CMake: Fix the build when XZ_TOOL_LZMADEC=OFF + + Co-developed-by: 榆柳松 (ZhengSen Wang) + Fixes: fb50c6ba1d4c9405e5b12b5988b01a3002638c5d + Fixes: https://github.com/tukaani-project/xz/pull/134 + + CMakeLists.txt | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 5e375987509fab484b7bef0b90be92f241c58c91 +Author: Lasse Collin +Date: 2024-08-22 11:01:07 +0300 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 6cd7c8607843c337edfe2c472aa316602a393754 +Author: Yifeng Li +Date: 2024-08-22 02:18:49 +0000 + + liblzma: Fix x86-64 movzw compatibility in range_decoder.h + + Support for instruction "movzw" without suffix in "GNU as" was + added in commit [1] and stabilized in binutils 2.27, released + in August 2016. Earlier systems don't accept this instruction + without a suffix, making range_decoder.h's inline assembly + unable to build on old systems such as Ubuntu 16.04, creating + error messages like: + + lzma_decoder.c: Assembler messages: + lzma_decoder.c:371: Error: no such instruction: `movzw 2(%r11),%esi' + lzma_decoder.c:373: Error: no such instruction: `movzw 4(%r11),%edi' + lzma_decoder.c:388: Error: no such instruction: `movzw 6(%r11),%edx' + lzma_decoder.c:398: Error: no such instruction: `movzw (%r11,%r14,4),%esi' + + Change "movzw" to "movzwl" for compatibility. + + [1] https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=c07315e0c610e0e3317b4c02266f81793df253d2 + + Suggested-by: Lasse Collin + Tested-by: Yifeng Li + Signed-off-by: Yifeng Li + Fixes: 3182a330c1512cc1f5c87b5c5a272578e60a5158 + Fixes: https://github.com/tukaani-project/xz/issues/121 + Closes: https://github.com/tukaani-project/xz/pull/136 + + src/liblzma/rangecoder/range_decoder.h | 24 ++++++++++++------------ + 1 file changed, 12 insertions(+), 12 deletions(-) + +commit bf901dee5d4c46609645e50311c0cb2dfdcf9738 +Author: Lasse Collin +Date: 2024-07-19 20:02:43 +0300 + + Build: Comment that elf_aux_info(3) will be available on OpenBSD >= 7.6 + + CMakeLists.txt | 2 +- + configure.ac | 17 +++++++++++------ + 2 files changed, 12 insertions(+), 7 deletions(-) + +commit f7103c2c2a8fa51d1f308ba7387beeff20a0d4dd +Author: Lasse Collin +Date: 2024-07-19 19:42:26 +0300 + + Revert "liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD" + + This reverts commit dc03f6290f5b9bd3d50c7e12e58dee870889d599. + + OpenBSD 7.6 will support elf_aux_info(3), and the detection code used + on FreeBSD will work on OpenBSD 7.6 too. Keep things simpler and drop + the OpenBSD-specific sysctl() method. + + Thanks to Christian Weisgerber. + + CMakeLists.txt | 6 ------ + configure.ac | 9 --------- + src/liblzma/check/crc32_arm64.h | 15 --------------- + src/liblzma/check/crc_common.h | 1 - + 4 files changed, 31 deletions(-) + +commit 7c292dd0bf23cefcdf4b1509f3666322e08a7ede +Author: Lasse Collin +Date: 2024-07-13 22:10:37 +0300 + + liblzma: Tweak a comment + + src/liblzma/simple/arm64.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 6408edac5529d6ec0abf52794074f229c8362303 +Author: Lasse Collin +Date: 2024-07-11 22:17:56 +0300 + + CMake: Bump maximum policy version to 3.30 + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 9231c39ffb518196d6664a86e5325e744621a21b +Author: Lasse Collin +Date: 2024-07-06 15:13:19 +0300 + + CMake: Require CMake 3.20 or later + + This allows a few cleanups. + + CMakeLists.txt | 78 ++++++++++++++++++++-------------------------------------- + 1 file changed, 27 insertions(+), 51 deletions(-) + +commit 028185dd4889e3d6235ff13560160ebca6985021 +Author: Lasse Collin +Date: 2024-07-09 14:27:51 +0300 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit baecfa142644eb5f5c6dd6f8e2f531c362fa3747 +Author: Lasse Collin +Date: 2024-07-06 14:04:48 +0300 + + xz: Remove the TODO comment about --recursive + + It won't be implemented. find + xargs is more flexible, for example, + it allows compressing small files in parallel. An example for that + has been included in the xz man page since 2010. + + src/xz/args.c | 1 - + 1 file changed, 1 deletion(-) + +commit f691d58fae82bd815c5f86ffad10fe9b6b59dad8 +Author: Lasse Collin +Date: 2024-07-06 14:04:16 +0300 + + Document --disable-loongarch-crc32 in INSTALL + + INSTALL | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit b3e53122f42796aaebd767bab920cf7bedf69966 +Author: Lasse Collin +Date: 2024-07-03 20:45:48 +0300 + + CMake: Link xz against Threads::Threads if using pthreads + + The liblzma target was recently changed to link against Threads::Threads + with the PRIVATE keyword. I had forgotten that xz itself depends on + pthreads too due to pthread_sigmask(). Thus, the build broke when + building shared liblzma and pthread_sigmask() wasn't in libc. + + Thanks to Peter Seiderer for the bug report. + + Fixes: ac05f1b0d7cda1e7ae79775a8dfecc54601d7f1c + Fixes: https://github.com/tukaani-project/xz/issues/129#issuecomment-2204522994 + + CMakeLists.txt | 13 +++++++++++++ + 1 file changed, 13 insertions(+) + +commit 5742ec1fc7f2cf1c82cfe3477bb90594a4658374 +Author: Lasse Collin +Date: 2024-07-02 22:49:33 +0300 + + Update THANKS + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 2d13d10357ecad243d7e4ff1de0e6b437c38a47a +Author: Lasse Collin +Date: 2024-07-02 20:23:35 +0300 + + CMake: Improve NLS error messages + + CMakeLists.txt | 11 +++++++---- + 1 file changed, 7 insertions(+), 4 deletions(-) + +commit 628d8d2c4fdf9e6a91c7bba7a743f400a94c2909 +Author: Lasse Collin +Date: 2024-07-02 20:19:47 +0300 + + CMake: Update the comment at the top of CMakeLists.txt + + While po/*.gmo files won't be used from the release tarball, + the generated translated man pages will be used still. Those + are text files and po4a has slightly more dependencies than + gettext tools so installing po4a might be a bit more challenging + in some situations. + + CMakeLists.txt | 17 +++++++---------- + 1 file changed, 7 insertions(+), 10 deletions(-) + +commit b4b23c94fd4429abc663ced28d5cdc9cf7eb7507 +Author: Lasse Collin +Date: 2024-07-02 20:12:40 +0300 + + CMake: Drop support for pre-generated po/*.gmo files + + When a release tarball is created using Autotools, the tarball includes + po/*.gmo files which are binary files generated from po/*.po. Other + tarball creation methods don't and won't create the .gmo files. + + It feels clearer if CMake will never install pre-generated binary files + from the source package. If people are able to install CMake, they + likely are able to install gettext tools as well (assuming they want + translations). + + CMakeLists.txt | 66 +++++++++++++++++++--------------------------------------- + 1 file changed, 21 insertions(+), 45 deletions(-) + +commit fb99f8e8c50171b898cb79fe1dc703d5f91e4f0a +Author: Lasse Collin +Date: 2024-07-02 19:14:50 +0300 + + CMake: Make XZ_NLS handling more robust + + If a user set XZ_NLS=ON but find_package(Intl) failed or CMake version + wasn't at least 3.20, the configuration would fail in a cryptic way. + + If XZ_NLS is enabled, require that CMake is new enough and that either + gettext tools or pre-generated .gmo files are available. Otherwise fail + the configuration. Previously missing gettext tools and .gmo files would + only result in a warning. + + Missing man page translations are still only a warning. + + Thanks to Peter Seiderer for the bug report. + + Fixes: https://github.com/tukaani-project/xz/issues/129 + Closes: https://github.com/tukaani-project/xz/pull/130 + + CMakeLists.txt | 82 ++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 46 insertions(+), 36 deletions(-) + +commit ec6157570ea8a8e38158894e530d35416ff6a0f8 +Author: Lasse Collin +Date: 2024-07-02 19:39:05 +0300 + + CI: Add gettext as a dependency to CMake builds + + .github/workflows/ci.yml | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 24f0f7e399de03bb2ff675d97b723d14f17ed6ac +Author: Lasse Collin +Date: 2024-07-02 18:43:56 +0300 + + CMake: Fix ENABLE_NLS comment too + + Fixes: 29f77c7b707f2458fb047e77497354b195e05b14 + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit a0df0676130bc565af0ec911e68a1d0fbc3ed0fb +Author: Lasse Collin +Date: 2024-07-02 18:02:50 +0300 + + CMake: The compile definition is ENABLE_NLS, not XZ_NLS + + The CMake variables were renamed and accidentally also + the compile definition was renamed. As a result, translation + support wasn't actually enabled in the executables. + + Fixes: 29f77c7b707f2458fb047e77497354b195e05b14 + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 45d08abc33ccc52d2f050dcec458badc2ce59d0b +Author: Lasse Collin +Date: 2024-07-01 17:33:20 +0300 + + Update AUTHORS and THANKS + + AUTHORS | 2 +- + THANKS | 1 + + 2 files changed, 2 insertions(+), 1 deletion(-) + +commit 7baf6835cfbf9c85ba37f9ffb7d4f87fb86a474e +Author: Xi Ruoyao +Date: 2024-06-28 13:36:43 +0300 + + liblzma: Speed up CRC32 calculation on 64-bit LoongArch + + The crc.w.{b/h/w/d}.w instructions in LoongArch can calculate the CRC32 + result for 1/2/4/8 bytes in a single operation. Using these is much + faster compared to the generic method. + + Optimized CRC32 is enabled unconditionally on 64-bit LoongArch because + the LoongArch specification says that CRC32 instructions shall be + implemented for 64-bit processors. Optimized CRC32 isn't enabled for + 32-bit LoongArch processors because not enough information is available + about them. + + Co-authored-by: Lasse Collin + + Closes: https://github.com/tukaani-project/xz/pull/86 + + CMakeLists.txt | 25 ++++++++++++++ + configure.ac | 40 +++++++++++++++++++++++ + src/liblzma/check/Makefile.inc | 3 +- + src/liblzma/check/crc32_fast.c | 2 ++ + src/liblzma/check/crc32_loongarch.h | 65 +++++++++++++++++++++++++++++++++++++ + src/liblzma/check/crc_common.h | 15 +++++++++ + 6 files changed, 149 insertions(+), 1 deletion(-) + +commit 0ed893668554fb0758003289f8a6af9bd08b89d1 +Author: Lasse Collin +Date: 2024-06-28 14:20:49 +0300 + + liblzma: ARM64 CRC32: Align the buffer faster + + Instead of doing it byte by byte, use the 1/2/4-byte CRC32 instructions. + + src/liblzma/check/crc32_arm64.h | 54 ++++++++++++++++++++++++++++++----------- + 1 file changed, 40 insertions(+), 14 deletions(-) + +commit 7e99856f66c07852c4e0de7aa01951e9147d86b0 +Author: Sam James +Date: 2024-06-28 14:18:35 +0300 + + CI: Speed up Valgrind job by using --trace-children-skip-by-arg=... + + This addresses the issue I mentioned in + 6c095a98fbec70b790253a663173ecdb669108c4 and speeds up the Valgrind + job a bit, because non-xz tools aren't run unnecessarily with + Valgrind by the script tests. + + .github/workflows/ci.yml | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2402e8a1ae92676fa0d4cb1b761d7f62f005c098 +Author: Lasse Collin +Date: 2024-06-25 16:00:22 +0300 + + Build: Prepend, not append, PTHREAD_CFLAGS to LIBS + + It shouldn't make any difference because LIBS should be empty + at that point in configure. But prepending is the correct way + because in general the libraries being added might require other + libraries that come later on the command line. + + configure.ac | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 7bb46f2b7b3989c1b589a247a251470f65e91cda +Author: Lasse Collin +Date: 2024-06-25 14:24:29 +0300 + + Build: Use AC_LINK_IFELSE to handle implicit function declarations + + It's more robust in case the compiler allows pre-C99 implicit function + declarations. If an x86 intrinsic is missing and gets treated as + implicit function, the linking step will very probably fail. This + isn't the only way to workaround implicit function declarations but + it might be the simplest and cleanest. + + The problem hasn't been observed in the wild. + + There are a couple more AC_COMPILE_IFELSE uses in configure.ac. + Of these, Landlock check calls prctl() and in theory could have + the same problem. In practice it doesn't as the check program + looks for several other things too. However, it was changed to + AC_LINK_IFELSE still to look more correct. + + Similarly, m4/tuklib_cpucores.m4 and m4/tuklib_physmem.m4 were + updated although they haven't given any trouble either. They + have worked all these years because those check programs rely + on specific headers and types: if headers or types are missing, + compilation will fail. Using the linker makes these checks more + similar to the ones in cmake/tuklib_*.cmake which always link. + + configure.ac | 8 ++++++-- + m4/tuklib_cpucores.m4 | 8 ++++---- + m4/tuklib_physmem.m4 | 17 +++++++++++------ + 3 files changed, 21 insertions(+), 12 deletions(-) + +commit 35eb57355ad1c415a838d26192d5af84abb7cf39 +Author: Lasse Collin +Date: 2024-06-24 23:35:59 +0300 + + Build: Use AC_LINK_IFELSE instead of -Werror + + AC_COMPILE_IFELSE needed -Werror because Clang <= 14 would merely + warn about the unsupported attribute and implicit function declaration. + Changing to AC_LINK_IFELSE handles the implicit declaration because + the symbol __crc32d is unlikely to exist in libc. + + Note that the other part of the check is that #include + must work. If the header is missing, most compilers give an error + and the linking step won't be attempted. + + Avoiding -Werror makes the check more robust in case CFLAGS contains + warning flags that break -Werror anyway (but this isn't the only check + in configure.ac that has this problem). Using AC_LINK_IFELSE also makes + the check more similar to how it is done in CMakeLists.txt. + + configure.ac | 12 +----------- + 1 file changed, 1 insertion(+), 11 deletions(-) + +commit 5a728813c378cc3c4c9c95793762452418d08f1b +Author: Lasse Collin +Date: 2024-06-24 23:34:34 +0300 + + Build: Sync the compile check changes from CMakeLists.txt + + It's nice to keep these in sync. The use of main() will later allow + AC_LINK_IFELSE usage too which may avoid the more fragile -Werror. + + configure.ac | 15 ++++++++------- + 1 file changed, 8 insertions(+), 7 deletions(-) + +commit 5279828635a95abdef82e691fc4979d362780e63 +Author: Lasse Collin +Date: 2024-06-24 20:14:43 +0300 + + CMake: Not experimental anymore + + While the CMake support has gotten a lot less testing than + the Autotools-based build, the supported features should now + be equal. The output may differ slightly, for example, + liblzma.pc may have + + Libs.private: -pthread -lpthread + + with Autotools on GNU/Linux. CMake doesn't put any options + in Libs.private because on modern glibc the pthread functions + are in libc. The options options aren't required to link static + liblzma into an application. + + Autotools-based build doesn't generate or install + lib/cmake/liblzma-*.cmake files. This means that on most + platforms one cannot rely on + + find_package(liblzma 5.2.5 REQUIRED CONFIG) + + or such finding those files. + + CMakeLists.txt | 9 ++++++--- + 1 file changed, 6 insertions(+), 3 deletions(-) + +commit de215a0517645d16343f3a5336d3df884a4f665f +Author: Lasse Collin +Date: 2024-06-25 16:11:13 +0300 + + CMake: Use configure_file() to copy a file + + I had missed this simpler method before. It does create a dependency + so that if .in.h changes the copying is done again. + + CMakeLists.txt | 17 +++++++---------- + 1 file changed, 7 insertions(+), 10 deletions(-) + +commit e620f35097c0ad20cd76d8258750aa706758ced9 +Author: Lasse Collin +Date: 2024-06-25 15:51:48 +0300 + + CMake: Always add pthread flags into CMAKE_REQUIRED_LIBRARIES + + It was weird to add CMAKE_THREAD_LIBS_INIT in CMAKE_REQUIRED_LIBRARIES + only if CLOCK_MONOTONIC is available. Alternative would be to remove + the thread libs from CMAKE_REQUIRED_LIBRARIES after the check for + pthread_condattr_setclock() but keeping the libs should be fine too. + Then it's ready in case more pthread functions were wanted some day. + + CMakeLists.txt | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 068a70e54932ca32ca2922aff5a67a62615c650b +Author: Sam James +Date: 2024-06-24 19:25:30 +0100 + + CMake: Tweak comments + + Co-authored-by: Lasse Collin + + CMakeLists.txt | 15 +++++++-------- + 1 file changed, 7 insertions(+), 8 deletions(-) + +commit 3c95c93bca593bdd54ac5cc01526b12c82c78faa +Author: Lasse Collin +Date: 2024-06-24 22:42:01 +0300 + + CMake: Edit white space for consistency + + CMakeLists.txt | 26 +++++++++++++------------- + 1 file changed, 13 insertions(+), 13 deletions(-) + +commit 114cba69dbb96003e676c8c87a2e9943b12d065f +Author: Lasse Collin +Date: 2024-06-24 22:41:10 +0300 + + CMake: Fix three checks if building with -flto + + In CMake, check_c_source_compiles() always links too. With + link-time optimization, unused functions may get omitted if + main() doesn't depend on them. Consider the following which + tries to check if somefunction() is available when + has been included: + + #include + int foo(void) { return somefunction(); } + int main(void) { return 0; } + + LTO may omit foo() completely because the program as a whole doesn't + need it and then the program will link even if the symbol somefunction + isn't available in libc or other library being linked in, and then + the test may pass when it shouldn't. + + What happens if doesn't declare somefunction()? + Shouldn't the test fail in the compilation phase already? It should + but many compilers don't follow the C99 and later standards that + prohibit implicit function declarations. Instead such compilers + assume that somefunction() exists, compilation succeeds (with a + warning), and then linker with LTO omits the call to somefunction(). + + Change the tests so that they are part of main(). If compiler accepts + implicitly declared functions, LTO cannot omit them because it has to + assume that they might have side effects and thus linking will fail. + On the other hand, if the functions/intrinsics being used are supported, + they might get optimized away but in that case it's fine because they + really are supported. + + It is fine to use __attribute__((target(...))) for main(). At least + it works with GCC 4.9 to 14.1 on x86-64. + + Reported-by: Sam James + + CMakeLists.txt | 19 ++++++++----------- + 1 file changed, 8 insertions(+), 11 deletions(-) + +commit 78e882205e1f1e91df2af2cb7da00fe205dede99 +Author: Lasse Collin +Date: 2024-06-24 21:19:14 +0300 + + CMake: Use MATCHES instead of multiple STREQUAL + + CMakeLists.txt | 11 ++++------- + 1 file changed, 4 insertions(+), 7 deletions(-) + +commit d3f20382fc1bd865eb70a65455d5022ed05caac8 +Author: Lasse Collin +Date: 2024-06-24 21:06:18 +0300 + + CMake: Improve the comment about LIBS + + CMakeLists.txt | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 33ec377729a3889e58d98934b2777b2754a3e045 +Author: Lasse Collin +Date: 2024-06-24 20:01:25 +0300 + + CMake: Fix a typo in a message + + It was spotted with codespell. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2a47be823cd6c717bc91fa29c7710c9b1ae0331f +Author: Lasse Collin +Date: 2024-06-24 19:58:54 +0300 + + Document CMake options in INSTALL + + INSTALL | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----- + 1 file changed, 106 insertions(+), 9 deletions(-) + +commit 3faf4e8079a46bd46e05cd1234365724a6a33802 +Author: Lasse Collin +Date: 2024-06-24 17:18:44 +0300 + + CI: Don't omit crc32 from the list with CMake anymore + + XZ_CHECKS accepts it but works without too. + + build-aux/ci_build.bash | 10 +--------- + 1 file changed, 1 insertion(+), 9 deletions(-) + +commit 1bf83cded2955282fe1a868f08c83d4e5d6dca4a +Author: Lasse Collin +Date: 2024-06-24 17:39:54 +0300 + + CI: Workaround buggy config.guess on Ubuntu 22.04LTS and 24.04LTS + + Check for the wrong triplet from config.guess and override it with + the --build option on the configure command line. Then i386 assembly + autodetection will work. + + These Ubuntu versions (and as of writing, also Debian unstable) + ship config.guess version 2022-01-09 which contains a bug that + was fixed in version 2022-05-08. It results in a wrong configure + triplet when using CC="gcc -m32" to build i386 binaries. + + Upstream fix: + https://git.savannah.gnu.org/cgit/config.git/commit/?id=f56a7140386d08a531bcfd444d632b28c61a6329 + + More information: + https://mail.gnu.org/archive/html/config-patches/2022-05/msg00003.html + + build-aux/ci_build.bash | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit dbcdabf68fee9ed694b68c3a82e6adbeff20b679 +Author: Lasse Collin +Date: 2024-06-24 15:24:52 +0300 + + CI: Use CC="gcc -m32" to get i386 compiler on x86-64 + + The old method put it in CFLAGS which is a wrong place because + config.guess doesn't read CFLAGS. + + .github/workflows/ci.yml | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 0c1e6d900bac127464fb30a854776e1810ab5f16 +Author: Lasse Collin +Date: 2024-06-24 14:54:17 +0300 + + CI: Let CMake use the CC environment variable + + CC from environment is used to initialize CMAKE_C_COMPILER so + setting CMAKE_C_COMPILER explicitly isn't needed. + + The syntax in ci_build.bash was broken in case one wished to put + spaces in CC. + + build-aux/ci_build.bash | 4 ---- + 1 file changed, 4 deletions(-) + +commit a3d6eb797c1bd9b0425ef6754e475e43e62bf075 +Author: Lasse Collin +Date: 2024-06-20 23:25:42 +0300 + + CMake: Add autodetection for 32-bit x86 CRC assembly usage + + CMakeLists.txt | 33 ++++++++++++++++++--------------- + 1 file changed, 18 insertions(+), 15 deletions(-) + +commit dbc14f213e5cf866f1f42b7c6381a91e1189908c +Author: Lasse Collin +Date: 2024-06-20 23:00:59 +0300 + + CMake: Move option(XZ_ASM_I386) downwards a few lines + + CMakeLists.txt | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit e5c2b07b489b155c1bebd5cb5e5b94325c2fef1a +Author: Lasse Collin +Date: 2024-06-20 18:45:41 +0300 + + DOS: Update Makefile and config.h for the CRC changes + + dos/Makefile | 4 ++-- + dos/config.h | 3 +++ + 2 files changed, 5 insertions(+), 2 deletions(-) + +commit fe77c4e130d62dc3f9c1de40a18c0c6caa5a4d88 +Author: Lasse Collin +Date: 2024-06-23 15:35:35 +0300 + + liblzma: Tidy up crc_common.h + + Prefix ARM64_RUNTIME_DETECTION with CRC_ and reorder it to be with + the other ARM64-specific lines. That macro isn't used outside this + file. + + ARM64 CLMUL implementation doesn't exist yet and thus CRC64_ARM64_CLMUL + isn't used anywhere yet. + + It's not ideal that the single-letter CRC utility macros are here + as they pollute the namespace of the LZ encoder files. Those could + be moved their own crc_macros.h like they were in 5.2.x but in practice + this is fine enough already. + + src/liblzma/check/crc_common.h | 62 ++++++++++++++++++++++++++++-------------- + 1 file changed, 42 insertions(+), 20 deletions(-) + +commit 7484d375384f551d475ff44a93590a225e0cb8f6 +Author: Lasse Collin +Date: 2024-06-23 14:22:08 +0300 + + liblzma: Move lzma_crcXX_table[][] declarations to crc_common.h + + LZ encoder needs lzma_crc32_table[0] but otherwise those tables + are private to the CRC code. In contrast, the other things in + check.h are needed in several places. + + src/liblzma/check/check.h | 18 ------------------ + src/liblzma/check/crc32_small.c | 3 +++ + src/liblzma/check/crc_common.h | 18 ++++++++++++++++++ + src/liblzma/lz/lz_encoder_hash.h | 4 ++-- + 4 files changed, 23 insertions(+), 20 deletions(-) + +commit 85b081f5d4598342b8c155a2c08697fb2adc372c +Author: Lasse Collin +Date: 2024-06-19 18:38:22 +0300 + + liblzma: Make 32-bit x86 CRC assembly co-exist with CLMUL + + Now runtime detection of CLMUL support can pick between the CLMUL and + the generic assembly implementations. Whatever overhead this has for + builds that omit CLMUL completely isn't important because builds for + any non-ancient system is likely to include the CLMUL code too. + + Handle the CRC tables in crcXX_fast.c files because now these files + are built even when assembly code is used. + + If 32-bit x86 assembly is enabled then it will always be built even + if compiler flags were such that CLMUL would be allowed unconditionally. + That is, runtime detection will be used anyway. This keeps the build + rules simpler. + + In LZ encoder, build and use lzma_lz_hash_table[256] if CLMUL CRC + is used without runtime detection. Previously this wasn't needed + because crc32_table.c included the lzma_crc32_table[][] in the build + unless encoder support had been disabled. Including an 8 KiB table + was silly when only 1 KiB is actually used. So now liblzma is 7 KiB + smaller if CLMUL is enabled without runtime detection. + + CMakeLists.txt | 8 ++------ + src/liblzma/check/Makefile.inc | 8 ++------ + src/liblzma/check/crc32_fast.c | 14 ++++++++++++- + src/liblzma/check/crc32_table.c | 42 --------------------------------------- + src/liblzma/check/crc32_x86.S | 14 +++++-------- + src/liblzma/check/crc64_fast.c | 18 +++++++++++++---- + src/liblzma/check/crc64_table.c | 37 ---------------------------------- + src/liblzma/check/crc64_x86.S | 14 +++++-------- + src/liblzma/check/crc_common.h | 18 +++++++++-------- + src/liblzma/check/crc_x86_clmul.h | 5 ----- + src/liblzma/lz/lz_encoder.c | 2 +- + src/liblzma/lz/lz_encoder_hash.h | 30 ++++++++++++++++++++-------- + 12 files changed, 74 insertions(+), 136 deletions(-) + +commit 6667d503b5dc9826654e3d9ad505e1883ff6c388 +Author: Lasse Collin +Date: 2024-06-19 17:44:41 +0300 + + liblzma: CRC: Rename crcXX_generic to lzma_crcXX_generic + + This prepares for the possibility that lzma_crc32_generic and + lzma_crc64_generic are extern functions. + + src/liblzma/check/crc32_fast.c | 6 +++--- + src/liblzma/check/crc64_fast.c | 6 +++--- + 2 files changed, 6 insertions(+), 6 deletions(-) + +commit 1dca581ff20aa1cde61e9e5267d3aeb0af9b6845 +Author: Lasse Collin +Date: 2024-06-20 22:55:22 +0300 + + CMake: Define HAVE_CRC_X86_ASM when 32-bit x86 CRC assembly is used + + CMakeLists.txt | 3 +++ + 1 file changed, 3 insertions(+) + +commit f76837acb65676e541d8ee79cd62dbbf27280a62 +Author: Lasse Collin +Date: 2024-05-10 16:00:26 +0300 + + Build: Define HAVE_CRC_X86_ASM when 32-bit x86 CRC assembly is used + + This makes it easier to determine when the CRC tables are needed. + + configure.ac | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +commit 9ce0866b070850da4dc837741ff055faa218bdd6 +Author: Lasse Collin +Date: 2024-06-21 00:46:09 +0300 + + CI: Update to the new renamed options in CMakeLists.txt + + build-aux/ci_build.bash | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit 0232e66d5bc5b01a25a447c657e51747626488ab +Author: Lasse Collin +Date: 2024-06-20 18:12:22 +0300 + + CMake: Add XZ_EXTERNAL_SHA256 + + CMakeLists.txt | 121 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--- + 1 file changed, 116 insertions(+), 5 deletions(-) + +commit 4535b80caead82a7ddf7feb988b8fbc773152522 +Author: Lasse Collin +Date: 2024-06-20 18:12:21 +0300 + + CMake: Move threading detection a few lines up + + It feels clearer this way, and when support for external SHA-256 + is added, this will keep the order of the library detection the + same as in configure.ac (check for pthreads before libmd) although + it shouldn't matter in practice. + + CMakeLists.txt | 176 ++++++++++++++++++++++++++++----------------------------- + 1 file changed, 88 insertions(+), 88 deletions(-) + +commit 94d062dbac34d366eb26625034200cc3457e6645 +Author: Lasse Collin +Date: 2024-06-20 18:12:21 +0300 + + CMake: Move the sandbox code out of the liblzma section + + Sandboxing is for the command line tools, not liblzma. + No functional changes. + + CMakeLists.txt | 214 ++++++++++++++++++++++++++++----------------------------- + 1 file changed, 107 insertions(+), 107 deletions(-) + +commit 75ce4797d49621710e6da95d8cb91541028c6d68 +Author: Lasse Collin +Date: 2024-06-20 18:12:21 +0300 + + CMake: Keep existing options in LIBS when adding -lrt + + This makes no difference yet because -lrt is currently the only option + that might be added to LIBS. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 47aaa92516fd9609821d04e5e94ca6558e56d62b +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Don't install scripts if the xz tool isn't built + + The scripts need the xz tool. + + CMakeLists.txt | 11 +++++++++-- + tests/tests.cmake | 2 +- + 2 files changed, 10 insertions(+), 3 deletions(-) + +commit fb50c6ba1d4c9405e5b12b5988b01a3002638c5d +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add XZ_TOOL_XZDEC and XZ_TOOL_LZMADEC + + CMakeLists.txt | 15 ++++++++++++++- + 1 file changed, 14 insertions(+), 1 deletion(-) + +commit def767f7d18ccbd81cd5e5b46c8b6031f3a1de34 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add XZ_TOOL_LZMAINFO + + CMakeLists.txt | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 5600e370fb7e11eafabc6c3ef5bf6510e859f4f0 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add XZ_TOOL_XZ + + CMakeLists.txt | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 6a3c4aaa43a90da441e1156c5ffd2e6098f5521f +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + Windows: Drop Visual Studio 2013 support + + This simplifies things a little. Building liblzma with VS2013 probably + still worked but building the command line tools was not supported. + + Microsoft ended support for VS2013 on 2024-04. + + CMakeLists.txt | 9 +++++++-- + src/common/sysdefs.h | 6 +----- + windows/INSTALL-MSVC.txt | 8 ++------ + 3 files changed, 10 insertions(+), 13 deletions(-) + +commit 5d5c92b26246936461a635dda1f95740d7de2058 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add XZ_TOOL_SCRIPTS + + CMakeLists.txt | 44 +++++++++++++++++++++++++++++--------------- + 1 file changed, 29 insertions(+), 15 deletions(-) + +commit d274a2bc00d235f07e96aaf82c149794cfe82b12 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add XZ_DOC + + CMakeLists.txt | 45 ++++++++++++++++++++++++--------------------- + 1 file changed, 24 insertions(+), 21 deletions(-) + +commit 188143a50ade67253ed256608f50f78aa1380403 +Author: Lasse Collin +Date: 2024-06-20 21:53:03 +0300 + + CMake: Refactor XZ_SYMBOL_VERSIONING to match configure.ac + + Make the available options and their behavior match + --enable-symbol-versions in configure.ac. + + Don't enable symbol versions on Linux if not using glibc. Previously + the generic variant was selected on Microblaze or if using NVHPC + without checking that libc is glibc. + + Leave the cache variable to "auto" or "yes" if that was specified + instead of setting it to the autodetected value by default. A downside + is that one cannot easily see which variant the autodetection code + has selected. The same applies to XZ_SANDBOX and XZ_THREADS though. + + CMakeLists.txt | 125 ++++++++++++++++++++++++++++++++++----------------------- + 1 file changed, 75 insertions(+), 50 deletions(-) + +commit cc52ef8ed3b75a581262c587f6c06c213a550f86 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Use the same option list for XZ_THREADS as in configure.ac + + Also clarify that "yes" will fail if no threading support is found. + If no threading is wanted, it has to be disabled manually. + + configure.ac doesn't behave this way at the moment. Instead it + assumes pthreads to be present if not targeting Windows. If pthreads + actually are missing, the build fails later. + + CMakeLists.txt | 18 ++++++++++-------- + 1 file changed, 10 insertions(+), 8 deletions(-) + +commit 37f7af3452bab0a34ce320c2ad532835f18752d9 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Use the same option list for XZ_SANDBOX as in configure.ac + + It's simpler to document this way. + + CMakeLists.txt | 20 ++++++++++---------- + 1 file changed, 10 insertions(+), 10 deletions(-) + +commit c715dec8e800b65145918cfb0ee9bbc90faa8aad +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Fix indentation + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ea379f2f180befabd2039342db8eaeb757fdd2b7 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add warning options for GCC and Clang + + The list was copied from configure.ac and should be kept in sync. + (Pretend that the deleted comment in CMakeLists.txt didn't exist.) + + There is no need to add equivalent of --enable-werror as CMake >= 3.24 + supports -DCMAKE_COMPILE_WARNING_AS_ERROR=ON. + + CMakeLists.txt | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++----- + 1 file changed, 59 insertions(+), 5 deletions(-) + +commit 74223338197b7dfcd69f56df78b6502805a75f23 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Use \040 instead of \x20 for a space + + This is for consistency with 4c81c9611f8b2e1ad65eb7fa166afc570c58607e + where \040 has to be used because \0x20F gets interpret at three hex + digits. Octals escapes are never longer than three digits. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit e8854b6bdc956c46dc4232bd07c17163034a00f2 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Add XZ_ASSUME_RAM + + CMakeLists.txt | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit e1127e75cb82e0385f02c995771d6fe1420f43c5 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename liblzma_INSTALL_CMAKEDIR to XZ_INSTALL_CMAKEDIR + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 96abfe98c15e431a50a6a31015c5bb05540ab2ff +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Refactor ADDITIONAL_CHECK_TYPES to XZ_CHECKS + + Now "crc32" is in the list too for completeness but it doesn't + actually have any effect. The description of the cache variable + says that "crc32 is always built" so it should be clear enough. + + CMakeLists.txt | 14 +++++++------- + tests/tests.cmake | 17 ++++++++--------- + 2 files changed, 15 insertions(+), 16 deletions(-) + +commit 679500ffe00ecb4f02292129e7529ab7392f3943 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename the cache variable POSIX_SHELL to XZ_POSIX_SHELL + + We still need the variable POSIX_SHELL for configure_file() + but it doesn't need to be a cache variable. + + CMakeLists.txt | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +commit e5c0eb2e50e5522a0a55e7ba83fe49b04c8a6eef +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENCODERS and DECODERS to use XZ_ prefix + + CMakeLists.txt | 34 +++++++++++++++++----------------- + tests/tests.cmake | 4 ++-- + 2 files changed, 19 insertions(+), 19 deletions(-) + +commit e7785e2061f95d44aa6c0856b09cc0fbad7d6154 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename MATCH_FINDERS to XZ_MATCH_FINDERS + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 63294806b488a27a28a0960f6a257695dd2b569a +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename SYMBOL_VERSIONING to XZ_SYMBOL_VERSIONING + + CMakeLists.txt | 9 +++++---- + 1 file changed, 5 insertions(+), 4 deletions(-) + +commit ad245b133675d285bca5d48123062e9d1e3f747e +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENABLE_THREADS to XZ_THREADS + + CMakeLists.txt | 24 +++++++++++------------- + 1 file changed, 11 insertions(+), 13 deletions(-) + +commit 4250d4de32e66e558cc2ebe73b05255633c933ed +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENABLE_SANDBOX to XZ_SANDBOX + + CMakeLists.txt | 23 +++++++++++------------ + 1 file changed, 11 insertions(+), 12 deletions(-) + +commit 0fdcd0c582f1a38542cd647dde449d9447d5888d +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENABLE_X86_ASM to XZ_ASM_I386 + + CMakeLists.txt | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit e017d5526e316003fdb2a3f76acbb83443f14ddf +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename CREATE_XZ_SYMLINKS to XZ_TOOL_SYMLINKS + + This only affects the names unxz and xzcat. The xz-prefixed script + symlinks (xzfgrep and such) are always created if scripts are enabled. + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 04cac14fcb9fb302c24e90b04ca4b77d3717b50c +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename CREATE_LZMA_SYMLINKS to XZ_TOOL_LZMA_SYMLINKS + + Update the description too. + + It affects creation of not only the legacy lzma, unlzma, lzcat symlinks + but also lzgrep and other legacy names for the scripts. The last + LZMA Utils release was made in 2008 but these names are still used + in some places to handle .lzma files. + + CMakeLists.txt | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +commit 612ccebf884eb1a9b6848e230c24f97a03fe917a +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ALLOW_ARM64_CRC32 to XZ_ARM64_CRC32 + + Update description too. + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 3dcc12290d6dffbe7f10f501c141d325bad65901 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ALLOW_CLMUL_CRC to XZ_CLMUL_CRC + + Update description too. + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 4b8faa72442da9aa1a356f5848aae798d8588a7d +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENABLE_DOXYGEN to XZ_DOXYGEN + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit b56273ae575bac350e50b0c689269dcab04b04b3 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename LZIP_DECODER to XZ_LZIP_DECODER + + CMakeLists.txt | 4 ++-- + tests/tests.cmake | 2 +- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 2343992fcbe8b436da6df888be37713cccaff0ab +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename MICROLZMA_ENCODER/DECODER to XZ_MICROLZMA_ENCODER/DECODER + + CMakeLists.txt | 8 ++++---- + tests/tests.cmake | 2 +- + 2 files changed, 5 insertions(+), 5 deletions(-) + +commit 96f0a6632cc0598a26d93255b0c444df18dc7891 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENABLE_SMALL to XZ_SMALL + + CMakeLists.txt | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +commit 29f77c7b707f2458fb047e77497354b195e05b14 +Author: Lasse Collin +Date: 2024-06-15 18:07:04 +0300 + + CMake: Rename ENABLE_NLS to XZ_NLS + + Also update the description to mention that this affects installation + of translated man pages too. + + Prefixing the cache variables with the project name helps if + the package is used as a subproject in another package. + It also makes the package-specific options group more nicely + in ccmake and cmake-gui. + + CMakeLists.txt | 28 +++++++++++++++------------- + 1 file changed, 15 insertions(+), 13 deletions(-) + +commit ac05f1b0d7cda1e7ae79775a8dfecc54601d7f1c +Author: Lasse Collin +Date: 2024-06-15 23:34:29 +0300 + + CMake: Link Threads::Threads as PRIVATE to liblzma + + This way pthread options aren't passed to the linker when linking + against shared liblzma but they are still passed when linking against + static liblzma. (Also, one never needs the include path of the + threading library to use liblzma since liblzma's API headers + don't #include . But tends to be in the + default include path so here this change makes no difference.) + + One cannot mix target_link_libraries() calls that use the scope + (PRIVATE, PUBLIC, or INTERFACE) keyword and calls that don't use it. + The calls without the keyword are like PUBLIC except perhaps when + they aren't, or something like that... It seems best to always + specify a scope keyword as the meanings of those three keywords + at least are clear. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 82986d8c691a294c78b48d8391303e5c428b5437 +Author: Lasse Collin +Date: 2024-06-16 19:39:32 +0300 + + CMake: Add empty lines + + CMakeLists.txt | 2 ++ + 1 file changed, 2 insertions(+) + +commit 2aecffe0f0e14f3ef635e8cd7b405420f2385de2 +Author: Lasse Collin +Date: 2024-06-16 19:37:36 +0300 + + CMake: Use CMAKE_THREAD_LIBS_INIT in liblzma.pc only with pthreads + + This shouldn't make much difference in practice as on Windows + no flags are needed anyway and unitialized variable (when threading + is disabled) expands to empty. But it's clearer this way. + + CMakeLists.txt | 8 +++++++- + 1 file changed, 7 insertions(+), 1 deletion(-) + +commit 664918bd3635ea8e773f06022286ecb0c485166c +Author: Lasse Collin +Date: 2024-06-17 18:20:14 +0300 + + Update THANKS + + THANKS | 3 +++ + 1 file changed, 3 insertions(+) + +commit 5ca96a93488d0f5a530c78b274cac317453807ff +Author: Lasse Collin +Date: 2024-06-16 19:25:07 +0300 + + CMake: Use native newlines in liblzma.pc + + vcpkg doesn't specify the newline type so it should be fine to + use native newlines in liblzma.pc on Windows. + + CMakeLists.txt | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +commit ebd155c3a1b87411edae06d3bdaa9659ec057522 +Author: Lasse Collin +Date: 2024-06-16 19:18:56 +0300 + + CMake: Use relative paths in liblzma.pc if possible + + Now liblzma.pc can be relocatable only if using CMake >= 3.20 + but that should be OK as now we shouldn't get broken liblzma.pc + if CMAKE_INSTALL_LIBDIR or CMAKE_INSTALL_INCLUDEDIR contain an + absolute path. + + Thanks to Eli Schwartz. + + CMakeLists.txt | 18 ++++++++++++++---- + 1 file changed, 14 insertions(+), 4 deletions(-) + +commit 7a366d93cfd74ce10201db400be8836199944e36 +Author: Lasse Collin +Date: 2024-06-16 18:33:08 +0300 + + Revert "CMake: Set only "prefix" as an absolute path in liblzma.pc" + + This reverts commit 5d1c649ba9eb7a5b9371252ebfbc2911dc774e69. + + While CMAKE_INSTALL_ tend to be relative paths, they don't need + to be. Thus the commit was broken. A fancier method is required. + + Thanks to Eli Schwartz for the bug report and explanation. + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 30a2d5d51006301a3ddab5ef1f5ff0a9d74dce6f +Author: Lasse Collin +Date: 2024-06-16 13:39:37 +0300 + + liblzma: CRC CLMUL: Omit is_arch_extension_supported() when not needed + + On E2K the function compiles only due to compiler emulation but the + function is never used. It's cleaner to omit the function when it's + not needed even though it's a "static inline" function. + + Thanks to Ilya Kurdyukov. + + src/liblzma/check/crc_x86_clmul.h | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 54eaea5ea49bb8bca4286d4412f19ac73187489e +Author: Lasse Collin +Date: 2024-06-16 13:21:34 +0300 + + liblzma: x86 CLMUL CRC: Rewrite + + It's faster with both tiny and large buffers and doesn't require + disabling any sanitizers. With large buffers the extra speed is + from folding four 16-byte chunks in parallel. + + The 32-bit x86 with MSVC reportedly still needs a workaround. + Now the simpler "__asm mov ebx, ebx" trick is enough but it + needs to be in lzma_crc64() instead of crc64_arch_optimized(). + Thanks to Iouri Kharon for testing and the fix. + + Thanks to Ilya Kurdyukov for testing the speed with aligned and + unaligned buffers on a few x86 processors and on E2K v6. + + Thanks to Sam James for general feedback. + + Fixes: https://github.com/tukaani-project/xz/issues/112 + Fixes: https://github.com/tukaani-project/xz/issues/122 + + src/liblzma/check/crc64_fast.c | 8 + + src/liblzma/check/crc_x86_clmul.h | 437 ++++++++++++++++++++------------------ + 2 files changed, 237 insertions(+), 208 deletions(-) + +commit c0e7eaae8d6eef1e313c9d0da20ccf126ec61f38 +Author: Lasse Collin +Date: 2024-06-01 14:44:04 +0300 + + sysdefs.h: Add alignas + + src/common/sysdefs.h | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit 20014c261451381d5e2f58e63e7b1fbefd4df4bf +Author: Lasse Collin +Date: 2024-06-11 12:47:59 +0300 + + liblzma: Use a single macro to select CLMUL CRC to build + + This way it's clearer that two things cannot be selected + at the same time. + + src/liblzma/check/crc32_fast.c | 2 +- + src/liblzma/check/crc64_fast.c | 2 +- + src/liblzma/check/crc_x86_clmul.h | 18 ++++++++++-------- + 3 files changed, 12 insertions(+), 10 deletions(-) + +commit d8fb0986171bd6a3066b236fc9a6b3d573c8e441 +Author: Lasse Collin +Date: 2024-06-10 15:31:01 +0300 + + liblzma: CRC32 CLMUL: Refactor the constants and simplify + + By using modulus scaled constants, the final reduction can + be simplified. + + src/liblzma/check/crc_x86_clmul.h | 52 +++++++-------------------------------- + 1 file changed, 9 insertions(+), 43 deletions(-) + +commit ef652ac391ff7e8cda656238dc5b5f83bc1554c2 +Author: Lasse Collin +Date: 2024-06-10 15:12:48 +0300 + + liblzma: CRC64 CLMUL: Refactor the constants + + Now it refers to crc_clmul_consts_gen.c. vfold8 was renamed to mu_p + and the p no longer has the lowest bit set (it makes no difference + as the output bits it affects are ignored). + + src/liblzma/check/crc_x86_clmul.h | 43 +++++++-------------------------------- + 1 file changed, 7 insertions(+), 36 deletions(-) + +commit 9f5fc17e32bf5c7c6cfadf40c29a1dedb4cc03ac +Author: Lasse Collin +Date: 2024-06-10 14:45:44 +0300 + + liblzma: Add crc_clmul_consts_gen.c + + It's a standalone program that prints the required constants. + It's won't be a part of the normal build of the package. + + src/liblzma/check/Makefile.inc | 1 + + src/liblzma/check/crc_clmul_consts_gen.c | 160 +++++++++++++++++++++++++++++++ + 2 files changed, 161 insertions(+) + +commit 71b147aab7fe4a60ed57b697d5bb490f099894be +Author: Lasse Collin +Date: 2024-05-09 21:44:03 +0300 + + liblzma: Remove CRC_USE_GENERIC_FOR_SMALL_INPUTS + + It was already commented out. + + src/liblzma/check/crc32_fast.c | 21 --------------------- + src/liblzma/check/crc64_fast.c | 5 ----- + src/liblzma/check/crc_common.h | 14 -------------- + src/liblzma/check/crc_x86_clmul.h | 9 +-------- + 4 files changed, 1 insertion(+), 48 deletions(-) + +commit f99a7be40645f86959a5b180dfae948dd165e07c +Author: Lasse Collin +Date: 2024-05-09 21:03:39 +0300 + + liblzma: Remove crc_attr_no_sanitize_address + + It's not enough to silence the address sanitizer. Also memory and + thread sanitizers would need to be silenced. They, at least currently, + aren't smart enough to see that the extra bytes are discarded from + the xmm registers by later instructions. + + Valgrind is smarter, possibly because this kind of code isn't weird + to write in assembly. Agner Fog's optimizing_assembly.pdf even mentions + this idea of doing an aligned read and then discarding the extra + bytes. The sanitizers don't instrument assembly code but Valgrind + checks all code. + + It's better to change the implementation to avoid the sanitization + attributes which also look scary in the code. (Somehow they can look + more scary than __asm__ which is implictly unsanitized.) + + See also: + https://github.com/tukaani-project/xz/issues/112 + https://github.com/tukaani-project/xz/issues/122 + + src/liblzma/check/crc_common.h | 9 --------- + src/liblzma/check/crc_x86_clmul.h | 3 --- + 2 files changed, 12 deletions(-) + +commit ead4d151996f8a18bf9b07eb1e175c0a1590e562 +Author: Lasse Collin +Date: 2024-06-10 15:37:49 +0300 + + Revert "Build: Temporarily disable CRC CLMUL to silence OSS Fuzz" + + This reverts commit 9f1a6d6f9a258886933a22239a5b81af34b28199. + + configure.ac | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +commit 2178acf8a4d40a93e970cfcf9b807d5ef6c8da92 +Author: Lasse Collin +Date: 2024-06-12 14:26:44 +0300 + + CMake: Prefer C11 with a fallback to C99 + + There is no need to make a similar change in configure.ac. + With Autoconf 2.72, the deprecated macro AC_PROG_CC_C99 + is an alias for AC_PROG_CC which prefers a C11 compiler. + + CMakeLists.txt | 17 +++++++++++------ + 1 file changed, 11 insertions(+), 6 deletions(-) + +commit c97e9c12fef4d1093ee2a75236742481361f50f5 +Author: Lasse Collin +Date: 2024-06-12 14:20:21 +0300 + + Update THANKS + + THANKS | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 89e9f12e03324b8a186e807b268f34f92d1b2f41 +Author: Lasse Collin +Date: 2024-06-11 11:15:49 +0300 + + Tests: Improve the CRC32 test + + A similar one was already there for CRC64 but nowadays also CRC32 + has a CLMUL implementation, so it's good to test it better too. + + tests/test_check.c | 17 +++++++++++++---- + 1 file changed, 13 insertions(+), 4 deletions(-) + +commit c7164b1927e3fe7cdba70ee4687e1a590a81043b +Author: Lasse Collin +Date: 2024-06-11 22:42:26 +0300 + + xz: Fix white space + + src/xz/list.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 0a32d2072c598de281058b26dc08920fbf0cd2a1 +Author: Lasse Collin +Date: 2024-06-11 21:59:09 +0300 + + liblzma: Fix a typo in a comment + + Thanks to Sam James for spotting it. + + Fixes: f644473a211394447824ea00518d0a214ff3f7f2 + + src/liblzma/check/crc_x86_clmul.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit afd9b4d282a10186808c3331dad4caf79c02d55f +Author: Lasse Collin +Date: 2024-05-10 15:52:26 +0300 + + liblzma: Fix a comment indentation + + src/liblzma/check/crc_common.h | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 50e6bff274568c568930e15094da8217e7d47d28 +Author: Lasse Collin +Date: 2024-05-09 22:09:12 +0300 + + liblzma: Fix white space + + src/liblzma/check/crc32_table.c | 10 +++++----- + src/liblzma/check/crc_x86_clmul.h | 6 +++--- + src/liblzma/check/sha256.c | 2 +- + 3 files changed, 9 insertions(+), 9 deletions(-) + +commit caea7844d3824755d053b4743c4913d73ac2db3d +Author: Lasse Collin +Date: 2024-06-01 14:25:29 +0300 + + tuklib: __STDC_VERSION__ in C23 is 202311 + + src/common/tuklib_common.h | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +commit 9e73918a4f14be754a23f74dda45ca431939a4a0 +Author: RainRat +Date: 2024-06-05 15:21:49 -0700 + + Fix typos + + Closes: https://github.com/tukaani-project/xz/pull/124 + + INSTALL | 2 +- + doc/examples/03_compress_custom.c | 2 +- + src/common/tuklib_integer.h | 2 +- + src/liblzma/api/lzma/container.h | 2 +- + src/xz/mytime.c | 2 +- + tests/test_filter_str.c | 2 +- + 6 files changed, 6 insertions(+), 6 deletions(-) + +commit 04b23addf3733873667675df2439725f076c2f36 +Author: Lasse Collin +Date: 2024-06-07 15:47:20 +0300 + + tuklib_integer: Fix building on OpenBSD/sparc64 that uses GCC 4.2 + + GCC 4.2 doesn't have __builtin_bswap16() and friends so tuklib_integer.h + tries to use OS-specific byte swap methods instead. On OpenBSD those + macros are swap16/32/64 instead of bswap16/32/64 like on other *BSDs + and Darwin. + + An alternative to "#ifdef __OpenBSD__" could be "#ifdef swap16" as it + is a macro. But since OpenBSD seems to be a special case under this + special case of "*BSDs and Darwin", checking for __OpenBSD__ seems + the more conservative choice now. + + Thanks to Christian Weisgerber and Brad Smith who both submitted + the same patch a few hours apart. + + Co-authored-by: Christian Weisgerber + Co-authored-by: Brad Smith + Closes: https://github.com/tukaani-project/xz/pull/126 + + src/common/tuklib_integer.h | 12 +++++++++--- + 1 file changed, 9 insertions(+), 3 deletions(-) + +commit dc03f6290f5b9bd3d50c7e12e58dee870889d599 +Author: Lasse Collin +Date: 2024-06-07 15:06:59 +0300 + + liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD + + The C code is from Christian Weisgerber, I merely reordered the OSes. + Then I added the build system checks without testing them. + + Also thanks to Brad Smith who submitted a similar patch on GitHub + a few hours after Christian had sent his via email. + + Co-authored-by: Christian Weisgerber + Closes: https://github.com/tukaani-project/xz/pull/125 + + CMakeLists.txt | 6 ++++++ + configure.ac | 9 +++++++++ + src/liblzma/check/crc32_arm64.h | 15 +++++++++++++++ + src/liblzma/check/crc_common.h | 1 + + 4 files changed, 31 insertions(+) + +commit f5c2ae58ec68c665e62c790b842657afcb31474c +Author: Lasse Collin +Date: 2024-06-05 13:55:43 +0300 + + Update THANKS + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit e5491dfab9c54dc7078a8d3d07fabb91d6e06418 +Author: Lasse Collin +Date: 2024-06-05 13:42:47 +0300 + + CMake: Include the "alpha" or "beta" suffix in PACKAGE_VERSION + + This way the version string gets into xzgrep and other scripts + in full and also into liblzma.pc. + + For the project() command, a suffixless string is required though. + + CMakeLists.txt | 16 +++++++++++++--- + 1 file changed, 13 insertions(+), 3 deletions(-) + +commit 1d3c61575fda0be6b2d50c9e32a343349d5cd5c0 +Author: Lasse Collin +Date: 2024-06-05 13:30:28 +0300 + + CMake: Fix wrong version variable + + liblzma_VERSION has never existed in the repository. xz_VERSION from + the project() command was used for liblzma SOVERSION so use xz_VERSION + here too. + + The wrong variable did no harm in practice as PROJECT_VERSION + was used as the fallback. It has the same value as xz_VERSION. + + Fixes: 7e3493d40eac0c3fa3d5124097745a70e15c41f6 + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 5d1c649ba9eb7a5b9371252ebfbc2911dc774e69 +Author: Lasse Collin +Date: 2024-06-05 12:59:59 +0300 + + CMake: Set only "prefix" as an absolute path in liblzma.pc + + CMake provides variables that are relative to CMAKE_INSTALL_PREFIX + so use them instead of repeating the full path. + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit e0d6d05ce0d464e966c0669bbf869202a43cc2f7 +Author: Lasse Collin +Date: 2024-06-04 23:59:29 +0300 + + CMake: Fix liblzma filename in Windows environments + + This is a mess because liblzma DLL outside Cygwin and MSYS2 + is liblzma.dll instead of lzma.dll to avoid a conflict with + lzma.dll from LZMA SDK. + + On Cygwin the name was "liblzma-5.dll" while "cyglzma-5.dll" + would have been correct (and match what Libtool produces). + MSYS2 likely was broken too as it uses the "msys-" prefix. + + This change has no effect with MinGW-w64 because with that + the "lib" prefix was correct already. + + With MSVC builds this is a small breaking change that requires developers + to adjust the library name when linking against liblzma. The liblzma.dll + name is kept as is but the import library and static library are now + lzma.lib instead of liblzma.lib. This is helpful when using pkgconf + because "pkgconf --msvc-syntax --libs liblzma" outputs "lzma.lib" + (it's converted from "-llzma" in liblzma.pc). It would be easy to + keep the liblzma.lib naming but the pkgconf compatibility seems worth + it in the long run. The lzma.lib name is compatible with MinGW-w64 + too as -llzma will find also lzma.lib. + + vcpkg had been patching CMakeLists.txt this way since 2022 but I + learned this only recently. The reasoning for the patch makes sense, + and while this is a small breaking change with MSVC, it seems like + a decent compromise as it keeps the DLL name the same. + + 2022 patch in vcpkg: https://github.com/microsoft/vcpkg/blob/0707a17ecf1466d64cf1a3c1ee18c8ff02aadb2d/ports/liblzma/win_output_name.patch + See the discussion: https://github.com/microsoft/vcpkg/pull/39024 + + Thanks to Vincent Torri for confirming the naming issue on Cygwin. + + CMakeLists.txt | 34 ++++++++++++++++++++++++++++++---- + 1 file changed, 30 insertions(+), 4 deletions(-) + +commit e7a42cda7c827e016619e8cab15e2faf5d4181ae +Author: Lasse Collin +Date: 2024-06-03 16:55:03 +0300 + + Fix version.sh compatiblity with Solaris + + The ancient /bin/tr on Solaris doesn't support '\n'. + With /usr/xpg4/bin/tr it works but it might not be in PATH. + + Another problem was that sed was given input that didn't have a newline + at the end. Text files must end with a newline to be portable. + + Fix both problems: + + - Handle multiline input within sed itself to avoid one tr invocation. + The default sed even on Solaris does understand \n. + + - Use octals in tr -d. \012 works for ASCII "line feed", it's even + used as an example in the Solaris man page. But we must strip + also ASCII "carriage return" \015 and EBCDIC "next line" \025. + The EBCDIC case got handled with \n previously. Stripping \012 + and \015 on EBCDIC system won't matter as those control chars + won't be present in the string in the first place. + + An awk-based solution could be an alternative but it might need + special casing on Solaris to used nawk instead of awk. The changes + in this commit are smaller and should have a smaller risk for + regressions. It's also possible that version.sh will be dropped + entirely at some point. + + build-aux/version.sh | 5 ++--- + 1 file changed, 2 insertions(+), 3 deletions(-) + +commit a61c9ab4751f2710dcd5459c7d74bbf20781f0f9 +Author: Lasse Collin +Date: 2024-06-03 17:07:11 +0300 + + CI: Don't require po4a on Solaris + + .github/workflows/solaris.yml | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 5229bdf5335ce18ed54beb7e646e39927663be86 +Author: Lasse Collin +Date: 2024-06-03 15:08:15 +0300 + + CI: Use set -e on Solaris too + + .github/workflows/solaris.yml | 1 + + 1 file changed, 1 insertion(+) + +commit afa938e429c1ce07d26d02999352fb014b62ff3d +Author: Lasse Collin +Date: 2024-06-03 17:44:50 +0300 + + CMake: Install liblzma.pc even with MSVC + + I had misunderstood that it wouldn't be useful with MSVC. + vcpkg had been installing liblzma.pc with custom rules since 2020, + years before liblzma.pc support was added to CMakeLists.txt. + + See: + https://github.com/microsoft/vcpkg/blob/eb895b95aac6fd7485373702f29f508c42a180a0/ports/liblzma/portfile.cmake + https://github.com/microsoft/vcpkg/pull/39024#issuecomment-2145064670 + + CMakeLists.txt | 8 +++----- + 1 file changed, 3 insertions(+), 5 deletions(-) + +commit 35f8649f08341639a627fd06350e938124ca3622 +Author: Sam James +Date: 2024-06-03 06:16:23 +0100 + + ci: don't pin official GH actions via commit, just tag + + There's no real value in doing it via commit for official GH actions. We + can keep using pinned commits for unofficial actions. It's hassle for no + gain. + + Maybe going forward we can limit this further by only being paranoid + for the jobs with any access to tokens. + + .github/workflows/ci.yml | 4 ++-- + .github/workflows/freebsd.yml | 2 +- + .github/workflows/netbsd.yml | 2 +- + .github/workflows/openbsd.yml | 2 +- + .github/workflows/solaris.yml | 2 +- + .github/workflows/windows-ci.yml | 4 ++-- + 6 files changed, 8 insertions(+), 8 deletions(-) + +commit e885dae37ff5b1dbc760dabc1e03e866a7302ef2 +Author: Christoph Junghans +Date: 2024-04-30 07:49:26 -0600 + + ci: set -e on openbsd + + Closes: https://github.com/tukaani-project/xz/pull/116 + + .github/workflows/openbsd.yml | 1 + + 1 file changed, 1 insertion(+) + +commit 21b02dd128cf9e8c76325ec124f70381862dcf19 +Author: Christoph Junghans +Date: 2024-04-30 07:48:58 -0600 + + ci: set -e on netbsd + + .github/workflows/netbsd.yml | 1 + + 1 file changed, 1 insertion(+) + +commit 8641f0c24c041136670c975b23408184b45431bc +Author: Christoph Junghans +Date: 2024-04-25 14:56:06 -0700 + + ci: actually fail on FreeBSD + + Without "set -e" the job will always be successful. + + See vmactions/freebsd-vm#72 + + .github/workflows/freebsd.yml | 1 + + 1 file changed, 1 insertion(+) + +commit ef616683ef11f11ffdfbe0624da33905e28a70f9 +Author: Andrew Murray +Date: 2024-04-25 09:24:46 +1000 + + Updated actions + + Closes: https://github.com/tukaani-project/xz/pull/115 + + .github/workflows/ci.yml | 4 ++-- + .github/workflows/windows-ci.yml | 6 +++--- + 2 files changed, 5 insertions(+), 5 deletions(-) + +commit 57b440d316da9ac9cb312ee7e6890f5382556f10 +Author: Sam James +Date: 2024-06-03 02:49:40 +0100 + + ci: add po4a + + .github/workflows/netbsd.yml | 2 +- + .github/workflows/openbsd.yml | 3 ++- + 2 files changed, 3 insertions(+), 2 deletions(-) + +commit 08cdf4be9a673d78efe393b53dd73bf43c81dd95 +Author: Sam James +Date: 2024-04-13 21:02:04 +0100 + + ci: add Solaris + + Inspired by https://github.com/RsyncProject/rsync/commit/3f2a38b01184cae9a931280b534acf5a3dae2e94. + + It runs on Solaris 5.11 via a VirtualBox VM. + + .github/workflows/solaris.yml | 31 +++++++++++++++++++++++++++++++ + 1 file changed, 31 insertions(+) + +commit b69768c8bd1a34fde311935c551d061ba52d9a3f +Author: Sam James +Date: 2024-04-14 08:08:00 +0100 + + xz: list: suppress -Wformat-nonliteral for Solaris + + Solaris' GCC can't understand that our use is fine, unlike modern compilers: + ``` + list.c: In function 'print_totals_basic': + list.c:1191:4: error: format not a string literal, argument types not checked [-Werror=format-nonliteral] + uint64_to_str(totals.files, 0)); + ^~~~~~~~~~~~~ + cc1: all warnings being treated as errors + ``` + + It's presumably because of older gettext missing format attributes. + + This is with `gcc (GCC) 7.3.0`. + + src/xz/list.c | 7 +++++++ + 1 file changed, 7 insertions(+) + +commit bb90e1f66d9beb490c4c99763e79519045968710 +Author: Lasse Collin +Date: 2024-06-03 11:44:28 +0300 + + license-check.sh: Fix reporting of unclear license info + + The main feature was broken because an old variable name hadn't + been updated to match the rest of the script. + + build-aux/license-check.sh | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit b8d134e61ede9f4a296226d97f5c20721fb4e8e2 +Author: Lasse Collin +Date: 2024-05-31 21:36:26 +0300 + + Update THANKS + + THANKS | 3 +++ + 1 file changed, 3 insertions(+) + +commit 162587d3fb3fcedc6eee61eda3ccaaf60c80f0de +Author: Lasse Collin +Date: 2024-05-29 17:47:13 +0300 + + Translations: Run po4a/update-po + + Now the files are in the new formatting without source file + line numbers. Future updates should keep the diffs much smaller. + + po4a/de.po | 1592 ++++++++++--------- + po4a/fr.po | 4450 +++++++++++++++++----------------------------------- + po4a/ko.po | 1592 ++++++++++--------- + po4a/pt_BR.po | 4817 ++++++++++++++++++--------------------------------------- + po4a/ro.po | 1592 ++++++++++--------- + po4a/uk.po | 1592 ++++++++++--------- + 6 files changed, 6114 insertions(+), 9521 deletions(-) + +commit 50cd8ed002473c5cd53980e70a53e5e6ad646ffe +Author: Lasse Collin +Date: 2024-05-29 17:44:53 +0300 + + Translations: Run "make -C po update-po" + + In the past this wasn't done before releases; the Git repository + just contained the files from the Translation Project. But this + way it is clearer when comparing release tarballs against the + Git repository. In future releases this might no longer be necessary + within a stable branch as the .po files won't change so easily anymore + when creating a tarball. + + po/ca.po | 567 +++++++++++++++++++++++++--------------- + po/cs.po | 821 +++++++++++++++++++++++++++++++++++++-------------------- + po/da.po | 809 +++++++++++++++++++++++++++++++++++--------------------- + po/de.po | 403 ++++++++++++++-------------- + po/eo.po | 403 ++++++++++++++-------------- + po/es.po | 403 ++++++++++++++-------------- + po/fi.po | 578 +++++++++++++++++++++++++--------------- + po/fr.po | 538 +++++++++++++++++++++++--------------- + po/hr.po | 403 ++++++++++++++-------------- + po/hu.po | 403 ++++++++++++++-------------- + po/it.po | 854 +++++++++++++++++++++++++++++++++++++++--------------------- + po/ko.po | 403 ++++++++++++++-------------- + po/pl.po | 403 ++++++++++++++-------------- + po/pt.po | 842 +++++++++++++++++++++++++++++++++++++++-------------------- + po/pt_BR.po | 567 +++++++++++++++++++++++++--------------- + po/ro.po | 403 ++++++++++++++-------------- + po/sr.po | 838 ++++++++++++++++++++++++++++++++++++++-------------------- + po/sv.po | 403 ++++++++++++++-------------- + po/tr.po | 567 +++++++++++++++++++++++++--------------- + po/uk.po | 403 ++++++++++++++-------------- + po/vi.po | 403 ++++++++++++++-------------- + po/zh_CN.po | 417 +++++++++++++++-------------- + po/zh_TW.po | 558 ++++++++++++++++++++++++--------------- + 23 files changed, 7257 insertions(+), 5132 deletions(-) + +commit 16dbd865c8833462e1604a1e13f7effe55bb3fe6 +Author: Lasse Collin +Date: 2024-05-29 18:03:04 +0300 + + Add NEWS for 5.6.2 + + NEWS | 130 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 130 insertions(+) + +commit a0eeb5f9369c43508610dcf00140edb8e2be92a6 +Author: Lasse Collin +Date: 2024-05-29 18:03:04 +0300 + + Add NEWS for 5.4.7 + + NEWS | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 89 insertions(+) + +commit 9b476fb93a9672f2e70b56e3e9c7e9cfedd6c162 +Author: Lasse Collin +Date: 2024-05-29 18:03:04 +0300 + + Add NEWS for 5.2.13 + + NEWS | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 115 insertions(+) + +commit 9284f1aea31f0eb23e2ea72f7218b271e2234762 +Author: Lasse Collin +Date: 2024-05-29 16:33:24 +0300 + + Build: Update po/*.po files only when needed + + When po/xz.pot doesn't exist, running "make" or "make dist" will + create it. Then the .po files will be updated but only if they + actually would change more than the POT-Creation-Date line. + Then the .gmo files would be generated from the .po files. + This is the case before and after this commit. + + However, "make dist" and thus "make mydist" did a forced update + to the files, updating them even if the only change was the + POT-Creation-Date line. This had pros and cons: It made it clear + that the .po file really is in sync with the recent strings in + the package. On the other hand, it added noise in form of changed + files in the source tree and distribution tarballs. It can be + ignored with something like "diff -I'^"POT-Creation-Date: '" but + it's still a minor annoyance *if* there's not enough value in + having the most recent timestamp. + + Setting DIST_DEPENDS_ON_UPDATE_PO = no means that such forced + update won't happen in "make dist" anymore. However, the "mydist" + target will use xz.pot-update target which is the same target that + is run when xz.pot doesn't exist at all yet. Thus "mydist" will + ensure that the translations are up to date, without noise from + changes that would affect only the POT-Creation-Date line. + + Note that po4a always uses msgmerge with --update, so POT-Creation-Date + in the man page translations is never the only change in .po files. + In that sense this commit makes the message translations behave more + similarly to the man page translations. + + Distribution tarballs will still have non-reproducible POT-Creation-Date + in po/xz.pot and po4a/xz-man.pot but those are just two files. Even they + could be made reproducible from a Git timestamp if desired. + + Makefile.am | 3 ++- + po/Makevars | 6 +++++- + 2 files changed, 7 insertions(+), 2 deletions(-) + +commit 4beba1cd62d7f8f7a6f1e899b68292d94c53b599 +Author: Lasse Collin +Date: 2024-05-28 21:10:33 +0300 + + po4a/update-po: Disable wrapping in .pot and .po files + + The .po files from the Translation Project come with unwrapped + strings so this matches it. + + This may reduce the noise in diffs too. When the beginning of + a paragraph had changed, the rest of the lines got rewrapped + in msgsid. Now it's just one very long line that changes when + a paragraph has been edited. + + The --add-location=file option was removed as redundant. The line + numbers don't exist in the .pot file due to --porefs file and thus + they cannot get copied to the .po files either. + + po4a/update-po | 15 +++++++++++---- + 1 file changed, 11 insertions(+), 4 deletions(-) + +commit b14c130a58a649f9a73392eeb122cb252327c569 +Author: Lasse Collin +Date: 2024-05-28 18:36:53 +0300 + + Update contact info in README + + README | 12 +++++++++--- + 1 file changed, 9 insertions(+), 3 deletions(-) + +commit 75f5f2e014b0ee646963f36bc6a9c840fb272353 +Author: Lasse Collin +Date: 2024-05-28 13:25:07 +0300 + + Translations: Use --package-name=xz-man with po4a + + This is to match reality. See the added comment. + + po4a/update-po | 9 ++++++++- + 1 file changed, 8 insertions(+), 1 deletion(-) + +commit eb217d016cfbbba1babc19a61095b3ea25898af6 +Author: Lasse Collin +Date: 2024-05-28 13:03:40 +0300 + + Translations: Omit --package-name from po/Makevars + + This is closer to the reality in the po/*.po files. + + po/Makevars | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + +commit d28a4b2520adeeaa1b9e921bf42c7c1f36552c06 +Author: Lasse Collin +Date: 2024-05-27 17:45:51 +0300 + + license-check.sh: Use '--' with slightly untrusted filenames + + Names from git ls-files should be safe but if one runs it on + a tree without the .git dir and there are extra files, it's + safer to have the end of arguments marked with '--'. + + build-aux/license-check.sh | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit fda0ec862a34094cf23fc25d0e0a95858c3a3ab5 +Author: Lasse Collin +Date: 2024-05-27 17:41:37 +0300 + + license-check.sh: Use xargs -0 instead of -d + + Neither are in POSIX but -0 is much more portable in practice. + + Despite the old comment, the grep usage should be portable already. + + build-aux/license-check.sh | 11 ++++++----- + 1 file changed, 6 insertions(+), 5 deletions(-) + +commit 9114267038deaecf4832a5cacb5acbe6591ac839 +Author: Lasse Collin +Date: 2024-05-28 01:17:45 +0300 + + Translations: Omit man page line numbers from .pot and .po files + + po4a/update-po | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 093490b58271e9424ce38a7b1b38bcf61b9c86c6 +Author: Lasse Collin +Date: 2024-05-28 01:06:30 +0300 + + Translations: Use the xgettext option --add-location=file + + po/Makevars | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit fccebe2b4fd513488fc920e4dac32562ed3c7637 +Author: Lasse Collin +Date: 2024-05-28 00:43:53 +0300 + + Translations: Use the msgmerge option --add-location=file + + This way the PO file diffs are less noisy but the locations of the + strings are still present at file level, just without line numbers. + + The option is available since gettext 0.19 (2014). + configure.ac requires 0.19.6. + + po/Makevars | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f361d9ae85707a87eb28db400eb7229cec103d58 +Author: Lasse Collin +Date: 2024-05-27 12:22:08 +0300 + + Build: Use $(SHELL) instead of sh to run scripts in Makefile.am + + Makefile.am | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +commit a26dece34793a09aac2476f954d162d03e9cf62b +Author: Lasse Collin +Date: 2024-05-23 17:25:13 +0300 + + Translations: Change the home page URLs in man page translations + + Since the source strings have changed, these would get marked as + fuzzy and the original string would be used instead. The original + and translated strings are identical in this case so it wouldn't + matter. But patching the translations helps still because then + po4a will show the correct translation percentage. + + po4a/de.po | 8 ++++---- + po4a/fr.po | 4 ++-- + po4a/ko.po | 4 ++-- + po4a/pt_BR.po | 4 ++-- + po4a/ro.po | 8 ++++---- + po4a/uk.po | 8 ++++---- + 6 files changed, 18 insertions(+), 18 deletions(-) + +commit 24387c234b4eed1ef9a7eaa107391740b4095568 +Author: Lasse Collin +Date: 2024-05-23 15:15:18 +0300 + + CMake: Add manual support for 32-bit x86 assembly files + + One has to pass -DENABLE_X86_ASM=ON to cmake to enable the + CRC assembly code. Autodetection isn't done. Looking at + CMAKE_SYSTEM_PROCESSOR might not work as it comes from uname + unless cross-compilation is done using a CMake toolchain file. + + On top of this, if the code is run on modern processors that support + the CLMUL instruction, then the C code should be faster (but then + one should also be using a x86-64 build if possible). + + CMakeLists.txt | 34 +++++++++++++++++++++++++++++++--- + 1 file changed, 31 insertions(+), 3 deletions(-) + +commit 0fb3c9c3f684f5a25bd425ed079a20a79f0c969d +Author: Lasse Collin +Date: 2024-05-23 14:26:45 +0300 + + CMake: Rename USE_DOXYGEN to ENABLE_DOXYGEN + + It's more consistent with the other option() uses. + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 6bbec3bda02bf87d24fa095074456e723589921f +Author: Lasse Collin +Date: 2024-05-22 15:21:53 +0300 + + Mention license-check.sh in COPYING + + COPYING | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 62733592a1cc6f0b41f46ef52e06d1a6fe1ff38a +Author: Lasse Collin +Date: 2024-05-22 15:21:53 +0300 + + Use more confident language in COPYING + + COPYING | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +commit a119a4209e8827e1d7c2cfd30cb9f5a9b76f9dff +Author: Lasse Collin +Date: 2024-05-22 15:21:53 +0300 + + Build: Run license-check.sh in "mydist" and "dist-hook" + + In mydist the point is to check using the file list from the Git + repository. In dist-hook it is to check that the TARBALL_IGNORE + patterns work when the .git dir or the "git" command aren't available. + + Refuse to create a distribution tarball if license issues are found. + + Makefile.am | 2 ++ + 1 file changed, 2 insertions(+) + +commit f3434ecfcb45154508752986f4fc670b8f0555dc +Author: Lasse Collin +Date: 2024-05-22 15:21:53 +0300 + + Add build-aux/license-check.sh + + This helps in spotting files that lack SPDX license identifier + and which haven't been explicitly white listed either. The script + requires the .git directory to be present as only the files that + are in the Git repository are checked. + + XZ Utils isn't FSFE REUSE compliant for now. + + Makefile.am | 1 + + build-aux/license-check.sh | 174 +++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 175 insertions(+) + +commit 9ae2ebc1e504a1814b0788de95fb5c58c0328dde +Author: Lasse Collin +Date: 2024-04-29 17:16:38 +0300 + + Add SPDX license identifiers to files under tests/ossfuzz + + tests/ossfuzz/Makefile | 2 ++ + tests/ossfuzz/config/fuzz_decode_alone.options | 2 ++ + tests/ossfuzz/config/fuzz_decode_stream.options | 2 ++ + tests/ossfuzz/config/fuzz_encode_stream.options | 2 ++ + tests/ossfuzz/config/fuzz_lzma.dict | 2 ++ + tests/ossfuzz/config/fuzz_xz.dict | 2 ++ + 6 files changed, 12 insertions(+) + +commit 9000d70eb9815bd7f43ffddc1c3316c507aa0e05 +Author: Lasse Collin +Date: 2024-04-29 17:16:06 +0300 + + Add SPDX license identifier to .codespellrc + + .codespellrc | 2 ++ + 1 file changed, 2 insertions(+) + +commit 903c16fcfa5bfad0cdb2a7383d941243bcb12e76 +Author: Lasse Collin +Date: 2024-05-22 15:12:09 +0300 + + Move entries po4a/.gitignore to the top level .gitignore + + The po4a directory is in EXTRA_DIST and thus all files there + are included in the package. .gitignore doesn't belong in the + package so keep that file out of the po4a directory. + + .gitignore | 4 ++++ + po4a/.gitignore | 3 --- + 2 files changed, 4 insertions(+), 3 deletions(-) + +commit 56f1d5ed68e84ba5dfa328ea2291b8f46c995125 +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + Tests: Make the config.h grep patterns Meson compatible + + Now the test scripts detect both + + #define HAVE_DECODER_ARM + #define HAVE_DECODER_ARM 1 + + as support for the ARM filter without confusing it with these: + + #define HAVE_DECODER_ARM64 + #define HAVE_DECODER_ARM64 1 + + Previously only the ones ending with " 1" were accepted for + the macros where this kind of confusion was possible. + + This should help with Meson support because Meson's built-in + features produce config.h entries that are either + + #define FOO 1 + #define FOO 0 + + or: + + #define FOO + #undef FOO + + The former method has a benefit that one can use "#if FOO" and -Wundef + will catch if a #define is missing (for example, it helps catching + typos). But XZ Utils has to use the latter since it has been + convenient with Autoconf's default behavior.[*] While it's easy to + emulate the Autoconf style (#define FOO 1 vs. no #define at all) + in Meson, it results in clumsy code. Thus it's better to change + the few places in the tests where this difference matters. + + [*] While most checks in Autoconf default to the second style above, + a few things use the first style (like AC_CHECK_DECLS). The mix + of both styles is the most confusing as one has to remember which + macro needs #ifdef and which #if. Currently HAVE_VISIBILITY is + only such config.h entry that is 1 or 0. It comes unmodified + from Gnulib's visibility.m4. + + tests/test_compress.sh | 4 ++-- + tests/test_files.sh | 2 +- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 9d997d6f9d4f042412e45c7b7a23a14ad2e4f9aa +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + CMake: Add comments + + tests/tests.cmake | 2 ++ + 1 file changed, 2 insertions(+) + +commit d35368b33e54bad2f566df99fac29ffea38e34de +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + CMake: Remove the note that some tests aren't run + + They are now in the common build configurations. + + CMakeLists.txt | 2 -- + 1 file changed, 2 deletions(-) + +commit dc232d584619b2819a9c52d6ad5d8b5d56b392ba +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + CMake: Add support for test_files.sh + + tests/tests.cmake | 20 ++++++++++++++++++++ + 1 file changed, 20 insertions(+) + +commit a7e9230af9d1f87f474fe38886eb977d4149dc9b +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + Tests: Make test_files.sh more flexible + + Add a new optional argument to specify the directory of the xz and + xzdec executables. + + If ../config.h doesn't exist, assume that all encoders and decoders + are available. + + tests/test_files.sh | 18 +++++++++++++----- + 1 file changed, 13 insertions(+), 5 deletions(-) + +commit b40e6efbb48d740b9b5b303e59e344801cbb5bd8 +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + CMake: Add support for test_compress.sh tests + + tests/tests.cmake | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +commit ac3222d2cb1ff3a15eb6d58f9ea9bc78e8bc3bb2 +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + Tests: Make test_compress.sh more flexible + + Add a new optional second argument: directory of the xz and xzdec + executables. This is need with the CMake build where the binaries + end up in the top-level build directory. + + If ../config.h doesn't exist, assume that all encoders and decoders + are available. This will make this script usable from CMake in the + most common build configuration. + + NOTE: Since the existence of ../config.h is checked, the working + directory of the test script must be a subdir in the build tree! + Otherwise ../config.h would look outside the build tree. + + Use the default check type instead of forcing CRC32 or CRC64. + Now the script doesn't need to check if CRC64 is available. + + tests/test_compress.sh | 41 +++++++++++++++++++++++++++++------------ + 1 file changed, 29 insertions(+), 12 deletions(-) + +commit 006040b29c83104403621e950ada0c8956c56b3d +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + CMake: Prepare to support the test_*.sh tests + + This is a bit hacky since the scripts grep config.h to know which + features were built but the CMake build doesn't create config.h. + So instead those test scripts will be run only when all relevant + features have been enabled. + + tests/tests.cmake | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 49 insertions(+) + +commit 6167607a6ea72fb74eefb943c4566e3cab528cd2 +Author: Lasse Collin +Date: 2024-05-20 16:55:00 +0300 + + Tests: test_suffix.sh: Add a comment + + tests/test_suffix.sh | 3 +++ + 1 file changed, 3 insertions(+) + +commit 4e9023857d287f624562156b60dc23d2b64c0f10 +Author: Lasse Collin +Date: 2024-05-18 00:34:07 +0300 + + Fix typos + + Thanks to xx on #tukaani. + + src/common/mythread.h | 2 +- + src/common/tuklib_integer.h | 2 +- + src/liblzma/api/lzma/base.h | 2 +- + src/liblzma/common/filter_buffer_decoder.c | 2 +- + src/liblzma/common/filter_common.c | 2 +- + src/scripts/xzgrep.in | 2 +- + 6 files changed, 6 insertions(+), 6 deletions(-) + +commit b14d08fbbc254485ace9ccfe7908674f608a62ae +Author: Lasse Collin +Date: 2024-05-18 00:23:52 +0300 + + liblzma: Fix white space + + Thanks to xx on #tukaani. + + src/liblzma/simple/simple_coder.h | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +commit 9f1a6d6f9a258886933a22239a5b81af34b28199 +Author: Lasse Collin +Date: 2024-05-15 23:14:17 +0300 + + Build: Temporarily disable CRC CLMUL to silence OSS Fuzz + + The code makes aligned 16-byte reads which may read up to 15 bytes + before the beginning or past the end of the buffer if the buffer + is misaligned. The unneeded bytes are then ignored. It cannot cross + page boundaries and thus cannot cause access violations. + + This inherently trips address sanitizer which was already disabled + with __attribute__((__no_sanitize_address__)). However, it also + trips memory sanitizer if the extra bytes are uninitialized because + memory sanitizer doesn't see that those bytes then get ignored by + byte shuffling in the xmm registers. + + The plan is to change the code so that all sanitizers pass but it's + not finished yet (performance shouldn't get worse) so as a temporary + measure to keep OSS Fuzz happy, the CLMUL CRC is now disabled even + though I think think the code is fine to use (and easy enough to review + the memory accesses in it too). + + configure.ac | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 142e670a413a7bce1a2647f1cf1f33f8ee2dbe88 +Author: Lasse Collin +Date: 2024-05-13 17:15:04 +0300 + + xz: Document the static function get_chains_memusage() + + src/xz/coder.c | 14 ++++++++++++-- + 1 file changed, 12 insertions(+), 2 deletions(-) + +commit 78e984399a64bfee5d11e7308e0bdbc1006db2ca +Author: Lasse Collin +Date: 2024-05-13 17:07:22 +0300 + + xz: Rename filters_memusage_max() to get_chains_memusage() + + src/xz/coder.c | 14 ++++++-------- + 1 file changed, 6 insertions(+), 8 deletions(-) + +commit 54c3db0a83d3e67d89aba92a0957f2dce9b111a7 +Author: Lasse Collin +Date: 2024-05-13 17:04:05 +0300 + + xz: Rename filter_memusages to chains_memusages + + src/xz/coder.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit d9e1ae79ec90d6a7eafeaceaf0ece4f0c83d4417 +Author: Lasse Collin +Date: 2024-05-12 22:26:30 +0300 + + xz: Simplify the memory usage scaling code + + This is closer to what it was before the --filtersX support was added, + just extended to support for scaling all filter chains. The method + before this commit was an extended version of the original too but + it was done in a more complex way for no clear reason. In case of + an error, the complex version printed fewer informative messages + (a good thing) but it's not a sigificant benefit. + + In the limit is too low even for single-threaded mode, the required + amount of memory is now reported like in 5.4.x instead of like in + 5.5.1alpha - 5.6.1 which showed the original non-scaled usage. It + had been a FIXME in the old code but it's not clear what message + makes the most sense. + + Fixes: 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a + + src/xz/coder.c | 163 ++++++++++++++++++++------------------------------------- + 1 file changed, 57 insertions(+), 106 deletions(-) + +commit 0ee56983d198b776878432703de664049b1be32e +Author: Lasse Collin +Date: 2024-05-13 12:14:00 +0300 + + xz: Edit comments + + src/xz/coder.h | 6 ++---- + 1 file changed, 2 insertions(+), 4 deletions(-) + +commit ec82a49c3553f7206104582dbfb8b64fa433b491 +Author: Lasse Collin +Date: 2024-05-13 12:03:51 +0300 + + xz: Rename chain_idx to chain_num + + src/xz/coder.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit a731a6993c34bbbd55abaf9c166718682b1da24f +Author: Lasse Collin +Date: 2024-05-12 22:29:11 +0300 + + xz: Edit coding style + + src/xz/coder.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 32eb176b89243fce3112347fe43a8ad14a9fd2be +Author: Lasse Collin +Date: 2024-05-12 22:16:05 +0300 + + xz: Edit comments + + Fixes: 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a + + src/xz/coder.c | 8 ++------ + 1 file changed, 2 insertions(+), 6 deletions(-) + +commit b90339f4daa510d2b1b8c550f855a99667f1d004 +Author: Lasse Collin +Date: 2024-05-12 21:57:49 +0300 + + xz: Fix grammar in a comment + + Fixes: cb3111e3ed84152912b5138d690c8d9f00c6ef02 + + src/xz/coder.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 4c0bdaf13d651b22ba13bd93f8379724d6ccdc13 +Author: Lasse Collin +Date: 2024-05-12 21:46:56 +0300 + + xz: Rename filter_memusages to encoder_memusages + + src/xz/coder.c | 12 ++++++------ + 1 file changed, 6 insertions(+), 6 deletions(-) + +commit b54aa023e0ec291b06e976e5f094ab0549e7b09b +Author: Lasse Collin +Date: 2024-05-12 21:42:05 +0300 + + xz: Edit coding style + + src/xz/coder.c | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +commit 49f67d3d3f42b640a7dfc4ca04c8934f658e10ce +Author: Lasse Collin +Date: 2024-05-12 21:31:02 +0300 + + xz: Rename filters_index to chain_num + + The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68. + + src/xz/args.c | 8 ++++---- + src/xz/coder.c | 8 ++++---- + src/xz/coder.h | 2 +- + 3 files changed, 9 insertions(+), 9 deletions(-) + +commit ff9e8b3d069ecfa52ec43dcdb198542d1692a492 +Author: Lasse Collin +Date: 2024-05-12 21:22:43 +0300 + + xz: Replace a few uint32_t with "unsigned" to reduce the number of casts + + These hold only tiny values. + + src/xz/args.c | 2 +- + src/xz/coder.c | 17 ++++++++--------- + src/xz/coder.h | 2 +- + 3 files changed, 10 insertions(+), 11 deletions(-) + +commit b5e6c1113b1ba02c282bd9163eccdb521c937a78 +Author: Lasse Collin +Date: 2024-05-12 21:10:45 +0300 + + xz: Rename filters_used_mask to chains_used_mask + + The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68. + + src/xz/coder.c | 30 +++++++++++++++--------------- + 1 file changed, 15 insertions(+), 15 deletions(-) + +commit 32500dfaadae2ea36fda2e17b49ae7d9ac1acf52 +Author: Lasse Collin +Date: 2024-05-12 17:14:43 +0300 + + xz: Move the setting of "check" in coder_set_compression_settings() + + It's more logical to do it in the beginning instead of in the middle + of the filter chain handling. + + Fixes: d6af7f347077b22403133239592e478931307759 + + src/xz/coder.c | 18 +++++++++--------- + 1 file changed, 9 insertions(+), 9 deletions(-) + +commit ad146b1f42bbb678175a503a45ce525e779f9b8b +Author: Lasse Collin +Date: 2024-05-12 17:09:17 +0300 + + xz: Rename "filters" to "chains" + + The convention is that + + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + contains the filters of a single filter chain. + It was so here as well before the commit + d6af7f347077b22403133239592e478931307759. + It changes "filters" to a ten-element array of filter chains. + It's clearer to call this array-of-arrays "chains". + + This also renames "filter_idx" to "chain_idx" which is used + as an index as in chains[chain_idx]. + + src/xz/coder.c | 68 +++++++++++++++++++++++++++++----------------------------- + 1 file changed, 34 insertions(+), 34 deletions(-) + +commit 5a4ae4e4d0105404184e9a82ee08f94e1b7783e0 +Author: Lasse Collin +Date: 2024-05-12 16:56:15 +0300 + + xz: Clean up a comment + + src/xz/coder.c | 9 +++------ + 1 file changed, 3 insertions(+), 6 deletions(-) + +commit 2de80494ed9a4dc7db395a32a5efb770ce769804 +Author: Lasse Collin +Date: 2024-05-12 16:52:09 +0300 + + xz: Add clarifying assertions + + src/xz/coder.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 1eaad004bf7748976324672db028e34f42802e61 +Author: Lasse Collin +Date: 2024-05-10 20:23:33 +0300 + + xz: Add a clarifying assertion + + Fixes: 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a + + src/xz/coder.c | 1 + + 1 file changed, 1 insertion(+) + +commit 605094329b986244833c967c04963cacc41a868d +Author: Lasse Collin +Date: 2024-05-12 16:47:17 +0300 + + xz: Clarify a comment + + src/xz/coder.c | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 8fac2577f2dbb9491afd8500f60d004c9071df3b +Author: Lasse Collin +Date: 2024-05-12 16:28:25 +0300 + + xz: Use the info collected in parse_block_list() + + This is slightly simpler and it avoids looping through + the opt_block_list array. + + src/xz/coder.c | 95 ++++++++++++++++++++++++---------------------------------- + 1 file changed, 39 insertions(+), 56 deletions(-) + +commit 81d350dab864b985b740742772f3b132d4c52914 +Author: Lasse Collin +Date: 2024-05-12 15:48:45 +0300 + + xz: Remember the filter chains and the largest Block in parse_block_list() + + src/xz/args.c | 18 ++++++++++++++++++ + src/xz/coder.c | 2 ++ + src/xz/coder.h | 13 +++++++++++++ + 3 files changed, 33 insertions(+) + +commit 46ab56968f7dfdac187710a1223659d832fa1565 +Author: Lasse Collin +Date: 2024-05-12 15:38:48 +0300 + + xz: Update a comment and initialization of filters_used_mask + + src/xz/coder.c | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit e89293a0baeb8663707c6b4a74fbb310ec698a8f +Author: Lasse Collin +Date: 2024-05-12 15:08:10 +0300 + + xz: parse_block_list: Edit integer type casting + + src/xz/args.c | 5 ++--- + 1 file changed, 2 insertions(+), 3 deletions(-) + +commit 87011e40c168255cd2edea129ee68c901770603b +Author: Lasse Collin +Date: 2024-05-12 14:51:37 +0300 + + xz: Make filter_memusages a local variable + + src/xz/coder.c | 35 +++++++++++++++++++++-------------- + 1 file changed, 21 insertions(+), 14 deletions(-) + +commit 347b412a9374e0456bef9da0d7d79174c0b6f1a5 +Author: Lasse Collin +Date: 2024-05-10 20:33:08 +0300 + + xz: Remove unused code and simplify + + opt_mode == MODE_COMPRESS isn't possible when HAVE_ENCODERS isn't + defined. Thus, when *encoding*, the message about *decoder* memory + usage is possible to show only when both encoder and decoder have + been built. + + Since the message is shown only at V_DEBUG, skip the memusage + calculation if verbosity level isn't high enough. + + Fixes: 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a + + src/xz/coder.c | 16 ++++------------ + 1 file changed, 4 insertions(+), 12 deletions(-) + +commit 31358c057c9de9d6aba96bae112b2d17942de7cb +Author: Lasse Collin +Date: 2024-05-10 20:22:58 +0300 + + xz: Fix integer type from uint64_t to uint32_t + + lzma_options_lzma.dict_size is uint32_t so use it here too. + + Fixes: 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a + + src/xz/coder.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 3f71e0f3a118e1012526f94fd640a626d30cb599 +Author: Lasse Collin +Date: 2024-05-08 21:40:07 +0300 + + debug/translation.bash: Remove an outdated test command + + Since 5.3.5beta, "xz --lzma2=mf=bt4,nice=2" works even though bt4 needs + at least nice=4. It is rounded up internally by liblzma when needed. + + Fixes: 5cd9f0df78cc4f8a7807bf6104adea13034fbb45 + + debug/translation.bash | 1 - + 1 file changed, 1 deletion(-) + +commit b05a516830095a0e1937aeb31c937fb0400408b6 +Author: Lasse Collin +Date: 2024-05-07 20:41:28 +0300 + + Fix the date of NEWS for 5.4.5 + + NEWS | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 6d336aeb97b69c496ddc626af403f6f21c753658 +Author: Lasse Collin +Date: 2024-05-07 16:21:15 +0300 + + Build: Update visibility.m4 from Gnulib + + This fixes the syntax of the "serial" line and renames + a temporary variable. + + m4/visibility.m4 | 13 +++++++------ + 1 file changed, 7 insertions(+), 6 deletions(-) + +commit ab51e8ee610e2a893906859848f93d5cb0d5ba83 +Author: Lasse Collin +Date: 2024-05-07 15:05:21 +0300 + + po4a/update-po: Delete the *.po.authors files + + These are temporary files that are needed only when running po4a. + The top-level Makefile.am puts the whole po4a directory into + distribution tarball (it's simpler) so deleting these temporary + files is needed to prevent them from getting into tarballs. + + po4a/update-po | 4 ++++ + 1 file changed, 4 insertions(+) + +commit e4780244a17420cc95d5498cd6e02ad10eac6e5f +Author: Lasse Collin +Date: 2024-05-07 13:12:17 +0300 + + xz: Edit comments and coding style + + src/xz/coder.c | 25 ++++++++++++------------- + 1 file changed, 12 insertions(+), 13 deletions(-) + +commit fe4d8b0c80eaeca3381be302eeb89aba871a7e7c +Author: Lasse Collin +Date: 2024-05-06 23:08:22 +0300 + + xz: Omit an incorrect comment + + It likely was a leftover from a development version of the code. + + Fixes: 183819bfd9efac8c184d9bf123325719b7eee30f + + src/xz/coder.c | 6 +----- + 1 file changed, 1 insertion(+), 5 deletions(-) + +commit 9bef5b8d17dd5e009d6a6b2becc2dc535da53937 +Author: Lasse Collin +Date: 2024-05-06 23:04:31 +0300 + + xz: Add braces to a for-statement and to an if-statement + + No functional changes. + + Fixes: 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a + Fixes: 479fd58d60622331fcbe48fddf756927b9f80d9a + + src/xz/coder.c | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit de06b9f0c0a3f72569829ecadbc9c0a3ef099f57 +Author: Lasse Collin +Date: 2024-05-06 23:00:09 +0300 + + liblzma: Omit an unneeded array from the x86 filter + + Fixes: 6aa2a6deeba04808a0fe4461396e7fb70277f3d4 + + src/liblzma/simple/x86.c | 5 +---- + 1 file changed, 1 insertion(+), 4 deletions(-) + +commit 7da488cb933fdf51cfc14cb5810beb0766224380 +Author: Lasse Collin +Date: 2024-05-06 22:56:31 +0300 + + CMake: Add test_suffix.sh to the tests + + tests/tests.cmake | 13 +++++++++++++ + 1 file changed, 13 insertions(+) + +commit a805594ed0b4cbf7b81aa28ff46a8ab3c83c6876 +Author: Lasse Collin +Date: 2024-05-06 22:55:54 +0300 + + Test: Add CMake support to test_suffix.sh + + It needs to find the xz executable from a different directory + and work without config.h. + + tests/test_suffix.sh | 12 +++++++----- + 1 file changed, 7 insertions(+), 5 deletions(-) + +commit 50e19489387774bab3c4a988397d0d9c7a142a46 +Author: Lasse Collin +Date: 2024-05-06 20:45:34 +0300 + + Update INSTALL about MINIX 3 + + The latest stable is 3.3.0 and it's from 2014. + Don't mention the older versions in INSTALL. + 3.3.0 ships with Clang already. + + Testing with 3.4.0beta6 shows that tuklib_physmem + works too so omit comments about that from INSTALL. + Visibility warnigns weren't a problem either. + + Thus it's enough to mention the need for --disable-threads + as configure doesn't autodetect the lack of pthreads. + + INSTALL | 20 +++++++------------- + 1 file changed, 7 insertions(+), 13 deletions(-) + +commit 68d18aea1422a2b86b98b71d0b019233d84e01b0 +Author: Lasse Collin +Date: 2024-05-02 23:00:16 +0300 + + Windows: Remove the "doc/api" line from README-Windows.txt + + Fixes: 252aa1d67bc015eeba462803ab72edeb7744d864 + + windows/README-Windows.txt | 2 -- + 1 file changed, 2 deletions(-) + +commit 8ede961374613aa302a13571d662cfaea1cf91f7 +Author: Lasse Collin +Date: 2024-05-02 22:59:04 +0300 + + Build: Don't copy doc/api from source tree to distribution tarball + + It was copied if it existed. This was intentional when autogen.sh + still built liblzma API docs with Doxygen. + + Fixes: d3a77ebc04bf1db8d52de2d9b0f07877bc4fd139 + + Makefile.am | 5 ----- + 1 file changed, 5 deletions(-) + +commit 9a6761aa35ed84d30bd2fda2333a4fdf3f46ecdc +Author: Sam James +Date: 2024-05-02 13:26:40 +0100 + + ci: add SPDX headers + + I've checked over each of these and they're straightforward applications + of the relevant Github Actions. + + .github/workflows/freebsd.yml | 2 ++ + .github/workflows/netbsd.yml | 2 ++ + .github/workflows/openbsd.yml | 2 ++ + 3 files changed, 6 insertions(+) + +commit 81efe6119f86e3274e512c9eca5ec22b2196c2b3 +Author: Yaroslav Halchenko +Date: 2024-03-29 14:37:24 -0400 + + codespell: Ignore the THANKS file and debbugs.gnu.org URL + + This way "codespell -i 0" is silent. + + This is the first commit from + https://github.com/tukaani-project/xz/pull/93 + with trivial edits by Lasse Collin. + + .codespellrc | 6 +++++- + 1 file changed, 5 insertions(+), 1 deletion(-) + +commit 905bfc74fe2670fd9c39014803017ab53d325401 +Author: Lasse Collin +Date: 2024-04-30 14:37:11 +0300 + + Add .gitattributes to clean up git-archive output + + .gitattributes | 7 +++++++ + 1 file changed, 7 insertions(+) + +commit 3334c71d3d4294a4f6569df3ba9bcf2443dfa501 +Author: Lasse Collin +Date: 2024-04-19 12:11:09 +0300 + + xzdec: Support Landlock ABI version 4 + + This was added to xz in 02e3505991233901575b7eabc06b2c6c62a96899 + but I forgot to do the same in xzdec. + + The Landlock sandbox in xzdec could be stricter as now it's + active only for the last file being decompressed. In xz, + read-only sandbox is used for multi-file case. On the other hand, + xz doesn't go to the strictest mode when processing the last file + when more than one file was specified; xzdec does. + + src/xzdec/xzdec.c | 18 ++++++++++++++---- + 1 file changed, 14 insertions(+), 4 deletions(-) + +commit 278563ef8f2b8d98d7f2c85e1a64ec1bc21d26d8 +Author: Lasse Collin +Date: 2024-04-30 22:22:45 +0300 + + liblzma: Fix incorrect function type error from sanitizer + + Clang 17 with -fsanitize=address,undefined: + + src/liblzma/common/filter_common.c:366:8: runtime error: + call to function encoder_find through pointer to incorrect + function type 'const lzma_filter_coder *(*)(unsigned long)' + src/liblzma/common/filter_encoder.c:187: note: + encoder_find defined here + + Use a wrapper function to get the correct type neatly. + This reduces the number of casts needed too. + + This issue could be a problem with control flow integrity (CFI) + methods that check the function type on indirect function calls. + + Fixes: 3b34851de1eaf358cf9268922fa0eeed8278d680 + + src/liblzma/common/filter_decoder.c | 15 ++++++++++++--- + src/liblzma/common/filter_encoder.c | 17 +++++++++++++---- + 2 files changed, 25 insertions(+), 7 deletions(-) + +commit 77c8f60547decefca8f2d0c905d9c708c38ee8ff +Author: Lasse Collin +Date: 2024-04-30 21:41:11 +0300 + + xz: Avoid arithmetic on a null pointer + + It's undefined behavior. The result wasn't ever used as it occurred + in the last iteration of a loop. + + Clang 17 with -fsanitize=address,undefined: + + $ src/xz/xz --block-list=123 + src/xz/args.c:164:12: runtime error: applying non-zero offset 1 + to null pointer + + Fixes: 88ccf47205d7f3aa314d358c72ef214f10f68b43 + Co-authored-by: Sam James + + src/xz/args.c | 8 +++++++- + 1 file changed, 7 insertions(+), 1 deletion(-) + +commit 64503cc2b76a388ced4ec5f68234a07f0dcddcd5 +Author: Lasse Collin +Date: 2024-04-27 20:42:00 +0300 + + CMake: Support building liblzma API docs using Doxygen + + This is disabled by default to match the default in Autotools. + Use -DUSE_DOXYGEN=ON to enable Doxygen usage. + + This uses the update-doxygen script, thus this is under if(UNIX) + although Doxygen itself can run on Windows too. + + CMakeLists.txt | 40 +++++++++++++++++++++++++++++++--------- + 1 file changed, 31 insertions(+), 9 deletions(-) + +commit 0a7f5a80d8532a1d8cfa0a902c9d1ad7651eca37 +Author: Lasse Collin +Date: 2024-04-20 23:36:39 +0300 + + CMake: List API headers in LIBLZMA_API_HEADERS variable + + This way the same list will be usable in more than one location. + + CMakeLists.txt | 21 ++++++++++++--------- + 1 file changed, 12 insertions(+), 9 deletions(-) + +commit 541406bee3f09e9813103c6406b10fc6ab2e0d30 +Author: Lasse Collin +Date: 2024-04-19 15:16:42 +0300 + + PACKAGERS: Document the optional Doxygen usage + + Also add a note that packagers should check the licensing + of the Doxygen output. + + PACKAGERS | 19 ++++++++++--------- + 1 file changed, 10 insertions(+), 9 deletions(-) + +commit e21efdf96f39378fe417479f89e97046680406f5 +Author: Lasse Collin +Date: 2024-04-27 17:47:09 +0300 + + Build: Add --enable-doxygen to generate and install API docs + + It requires Doxygen. This option is disabled by default. + + INSTALL | 6 ++++++ + configure.ac | 10 +++++++++- + src/liblzma/api/Makefile.am | 19 +++++++++++++++++++ + 3 files changed, 34 insertions(+), 1 deletion(-) + +commit 0ece09a575d7e542bda8825808ddd6cf7de8cc4b +Author: Lasse Collin +Date: 2024-04-19 15:15:17 +0300 + + Doxygen: update-doxygen: Support out-of-tree builds + + Also, now $0 is used to refer to the script itself. + + doxygen/update-doxygen | 110 ++++++++++++++++++++++++++++++------------------- + 1 file changed, 68 insertions(+), 42 deletions(-) + +commit 2c519f641f266fd897edf680827d9c905f411440 +Author: Lasse Collin +Date: 2024-04-28 21:08:00 +0300 + + Doxygen: Simplify Doxyfile and add SPDX license identifier + + This omits all comments and a few non-default options that weren't + needed. Now it contains no copyrighted content from Doxygen itself. + + doxygen/Doxyfile | 2698 +----------------------------------------------------- + 1 file changed, 25 insertions(+), 2673 deletions(-) + +commit bdba39a57530d11b88440df8024002be3d09e4a1 +Author: Lasse Collin +Date: 2024-04-19 15:14:02 +0300 + + Doxygen: Don't strip JavaScript anymore + + The stripping method worked well with Doxygen 1.8 and 1.9 but + it doesn't work with Doxygen 1.10 anymore. Since we won't ship + pre-generated liblzma API docs anymore, the extra bloat and + extra license info of the JavaScript files won't affect the + upstream source package anymore. + + doxygen/update-doxygen | 21 --------------------- + 1 file changed, 21 deletions(-) + +commit d3a77ebc04bf1db8d52de2d9b0f07877bc4fd139 +Author: Lasse Collin +Date: 2024-04-19 17:26:41 +0300 + + Build: Remove old Doxygen rules from top-level Makefile.am + + Makefile.am | 12 ------------ + 1 file changed, 12 deletions(-) + +commit fd7faa4c338a42a6a40e854b837d285ae2e8c609 +Author: Lasse Collin +Date: 2024-04-19 15:10:06 +0300 + + Update COPYING to match the autogen.sh and mydist changes + + COPYING | 11 ----------- + 1 file changed, 11 deletions(-) + +commit b2bc55d8a0a9f2f59bfd4302067300e650f6baa3 +Author: Lasse Collin +Date: 2024-04-19 17:23:43 +0300 + + Build: Don't run update-doxygen as part of "make mydist" + + Makefile.am | 1 - + 1 file changed, 1 deletion(-) + +commit e9be74f5b129fe8a5388d588e68b1b7f5168a310 +Author: Lasse Collin +Date: 2024-04-19 15:09:48 +0300 + + autogen.sh: Don't generated Doxygen docs anymore + + autogen.sh | 18 +++--------------- + 1 file changed, 3 insertions(+), 15 deletions(-) + +commit 252aa1d67bc015eeba462803ab72edeb7744d864 +Author: Lasse Collin +Date: 2024-04-19 17:41:36 +0300 + + windows/build.bash: Omit Doxygen docs from the package + + They will be omitted from the source tarball and I don't want + to make Doxygen a dependency of build.bash. + + windows/build.bash | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 634095364d87444d62d8ec54c134c0cd4705f5d7 +Author: Lasse Collin +Date: 2024-04-19 14:14:47 +0300 + + README: Don't mention PDF man pages anymore + + README | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit dc684bf76ea23574ee9d88382057381e04e6089a +Author: Lasse Collin +Date: 2024-04-19 14:10:39 +0300 + + Build: Omit PDF man pages from the package + + pdf-local rule was added to create the PDFs still with "make pdf". + The install rules are missing but that likely doesn't matter at all. + + Makefile.am | 29 +++++++++++++++++++---------- + 1 file changed, 19 insertions(+), 10 deletions(-) + +commit e3531ab4125cbd5c01ebd3200791350960547189 +Author: Lasse Collin +Date: 2024-04-19 13:54:39 +0300 + + windows/build.bash: Don't copy PDF man pages to the package + + windows/README-Windows.txt | 2 +- + windows/build.bash | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit 710a4573ef2cbd19c66318c3b2d1388e418e26c7 +Author: Lasse Collin +Date: 2024-04-28 01:34:50 +0300 + + Tests: test_index: Fix failures when features are disabled + + Fixes: cd88423e76d54eb72aea037364f3ebb21f122503 + + tests/test_index.c | 13 ++++++++----- + 1 file changed, 8 insertions(+), 5 deletions(-) + +commit aaff75c3486c4489ce88b0efb36b41cf138af7c3 +Author: Lasse Collin +Date: 2024-04-20 17:09:11 +0300 + + CMake: Keep the build working if the "tests" directory is missing + + This moves the tests section as is from CMakeLists.txt into + tests/tests.cmake. CMakeLists.txt now includes tests/tests.cmake + if the latter file exists. + + Now it's possible to delete the whole "tests" directory and + building with CMake will still work normally, just without + the tests. This way the tests are readily available for those + who want them, and those who won't run the tests anyway have + a straightforward way to ensure that nothing from the "tests" + directory can affect the build process. + + CMakeLists.txt | 76 ++--------------------------------------------- + tests/Makefile.am | 1 + + tests/tests.cmake | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 3 files changed, 92 insertions(+), 73 deletions(-) + +commit a5f2aa5618fe9183706c9c514c3067985f6c338b +Author: Lasse Collin +Date: 2024-04-20 13:12:50 +0300 + + Tests: Remove x86 and SPARC BCJ tests + + These are very old but the exact test file isn't easy to reproduce + as it was compiled from a short C program (bcj_test.c) long ago. + These tests weren't very good anyway, just a little better than nothing. + + tests/Makefile.am | 7 ---- + tests/bcj_test.c | 64 --------------------------------- + tests/compress_prepared_bcj_sparc | Bin 1240 -> 0 bytes + tests/compress_prepared_bcj_x86 | Bin 1388 -> 0 bytes + tests/files/README | 8 ----- + tests/files/good-1-sparc-lzma2.xz | Bin 612 -> 0 bytes + tests/files/good-1-x86-lzma2.xz | Bin 716 -> 0 bytes + tests/test_compress_prepared_bcj_sparc | 4 --- + tests/test_compress_prepared_bcj_x86 | 4 --- + 9 files changed, 87 deletions(-) + +commit d879686469c9c4bf2a7c0bb6420ebe4530fc8f07 +Author: Lasse Collin +Date: 2024-04-27 18:30:40 +0300 + + Tests: test_index: Edit a misleading test + + tests/test_index.c | 7 +++++-- + 1 file changed, 5 insertions(+), 2 deletions(-) + +commit 612005bbdb0dea9dc09e9e2e9cc16a15c1480acd +Author: Lasse Collin +Date: 2024-04-27 16:46:01 +0300 + + Tests: test_index: Use minimal values to test integer overflow + + tests/test_index.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 4ad88b2544c2aaf8de8f38af54587098cbe66c1d +Author: Lasse Collin +Date: 2024-04-27 15:13:39 +0300 + + Tests: test_index: Test lzma_index_buffer_decode() more + + tests/test_index.c | 29 ++++++++++++++++++++++++++--- + 1 file changed, 26 insertions(+), 3 deletions(-) + +commit 575b11b0d291e66c5fce31ce7a72f11436d57c83 +Author: Lasse Collin +Date: 2024-04-27 15:08:29 +0300 + + Tests: test_index: Test that *i = NULL is done on LZMA_PROG_ERROR + + On LZMA_DATA_ERROR from lzma_index_buffer_decode(), *i = NULL was + already done but this adds a test for that case too. + + tests/test_index.c | 31 +++++++++++++++++++++++++++---- + 1 file changed, 27 insertions(+), 4 deletions(-) + +commit 2c970debdb285823f01f75e875561d893345ac2b +Author: Lasse Collin +Date: 2024-04-27 15:01:25 +0300 + + Tests: test_index: Test lzma_index_buffer_encode() with empty output buf + + tests/test_index.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit cd88423e76d54eb72aea037364f3ebb21f122503 +Author: Lasse Collin +Date: 2024-04-27 14:59:55 +0300 + + Tests: test_index: Replace if-statements with tuktest assertions + + tests/test_index.c | 22 +++++++++------------- + 1 file changed, 9 insertions(+), 13 deletions(-) + +commit 7f865577a6224fbbb5f5ca52574b62ea8ac9bf51 +Author: Lasse Collin +Date: 2024-04-27 14:56:16 +0300 + + Tests: test_index: Make it clear that my_alloc() has no integer overflows + + liblzma guarantees that the product of the allocation size arguments + will fit in size_t. + + Putting the pre-increment in the if-statement was clearly wrong + although in practice it didn't matter here as the function is + called only a couple of times. + + tests/test_index.c | 5 ++++- + 1 file changed, 4 insertions(+), 1 deletion(-) + +commit 12313a3b6596cdcf012e180597f84d231f8730d3 +Author: Lasse Collin +Date: 2024-04-27 14:51:52 +0300 + + Tests: test_index: Verify also iter.block.number_in_stream + + tests/test_index.c | 2 ++ + 1 file changed, 2 insertions(+) + +commit ad2654010d9d641ce1601beeff00630027e6bcd4 +Author: Lasse Collin +Date: 2024-04-27 14:51:06 +0300 + + Tests: test_index: Check cases that aren't a multiple of 4 bytes + + tests/test_index.c | 33 +++++++++++++++++++++++++-------- + 1 file changed, 25 insertions(+), 8 deletions(-) + +commit 2524fcf2b68b662035437cee8edbe80067c0c240 +Author: Lasse Collin +Date: 2024-04-27 14:40:25 +0300 + + Tests: test_index: Edit comments and white space + + tests/test_index.c | 18 +++++++++++------- + 1 file changed, 11 insertions(+), 7 deletions(-) + +commit 71eed2520e2eecae89bade9dceea16e56cfa2ea0 +Author: Lasse Collin +Date: 2024-04-27 14:33:38 +0300 + + liblzma: index_decoder: Fix missing initializations on LZMA_PROG_ERROR + + If the arguments to lzma_index_decoder() or lzma_index_buffer_decode() + were such that LZMA_PROG_ERROR was returned, the lzma_index **i + argument wasn't touched even though the API docs say that *i = NULL + is done if an error occurs. This obviously won't be done even now + if i == NULL but otherwise it is best to do it due to the wording + in the API docs. + + In practice this matters very little: The problem can occur only + if the functions are called with invalid arguments, that is, + the calling application must already have a bug. + + src/liblzma/common/index_decoder.c | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit 0478473953f50716a2bc37b619b1c7dc2682b1ad +Author: Lasse Collin +Date: 2024-04-26 18:25:18 +0300 + + CMake: Bump maximum policy version to 3.29 + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit a607e2b40d23f7d998dbaba76692aa30b4c3d9d3 +Author: Sam James +Date: 2024-04-13 22:30:44 +0100 + + ci: add NetBSD + + .github/workflows/netbsd.yml | 29 +++++++++++++++++++++++++++++ + 1 file changed, 29 insertions(+) + +commit 72c210336de26fb87a928160d025fa10a638d23b +Author: Sam James +Date: 2024-04-13 23:49:26 +0100 + + ci: add FreeBSD + + .github/workflows/freebsd.yml | 29 +++++++++++++++++++++++++++++ + 1 file changed, 29 insertions(+) + +commit b526ec2dbfb5889845ea60548c4f5b1f97d84ab2 +Author: Sam James +Date: 2024-04-13 23:16:08 +0100 + + ci: add OpenBSD + + .github/workflows/openbsd.yml | 31 +++++++++++++++++++++++++++++++ + 1 file changed, 31 insertions(+) + +commit c7ef767c49351743d8d011574abb9e200bf6b24f +Author: Sam James +Date: 2024-04-15 05:53:01 +0100 + + liblzma: outqueue: add header guard + + Reported by github's codeql. + + src/liblzma/common/outqueue.h | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 55dcae3056d95cb2ddb8b560c12ba7596bc79f2c +Author: Sam James +Date: 2024-04-15 05:53:56 +0100 + + liblzma: easy_preset: add header guard + + Reported by github's codeql. + + src/liblzma/common/easy_preset.h | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 4ffc60f32397371769b7d6b5e3ed8626292d58df +Author: Lasse Collin +Date: 2024-04-25 14:00:57 +0300 + + tuklib_integer: Rename bswapXX to byteswapXX + + The __builtin_bswapXX from GCC and Clang are preferred when + they are available. This can allow compilers to emit the x86 MOVBE + instruction instead of doing a load + byteswap as two instructions + (which would happen if the byteswapping is done in inline asm). + + bswap16, bswap32, and bswap64 exist in system headers on *BSDs + and Darwin. #defining bswap16 on NetBSD results in a warning about + macro redefinition. It's safest to avoid this namespace conflict + completely. + + No OS supported by tuklib_integer.h uses byteswapXX names and + a web search doesn't immediately find any obvious danger of + namespace conflicts. So let's try these still-pretty-short names + for the macros. + + Thanks to Sam James for pointing out the compiler warning on + NetBSD 10.0. + + src/common/tuklib_integer.h | 47 ++++++++++++++++++++------------------ + src/liblzma/check/crc32_fast.c | 4 ++-- + src/liblzma/check/crc32_tablegen.c | 2 +- + src/liblzma/check/crc64_fast.c | 4 ++-- + src/liblzma/check/crc64_tablegen.c | 2 +- + 5 files changed, 31 insertions(+), 28 deletions(-) + +commit 08ab0966a75b501aa7c717622223f0c13a113c75 +Author: Lasse Collin +Date: 2024-04-24 01:20:26 +0300 + + liblzma: API doc cleanups + + src/liblzma/api/lzma/container.h | 2 +- + src/liblzma/api/lzma/index.h | 6 +++--- + src/liblzma/api/lzma/vli.h | 5 ++--- + 3 files changed, 6 insertions(+), 7 deletions(-) + +commit 3ac8a9bb4cccbee88350696dc9c645c48d77c989 +Author: Lasse Collin +Date: 2024-04-23 16:35:33 +0300 + + Tests: test_filter_str: Add a few assertions + + tests/test_filter_str.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 26c69be80523b05c84dea86c47c4ddd9a10945d7 +Author: Lasse Collin +Date: 2024-04-23 16:35:08 +0300 + + Tests: test_filter_str: Move one assertion and add a comment + + tests/test_filter_str.c | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 4f6af853bc99904efb8b6c28a0af7b81a8476c1b +Author: Lasse Collin +Date: 2024-04-23 16:26:06 +0300 + + Tests: test_filter_str: Tweak comments and white space + + tests/test_filter_str.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit c92663aa1bd576e0615498a4189acf0df12e84b9 +Author: Lasse Collin +Date: 2024-04-23 16:25:22 +0300 + + Tests: test_filter_str: Add missing RISC-V case + + Fixes: 89ea1a22f4ed3685b053b7260bc5acf6c75d1664 + + tests/test_filter_str.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit b0366df1d7ed26268101f9303a001c91c0806dfc +Author: Lasse Collin +Date: 2024-04-22 22:23:32 +0300 + + Tests: test_filter_str: Test *error_pos more thoroughly + + tests/test_filter_str.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++- + 1 file changed, 76 insertions(+), 1 deletion(-) + +commit 70d12dd069bb9bb0d6bb1c8fafc4e6f77780263d +Author: Lasse Collin +Date: 2024-04-22 21:54:39 +0300 + + liblzma: lzma_str_to_filters: Set *error_pos on all errors + + The API docs clearly say that if error_pos isn't NULL then *error + is always set on any error. However, it wasn't touched if str == NULL + or filters == NULL or unsupported flags were specified. + + Fixes: cedeeca2ea6ada5b0411b2ae10d7a859e837f203 + + src/liblzma/common/string_conversion.c | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit ed8e552395701fbf046027cebc8be4a6755b263f +Author: Lasse Collin +Date: 2024-04-22 20:31:25 +0300 + + liblzma: Clean up white space + + src/liblzma/lz/lz_encoder.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2f06920f20b1ad63b7953dc09569e1d424998849 +Author: Lasse Collin +Date: 2024-04-22 18:35:19 +0300 + + Tests: test_filter_flags: Edit comments and style + + tests/test_filter_flags.c | 13 +++++++++---- + 1 file changed, 9 insertions(+), 4 deletions(-) + +commit b101e1d1dbc81577c0c9aa0cb89cf2e46a15eb82 +Author: Lasse Collin +Date: 2024-04-22 16:39:44 +0300 + + Tests: Fix C99/C11 compatibility when features are disabled + + The array could become empty and then the initializer would be + simply {} which is allowed only in GNU-C and C23. + + tests/test_filter_flags.c | 18 ++++++++---------- + 1 file changed, 8 insertions(+), 10 deletions(-) + +commit f8f3a220ac8afcb8cb2812917d3b77e00c2eab0d +Author: Lasse Collin +Date: 2024-04-21 20:32:16 +0300 + + DOS: Omit useless defines from config.h + + dos/config.h | 12 ------------ + 1 file changed, 12 deletions(-) + +commit fc1921b04b8840caaa777c2bd5340d41b259da20 +Author: Lasse Collin +Date: 2024-04-21 20:27:50 +0300 + + Build: Omit useless checks for fcntl.h, limits.h, and sys/time.h + + configure.ac | 6 ------ + 1 file changed, 6 deletions(-) + +commit 6aa2a6deeba04808a0fe4461396e7fb70277f3d4 +Author: Lasse Collin +Date: 2024-04-19 22:04:21 +0300 + + liblzma: Silence a warning from Coverity static analysis + + It is logical why it cannot know for sure that the value has + to be at most 4 if it is less than 16. + + The x86 filter is based on a very old LZMA SDK version. Newer + ones have quite a different implementation for the same filter. + + Thanks to Sam James. + + src/liblzma/simple/x86.c | 12 +++++------- + 1 file changed, 5 insertions(+), 7 deletions(-) + +commit e89d3e83b4496d0b5410870634970c0aa9721d59 +Author: Lasse Collin +Date: 2024-04-19 23:18:19 +0300 + + Update .gitignore + + .gitignore | 21 ++++++++------------- + 1 file changed, 8 insertions(+), 13 deletions(-) + +commit 86fc4ee859709da0ff9617a1490f13ddac0a109b +Author: Lasse Collin +Date: 2024-04-19 20:53:24 +0300 + + Tests: test_lzip_decoder: Tweak coding style and comments + + tests/test_lzip_decoder.c | 58 +++++++++++++++++++++++------------------------ + 1 file changed, 28 insertions(+), 30 deletions(-) + +commit 38be573a279bd7b608ee7d8509ec10884e6fb0d5 +Author: Lasse Collin +Date: 2024-04-19 20:51:36 +0300 + + Tests: test_lzip_decoder: Remove redundant initializations + + tests/test_lzip_decoder.c | 6 ++---- + 1 file changed, 2 insertions(+), 4 deletions(-) + +commit d7e4bc53eacfab9f3de95d8252bdfdc9419079c9 +Author: Lasse Collin +Date: 2024-04-19 20:47:24 +0300 + + Tests: test_lzip_decoder: Remove unneeded tuktest_malloc() calls + + tests/test_lzip_decoder.c | 12 ++---------- + 1 file changed, 2 insertions(+), 10 deletions(-) + +commit eeca8f7c5baf1ad69606bb734d5001763466d58f +Author: Lasse Collin +Date: 2024-04-15 20:35:07 +0300 + + xz: Fix white space error. + + Thanks to xx on #tukaani. + + src/xz/args.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 462ca9409940a19f743daee6b3bcc611277d0007 +Author: Sam James +Date: 2024-04-11 23:01:44 +0100 + + xz: add missing noreturn for message_filters_help + + Fixes: a165d7df1964121eb9df715e6f836a31c865beef + + src/xz/message.h | 1 + + 1 file changed, 1 insertion(+) + +commit 863f13d2828b99b0539ce73f9cf85bde32358034 +Author: Sam James +Date: 2024-04-11 19:34:04 +0100 + + xz: signals: suppress -Wsign-conversion on macOS + + On macOS, we get: + ``` + signals.c: In function 'signals_init': + signals.c:76:17: error: conversion to 'sigset_t' {aka 'unsigned int'} from 'int' may change the sign of the result [-Werror=sign-conversion] + 76 | sigaddset(&hooked_signals, sigs[i]); + | ^~~~~~~~~ + signals.c:81:17: error: conversion to 'sigset_t' {aka 'unsigned int'} from 'int' may change the sign of the result [-Werror=sign-conversion] + 81 | sigaddset(&hooked_signals, message_progress_sigs[i]); + | ^~~~~~~~~ + signals.c:86:9: error: conversion to 'sigset_t' {aka 'unsigned int'} from 'int' may change the sign of the result [-Werror=sign-conversion] + 86 | sigaddset(&hooked_signals, SIGTSTP); + | ^~~~~~~~~ + ``` + + We use `int` for `hooked_signals` but we can't just cast to whatever + `sigset_t` is because `sigset_t` is an opaque type. It's an unsigned int + on macOS. On macOS, `sigaddset` is implemented as a macro. + + Just suppress -Wsign-conversion for `signals_init` for macOS given + there's no real nice way of fixing this. + + src/xz/signals.c | 7 +++++++ + 1 file changed, 7 insertions(+) + +commit fcbd0d199933a69713cb293cbd7409a757d854cd +Author: Lasse Collin +Date: 2024-04-13 22:19:40 +0300 + + Tests: test_microlzma: Add a "FIXME?" about LZMA_FINISH handling + + tests/test_microlzma.c | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit 0fe2dfa68355d2b165544b2bc8babf77dcc2039e +Author: Lasse Collin +Date: 2024-04-13 18:05:31 +0300 + + Tests: test_microlzma: Tweak comments, coding style, and minor details + + A few lines were reordered, a few ARRAY_SIZE were changed to sizeof, + and a few uint32_t were changed to size_t. No real functional changes + were intended. + + tests/test_microlzma.c | 149 +++++++++++++++++++++++++++---------------------- + 1 file changed, 83 insertions(+), 66 deletions(-) + +commit 97f0ee0f1f903f4e7c4ea23e9b89d687025d2992 +Author: Ryan Carsten Schmidt +Date: 2024-04-12 19:31:13 -0500 + + CI: Use only the active CPUs on macOS + + hw.ncpu counts all CPUs including inactive ones. hw.activecpu counts + only the active CPUs. + + build-aux/ci_build.bash | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 73f629e321b74f68c9954728fa4f19261afccf46 +Author: Sam James +Date: 2024-04-10 18:33:55 +0100 + + ci: rename ci_build.sh -> ci_build.bash + + We discussed the name and it's less cognitive load to just call it '.bash' + so you don't have an immediate question about if bashisms are OK. + + .github/workflows/ci.yml | 52 ++++++++++++++++---------------- + .github/workflows/windows-ci.yml | 20 ++++++------ + build-aux/{ci_build.sh => ci_build.bash} | 0 + 3 files changed, 36 insertions(+), 36 deletions(-) + +commit 8709407a9ef8e7e8aec117879400e4dd3e227ada +Author: Sam James +Date: 2024-04-10 17:42:23 +0100 + + ci: build in parallel by default + + build-aux/ci_build.sh | 2 ++ + 1 file changed, 2 insertions(+) + +commit 65bf7e0a1ca6386f17608e8afb84ac470c18d23f +Author: Sam James +Date: 2024-04-10 15:41:08 +0100 + + ci: default to -O2 + + We need this for when we're passing sanitizer flags or -gdwarf-4 for Clang + with Valgrind. Just always start with -O2 if CFLAGS isn't set in the + environment and append what was passed on the command line. + + build-aux/ci_build.sh | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit bc899f9e0700ad153bd65f4804c4de7515c8a847 +Author: Sam James +Date: 2024-04-10 15:17:47 +0100 + + ci: make automake's test runner verbose on failures + + This is a lot easier to work with than the save-logs thing the action + tries to do... + + build-aux/ci_build.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b5e3470442531717b2457b40ab412740296af1bc +Author: Sam James +Date: 2024-04-10 12:38:51 +0100 + + ci: make UBSAN abort on errors + + Unfortunately, UBSAN doesn't do this by default. See also the change + I made in Meson for this in October [0]. + + [0] https://github.com/mesonbuild/meson/commit/7b7d2e060b447de9c2642848847370a58711ac1c + + .github/workflows/ci.yml | 1 + + 1 file changed, 1 insertion(+) + +commit 6c095a98fbec70b790253a663173ecdb669108c4 +Author: Sam James +Date: 2024-04-10 11:43:10 +0100 + + ci: test Valgrind + + Using `--trace-children=yes` has a trade-off here, as it makes + `test_scripts.sh` pretty slow when calling various non-xz utilities. + + But I also feel like it's not useless to have Valgrind used there and it's + not easy to exclude Valgrind just for that one test... + + I did consider using AX_VALGRIND_CHECK [0][1] but I couldn't get it working + immediately with some conditionally-built tests and I wondered if it was + worth spending time on at least while we're debating xz's future build + system situation. + + [0] https://www.gnu.org/software/autoconf-archive/ax_valgrind_check.html + [1] https://tecnocode.co.uk/2014/12/23/automatically-valgrinding-code-with-ax_valgrind_check/ + + .github/workflows/ci.yml | 11 ++++++++++- + build-aux/ci_build.sh | 8 +++++--- + 2 files changed, 15 insertions(+), 4 deletions(-) + +commit 6286c1900c2d2ca33d9b1b397122c7bcdb9a4d59 +Author: Lasse Collin +Date: 2024-04-10 23:20:02 +0300 + + liblzma: CRC: Simplify table omission macros + + A macro is useful to prevent a single #if directive from + getting too ugly but only one macro is needed for all archs. + + src/liblzma/check/crc32_table.c | 10 ++++------ + src/liblzma/check/crc64_table.c | 4 ++-- + src/liblzma/check/crc_common.h | 5 +++-- + 3 files changed, 9 insertions(+), 10 deletions(-) + +commit 45da936c879acf4f053a3055665bf1b10ded4462 +Author: Lasse Collin +Date: 2024-04-10 23:09:40 +0300 + + liblzma: ARM64 CRC: Fix omission of CRC32 table + + The macro name had an odd typo so the table wasn't omitted + when it should have. + + Fixes: 1940f0ec28f08c0ac72c1413d9706fb82eabe6ad + + src/liblzma/check/crc32_table.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 308a9af85400b0e2019f0f012c8354e831d06d65 +Author: Lasse Collin +Date: 2024-04-10 22:21:51 +0300 + + Build: If ARM64 feature detection func is found, stop looking for others + + This can speed up configure a tiny bit. + + Fixes: c5f6d79cc9515a7f22d7ea4860c6cc394b295732 + + configure.ac | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit fc43cecd32bf9d5f8caa599206b15c9569af1eb6 +Author: Lasse Collin +Date: 2024-04-10 22:04:27 +0300 + + liblzma: ARM64 CRC32: Change style of the macOS code to match FreeBSD + + I didn't test this but it shouldn't change any functionality. + + Fixes: 761f5b69a4c778c8bcb09279b845b07c28790575 + + src/liblzma/check/crc32_arm64.h | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +commit 1024cd4cd966b998fedec51e385e9ee9a49b3c57 +Author: Lasse Collin +Date: 2024-04-10 21:59:27 +0300 + + liblzma: ARM64 CRC32: Add error checking to FreeBSD-specific code + + Also add parenthesis to the return statement. + + I didn't test this. + + Fixes: 761f5b69a4c778c8bcb09279b845b07c28790575 + + src/liblzma/check/crc32_arm64.h | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 2337f7021c860b026e3e849e60a9ae8d09ec0ea0 +Author: Lasse Collin +Date: 2024-04-10 21:56:33 +0300 + + liblzma: ARM64 CRC32: Use negation instead of subtracting from 8 + + Subtracting from 0 is negation, this just keeps warnings away. + + Fixes: 761f5b69a4c778c8bcb09279b845b07c28790575 + + src/liblzma/check/crc32_arm64.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit d8fffd01aa1a3c18e437a222abd34699e23ff5e7 +Author: Lasse Collin +Date: 2024-04-10 21:55:10 +0300 + + liblzma: ARM64 CRC32: Tweak coding style and comments + + src/liblzma/check/crc32_arm64.h | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit 780d2c236de0e4749655696c2e0c26fb7565afd3 +Author: Lasse Collin +Date: 2024-04-09 21:55:01 +0300 + + Update SECURITY.md. + + .github/SECURITY.md | 25 ++++++++----------------- + 1 file changed, 8 insertions(+), 17 deletions(-) + +commit 986865ea2f9d1f8dbef4a130926df106b0f6d41a +Author: Lasse Collin +Date: 2024-04-09 17:47:01 +0300 + + CI: Remove ifunc support. + + .github/workflows/ci.yml | 13 +++---------- + build-aux/ci_build.sh | 5 +---- + 2 files changed, 4 insertions(+), 14 deletions(-) + +commit 689ae2427342a2ea1206eb5ca08301baf410e7e0 +Author: Lasse Collin +Date: 2024-04-09 17:43:16 +0300 + + liblzma: Remove ifunc support. + + This is *NOT* done for security reasons even though the backdoor + relied on the ifunc code. Instead, the reason is that in this + project ifunc provides little benefits but it's quite a bit of + extra code to support it. The only case where ifunc *might* matter + for performance is if the CRC functions are used directly by an + application. In normal compression use it's completely irrelevant. + + CMakeLists.txt | 79 --------------------------------------- + INSTALL | 8 ---- + configure.ac | 79 --------------------------------------- + src/liblzma/check/crc32_fast.c | 48 +++--------------------- + src/liblzma/check/crc64_fast.c | 21 ----------- + src/liblzma/check/crc_common.h | 9 +---- + src/liblzma/check/crc_x86_clmul.h | 11 +----- + 7 files changed, 8 insertions(+), 247 deletions(-) + +commit 6b4c859059a7eb9b0547590c081668e14ecf8af6 +Author: Lasse Collin +Date: 2024-04-08 22:04:41 +0300 + + tests/files/README: Update the main heading. + + tests/files/README | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 2a851e06b891ce894f918faff32a6cca6fdecee6 +Author: Lasse Collin +Date: 2024-04-08 22:02:45 +0300 + + tests/files/README: Explain how to recreate the ARM64 test files. + + tests/files/README | 15 ++++++++++++++- + 1 file changed, 14 insertions(+), 1 deletion(-) + +commit 3d09b721b94e18fe1f853a04799697f5de10b291 +Author: Lasse Collin +Date: 2024-04-08 21:51:55 +0300 + + debug: Add generator for the ARM64 test file data. + + debug/Makefile.am | 3 +- + debug/testfilegen-arm64.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 118 insertions(+), 1 deletion(-) + +commit 31ef676567c9d6fcc4ec9fc833c312f7a7c21c48 +Author: Lasse Collin +Date: 2024-04-08 21:19:38 +0300 + + xz man page: Use .ft CR instead of CW to silence warnings from groff. + + src/xz/xz.1 | 32 ++++++++++++++++---------------- + 1 file changed, 16 insertions(+), 16 deletions(-) + +commit 780cbf29d5a88db2b546e9b7b019c4c33ca72685 +Author: Lasse Collin +Date: 2024-04-08 19:28:35 +0300 + + Fix NEWS for 5.6.0 and 5.6.1. + + NEWS | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit bfd0c7c478e93a1911b845459549ff94587b6ea2 +Author: Lasse Collin +Date: 2024-04-08 19:22:26 +0300 + + Remove the XZ logo. + + COPYING | 5 - + COPYING.CC-BY-SA-4.0 | 427 --------------------------------------------------- + Makefile.am | 2 - + README | 2 - + doc/xz-logo.png | Bin 6771 -> 0 bytes + doxygen/Doxyfile | 6 +- + doxygen/footer.html | 13 -- + 7 files changed, 3 insertions(+), 452 deletions(-) + +commit 77a294d98a9d2d48f7e4ac273711518bf689f5c4 +Author: Lasse Collin +Date: 2024-04-08 18:27:39 +0300 + + Update maintainer and author info. + + The other maintainer suddenly disappeared. + + AUTHORS | 9 +++++++-- + README | 10 +++------- + THANKS | 1 - + src/liblzma/api/lzma.h | 2 +- + 4 files changed, 11 insertions(+), 11 deletions(-) + +commit 8dd03d4484ccf80022722a16d0ed9b37f2b58072 +Author: Lasse Collin +Date: 2024-04-08 18:05:32 +0300 + + Docs: Update .xz file format specification to 1.2.1. + + This only reverts the XZ URL changes. + + doc/xz-file-format.txt | 12 ++++++++---- + 1 file changed, 8 insertions(+), 4 deletions(-) + +commit 17aa2e1a796d3f758802df29afc89dcf335db567 +Author: Lasse Collin +Date: 2024-04-08 17:33:56 +0300 + + Update website URLs back to tukaani.org. + + The XZ projects were moved back to their original URLs. + + .github/SECURITY.md | 2 +- + CMakeLists.txt | 2 +- + COPYING | 3 +-- + README | 4 ++-- + configure.ac | 2 +- + doc/faq.txt | 2 +- + doc/lzma-file-format.txt | 12 ++++++------ + dos/config.h | 2 +- + src/liblzma/api/lzma.h | 2 +- + src/xz/xz.1 | 6 +++--- + src/xzdec/xzdec.1 | 4 ++-- + windows/README-Windows.txt | 2 +- + 12 files changed, 21 insertions(+), 22 deletions(-) + +commit 2739db981023373a2ddabc7b456c7e658bb4f582 +Author: Lasse Collin +Date: 2024-04-08 17:07:08 +0300 + + xzdec: Tweak coding style and comments. + + src/xzdec/xzdec.c | 32 +++++++++++++++++++++----------- + 1 file changed, 21 insertions(+), 11 deletions(-) + +commit 408b6adb2a07d07c6535f859571cca38837caaf3 +Author: Lasse Collin +Date: 2024-04-08 15:53:46 +0300 + + tests/ossfuzz: Tiny fix to a comment. + + tests/ossfuzz/fuzz_decode_stream.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit db4dd74a344580e0b81436598d9741a3454245b0 +Author: Lasse Collin +Date: 2024-04-09 18:22:16 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit e93e13c8b3bec925c56e0c0b675d8000a0f7f754 +Author: Lasse Collin +Date: 2024-04-08 15:32:58 +0300 + + Remove the backdoor found in 5.6.0 and 5.6.1 (CVE-2024-3094). + + While the backdoor was inactive (and thus harmless) without inserting + a small trigger code into the build system when the source package was + created, it's good to remove this anyway: + + - The executable payloads were embedded as binary blobs in + the test files. This was a blatant violation of the + Debian Free Software Guidelines. + + - On machines that see lots bots poking at the SSH port, the backdoor + noticeably increased CPU load, resulting in degraded user experience + and thus overwhelmingly negative user feedback. + + - The maintainer who added the backdoor has disappeared. + + - Backdoors are bad for security. + + This reverts the following without making any other changes: + + 6e636819 Tests: Update two test files. + a3a29bbd Tests: Test --single-stream can decompress bad-3-corrupt_lzma2.xz. + 0b4ccc91 Tests: Update RISC-V test files. + 8c9b8b20 liblzma: Fix typos in crc32_fast.c and crc64_fast.c. + 82ecc538 liblzma: Fix false Valgrind error report with GCC. + cf44e4b7 Tests: Add a few test files. + 3060e107 Tests: Use smaller dictionary size in RISC-V test files. + e2870db5 Tests: Add two RISC-V Filter test files. + + The RISC-V test files also have real content that tests the filter + but the real content would fit into much smaller files. A generator + program would need to be available as well. + + Thanks to Andres Freund for finding and reporting it and making + it public quickly so others could act without a delay. + See: https://www.openwall.com/lists/oss-security/2024/03/29/4 + + src/liblzma/check/crc32_fast.c | 7 +++++-- + src/liblzma/check/crc64_fast.c | 4 +++- + src/liblzma/check/crc_common.h | 25 ------------------------- + tests/files/README | 27 --------------------------- + tests/files/bad-3-corrupt_lzma2.xz | Bin 512 -> 0 bytes + tests/files/bad-dict_size.lzma | Bin 41 -> 0 bytes + tests/files/good-1-riscv-lzma2-1.xz | Bin 7424 -> 0 bytes + tests/files/good-1-riscv-lzma2-2.xz | Bin 7432 -> 0 bytes + tests/files/good-2cat.xz | Bin 136 -> 0 bytes + tests/files/good-large_compressed.lzma | Bin 35421 -> 0 bytes + tests/files/good-small_compressed.lzma | Bin 258 -> 0 bytes + tests/test_files.sh | 11 ----------- + 12 files changed, 8 insertions(+), 66 deletions(-) + +commit f9cf4c05edd14dedfe63833f8ccbe41b55823b00 +Author: Lasse Collin +Date: 2024-03-30 14:36:28 +0200 + + CMake: Fix sabotaged Landlock sandbox check. + + It never enabled it. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit af071ef7702debef4f1d324616a0137a5001c14c +Author: Jia Tan +Date: 2024-03-26 01:50:02 +0800 + + Docs: Simplify SECURITY.md. + + .github/SECURITY.md | 8 +------- + 1 file changed, 1 insertion(+), 7 deletions(-) + +commit 0b99783d63f27606936bb79a16c52d0d70c0b56f +Author: Lasse Collin +Date: 2024-03-22 17:46:30 +0200 + + liblzma: memcmplen.h: Add a comment why subtraction is used. + + src/liblzma/common/memcmplen.h | 13 +++++++++++++ + 1 file changed, 13 insertions(+) + +commit 8a25ba024d55610c448c6e4f1400a00bae51b493 +Author: Lasse Collin +Date: 2024-03-15 17:43:39 +0200 + + INSTALL: Document arguments of --enable-symbol-versions. + + INSTALL | 43 +++++++++++++++++++++++++++++++++++++++---- + 1 file changed, 39 insertions(+), 4 deletions(-) + +commit 49324b711f9d42b3543bf2f3ae598eaa03360bd5 +Author: Lasse Collin +Date: 2024-03-15 17:15:50 +0200 + + Build: Use only the generic symbol versioning with NVIDIA HPC Compiler. + + This does the previous commit with CMake. + + AC_EGREP_CPP uses AC_REQUIRE so the outermost if-commands must + be changed to AS_IF to ensure that things wont break some day. + See 5a5bd7f871818029d5ccbe189f087f591258c294. + + configure.ac | 18 +++++++++++++----- + 1 file changed, 13 insertions(+), 5 deletions(-) + +commit c273123ed0ebaebf49994057a7fe98aae7f42c40 +Author: Lasse Collin +Date: 2024-03-15 16:36:35 +0200 + + CMake: Use only the generic symbol versioning with NVIDIA HPC Compiler. + + It doesn't support the __symver__ attribute or __asm__(".symver ..."). + The generic symbol versioning can still be used since it only needs + linker support. + + CMakeLists.txt | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit df7f487648d18a3992386a59b8a061edca862d17 +Author: Lasse Collin +Date: 2024-03-13 21:38:24 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 3217b82b3ec023bf8338249134a076bea0ea30ec +Author: Lasse Collin +Date: 2024-03-13 21:30:18 +0200 + + liblzma: Minor comment edits. + + src/liblzma/common/string_conversion.c | 4 ++-- + src/liblzma/delta/delta_decoder.c | 2 ++ + 2 files changed, 4 insertions(+), 2 deletions(-) + +commit 096bc0e3f8fb4bfc4d2f3f64a7f219401ffb4c31 +Author: Sergey Kosukhin +Date: 2024-03-13 13:07:13 +0100 + + liblzma: Fix building with NVHPC (NVIDIA HPC SDK). + + NVHPC compiler has several issues that make it impossible to + build liblzma: + - the compiler cannot handle unions that contain pointers that + are not the first members; + - the compiler cannot handle the assembler code in range_decoder.h + (LZMA_RANGE_DECODER_CONFIG has to be set to zero); + - the compiler fails to produce valid code for delta_decode if the + vectorization is enabled, which results in failed tests. + + This introduces NVHPC-specific workarounds that address the issues. + + src/liblzma/common/string_conversion.c | 6 ++++-- + src/liblzma/delta/delta_decoder.c | 3 +++ + src/liblzma/rangecoder/range_decoder.h | 1 + + 3 files changed, 8 insertions(+), 2 deletions(-) + +commit 2ad7fad67080e88fa7fc191f9d613d8b7add9c62 +Author: Lasse Collin +Date: 2024-03-13 21:17:10 +0200 + + CMake: Disable symbol versioning on non-glibc Linux. + + This better matches what configure.ac does. For example, musl has + only basic symbol versioning support: + + https://wiki.musl-libc.org/functional-differences-from-glibc.html#Symbol_versioning + + configure.ac tries to enable symbol versioning only with glibc + so now CMake does the same. + + CMakeLists.txt | 22 ++++++++++++++++++++-- + 1 file changed, 20 insertions(+), 2 deletions(-) + +commit 82f0c0d39eb2c026b1d96ee706f70ace868d4ed4 +Author: Lasse Collin +Date: 2024-03-13 20:32:46 +0200 + + CMake: Make symbol versioning configurable. + + CMakeLists.txt | 62 +++++++++++++++++++++++++++++++++++++++------------------- + 1 file changed, 42 insertions(+), 20 deletions(-) + +commit 45d33bfc45e4295b8ad743bc2ae61cc724f98076 +Author: Lasse Collin +Date: 2024-03-13 19:47:36 +0200 + + Build: Style tweaks to configure.ac. + + The AC_MSG_ERROR line is overlong anyway as are a few other + AC_MSG_ERROR lines already. + + configure.ac | 16 +++++++++------- + 1 file changed, 9 insertions(+), 7 deletions(-) + +commit f56ed6fac6619b56b005878d3b5210e2f0d721c0 +Author: Sergey Kosukhin +Date: 2024-03-12 20:03:49 +0100 + + Build: Let the users override the symbol versioning variant. + + There are cases when the users want to decide themselves whether + they want to have the generic (even on GNU/Linux) or the linux + (even if we do not recommend that) symbol versioning variant. + The former might be needed to circumvent compiler issues (i.e. + the compiler does not support all features that are required + for the linux versioning), the latter might help in overriding + the assumptions made in the configure script. + + configure.ac | 91 +++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 50 insertions(+), 41 deletions(-) + +commit a4f2e20d8466369b1bb277c66f75c9e4ba9cc378 +Author: Jia Tan +Date: 2024-03-09 11:27:27 +0800 + + Add NEWS for 5.6.1 + + NEWS | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +commit f01be8ad754a905d8c418601767480ec11621b02 +Author: Jia Tan +Date: 2024-03-09 10:43:20 +0800 + + Translations: Add missing --riscv option to man page translations. + + po4a/de.po | 702 +++++++++++++++++++++++++++++----------------------------- + po4a/fr.po | 549 ++++++++++++++++++++++----------------------- + po4a/ko.po | 702 +++++++++++++++++++++++++++++----------------------------- + po4a/pt_BR.po | 641 +++++++++++++++++++++++++++-------------------------- + po4a/ro.po | 702 +++++++++++++++++++++++++++++----------------------------- + po4a/uk.po | 702 +++++++++++++++++++++++++++++----------------------------- + 6 files changed, 2024 insertions(+), 1974 deletions(-) + +commit 6e636819e8f070330d835fce46289a3ff72a7b89 +Author: Jia Tan +Date: 2024-03-09 10:18:29 +0800 + + Tests: Update two test files. + + The original files were generated with random local to my machine. + To better reproduce these files in the future, a constant seed was used + to recreate these files. + + tests/files/bad-3-corrupt_lzma2.xz | Bin 484 -> 512 bytes + tests/files/good-large_compressed.lzma | Bin 35430 -> 35421 bytes + 2 files changed, 0 insertions(+), 0 deletions(-) + +commit a3a29bbd5d86183fc7eae8f0182dace374e778d8 +Author: Jia Tan +Date: 2024-03-09 10:08:32 +0800 + + Tests: Test --single-stream can decompress bad-3-corrupt_lzma2.xz. + + The first stream in this file is valid, so this tests that xz properly + stops after decompressing it. + + tests/test_files.sh | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit 0b4ccc91454dbcf0bf521b9bd51aa270581ee23c +Author: Jia Tan +Date: 2024-03-09 10:05:32 +0800 + + Tests: Update RISC-V test files. + + This increases code coverage and tests for possible shifting bugs. + + tests/files/good-1-riscv-lzma2-1.xz | Bin 7512 -> 7424 bytes + tests/files/good-1-riscv-lzma2-2.xz | Bin 7512 -> 7432 bytes + 2 files changed, 0 insertions(+), 0 deletions(-) + +commit 8c9b8b2063daa78ead9f648c2ec3c91e8615dffb +Author: Jia Tan +Date: 2024-03-09 09:52:32 +0800 + + liblzma: Fix typos in crc32_fast.c and crc64_fast.c. + + src/liblzma/check/crc32_fast.c | 4 ++-- + src/liblzma/check/crc64_fast.c | 3 +-- + 2 files changed, 3 insertions(+), 4 deletions(-) + +commit b93a8d7631d9517da63f03e0185455024a4609e8 +Author: Jia Tan +Date: 2024-03-09 09:49:55 +0800 + + Tests: Replace HAVE_MICROLZMA usage in CMake and Autotools builds. + + This reverts commit adaacafde6661496ca2814b1e94a3ba5186428cb. + + CMakeLists.txt | 15 ++++++++++----- + configure.ac | 9 ++------- + tests/Makefile.am | 9 ++++++--- + tests/test_microlzma.c | 12 ++++-------- + 4 files changed, 22 insertions(+), 23 deletions(-) + +commit 82ecc538193b380a21622aea02b0ba078e7ade92 +Author: Jia Tan +Date: 2024-03-09 09:20:57 +0800 + + liblzma: Fix false Valgrind error report with GCC. + + With GCC and a certain combination of flags, Valgrind will falsely + trigger an invalid write. This appears to be due to the omission of + instructions to properly save, set up, and restore the frame pointer. + + The IFUNC resolver is a leaf function since it only calls a function + that is inlined. So sometimes GCC omits the frame pointer instructions + in the resolver unless this optimization is explictly disabled. + + This fixes https://bugzilla.redhat.com/show_bug.cgi?id=2267598. + + src/liblzma/check/crc32_fast.c | 9 +++------ + src/liblzma/check/crc64_fast.c | 7 +++---- + src/liblzma/check/crc_common.h | 25 +++++++++++++++++++++++++ + 3 files changed, 31 insertions(+), 10 deletions(-) + +commit 3007e74ef250f0ce95d97ffbdf2282284f93764d +Author: Lasse Collin +Date: 2024-03-05 23:21:26 +0200 + + liblzma: Fix a typo in a comment in the RISC-V filter. + + src/liblzma/simple/riscv.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 72d2933bfae514e0dbb123488e9f1eb7cf64175f +Author: Jia Tan +Date: 2024-03-05 00:34:46 +0800 + + liblzma: Use attribute no_profile_instrument_function with ifunc. + + Thanks to Sam James for determining this was the attribute needed to + workaround the GCC bug and for his version of the patch in Gentoo. + + src/liblzma/check/crc32_fast.c | 5 +++++ + src/liblzma/check/crc64_fast.c | 3 +++ + 2 files changed, 8 insertions(+) + +commit e5faaebbcf02ea880cfc56edc702d4f7298788ad +Author: Jia Tan +Date: 2024-03-05 00:27:31 +0800 + + Build: Require attribute no_profile_instrument_function for ifunc usage. + + Using __attribute__((__no_profile_instrument_function__)) on the ifunc + resolver works around a bug in GCC -fprofile-generate: + it adds profiling code even to ifunc resolvers which can make + the ifunc resolver crash at program startup. This attribute + was not introduced until GCC 7 and Clang 13, so ifunc won't + be used with prior versions of these compilers. + + This bug was brought to our attention by: + + https://bugs.gentoo.org/925415 + + And was reported to upstream GCC by: + + https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411 + + CMakeLists.txt | 7 +++++++ + configure.ac | 7 +++++++ + 2 files changed, 14 insertions(+) + +commit 7eeadd279a24c26ca7ff1292b7df802b89409eb7 +Author: Lasse Collin +Date: 2024-03-04 19:23:18 +0200 + + liblzma: Fix a comment in the RISC-V filter. + + src/liblzma/simple/riscv.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 5f3d0595296cc3035eae9e7bb6c3ffb1e1267333 +Author: Lasse Collin +Date: 2024-02-29 16:35:52 +0200 + + CMake: Warn if translated man pages are missing. + + CMakeLists.txt | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit 4cd1042ee752d61370c685d0d8b20c1e935672f7 +Author: Lasse Collin +Date: 2024-02-29 16:35:52 +0200 + + CMake: Warn if gettext tools and pre-created .gmo files are missing. + + It's only done with CMake >= 3.20 and if library support + for translation was already found. + + Sort of fixes: https://github.com/tukaani-project/xz/issues/82 + + CMakeLists.txt | 25 +++++++++++++++++++++++++ + 1 file changed, 25 insertions(+) + +commit a94b42362c8e807f92236d6d63373f04991e3a50 +Author: Lasse Collin +Date: 2024-02-28 18:26:25 +0200 + + xz: Add comments. + + src/xz/coder.c | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit bbf112e32307a75a54a9e170bc392811443d5c87 +Author: Jia Tan +Date: 2024-02-27 23:42:41 +0800 + + xz: Change logging level for thread reduction to highest verbosity only. + + Now that multi threaded encoding is the default, users do not need to + see a warning message everytime the number of threads is reduced. On + some machines, this could happen very often. It is not unreasonable for + users to need to set double verbose mode to see this kind of + information. + + To see these warning messages -vv or --verbose --verbose must be passed + to set xz into the highest possible verbosity mode. + + These warnings had caused automated testing frameworks to fail when they + expected no output to stderr. + + Thanks to Sebastian Andrzej Siewior for reporting this and for the + initial version of the patch. + + src/xz/coder.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 649f6447441510d593a88475ad6df4bcdf74ce48 +Author: Lasse Collin +Date: 2024-02-26 23:06:13 +0200 + + Fix sorting in THANKS. + + THANKS | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 1255b7d849bf53f196a842ef2a508ed0ff577eaa +Author: Jia Tan +Date: 2024-02-26 23:39:29 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit eee579fff50099ba163c12305e81a4bd42b7dd53 +Author: Chien Wong +Date: 2024-02-25 21:38:13 +0800 + + xz: Add missing RISC-V on the filter list in the man page + + Signed-off-by: Chien Wong + + src/xz/xz.1 | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 328c52da8a2bbb81307644efdb58db2c422d9ba7 +Author: Jia Tan +Date: 2024-02-26 23:02:06 +0800 + + Build: Fix Linux Landlock feature test in Autotools and CMake builds. + + The previous Linux Landlock feature test assumed that having the + linux/landlock.h header file was enough. The new feature tests also + requires that prctl() and the required Landlock system calls are + supported. + + CMakeLists.txt | 25 ++++++++++++++++++++++--- + configure.ac | 27 ++++++++++++++++++++++++++- + src/xz/sandbox.c | 2 +- + src/xz/sandbox.h | 2 +- + src/xzdec/xzdec.c | 8 ++++---- + 5 files changed, 54 insertions(+), 10 deletions(-) + +commit eb8ad59e9bab32a8d655796afd39597ea6dcc64d +Author: Jia Tan +Date: 2024-02-26 20:06:10 +0800 + + Tests: Add test_microlzma to .gitignore and CMakeLists.txt. + + .gitignore | 1 + + CMakeLists.txt | 1 + + 2 files changed, 2 insertions(+) + +commit 9eed1b9a3ae140e93a82febc05a0181e9a4f5093 +Author: Jia Tan +Date: 2024-02-26 19:56:25 +0800 + + Tests: Correct license header in test_microlzma.c. + + tests/test_microlzma.c | 5 ++--- + 1 file changed, 2 insertions(+), 3 deletions(-) + +commit 8bf9f72ee1c05b9e205a72807e8a9e304785673d +Author: Jia Tan +Date: 2024-02-25 21:41:55 +0800 + + Fix typos in NEWS and CMakeLists. + + CMakeLists.txt | 2 +- + NEWS | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit 5d8d915ebe2e345820a0f54d1baf8d7d4824c0c7 +Author: Jia Tan +Date: 2024-02-24 16:30:06 +0800 + + Bump version and soname for 5.7.0alpha. + + Like 5.5.0alpha, 5.7.0alpha won't be released, it's just to mark that + the branch is not stable. + + Once again there is no API/ABI stability for new features in devel + versions. The major soname won't be bumped even if API/ABI of new + features breaks between devel releases. + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 6 +++--- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 4 files changed, 6 insertions(+), 6 deletions(-) + +commit a18fb1edef0d0aac12a09eed05e9c448c777af7b +Author: Jia Tan +Date: 2024-02-24 15:50:36 +0800 + + Add NEWS for 5.6.0. + + NEWS | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 143 insertions(+) + +commit 24355c5280bc95e3d594432d60bb8432aa6af173 +Author: Jia Tan +Date: 2024-02-22 22:27:01 +0800 + + Translations: Remove obsolete and fuzzy matches from some translations. + + The French and Brazilian Portuguese man page translations have not been + updated since the switch from public domain to 0BSD. The old GPLv2 + strings have now been removed from these files. + + po4a/fr.po | 4702 +++++++++++++++++++++++++++++++++++++---------------- + po4a/pt_BR.po | 4987 ++++++++++++++++++++++++++++++++++++++++----------------- + 2 files changed, 6832 insertions(+), 2857 deletions(-) + +commit 02ca4a7d7b703e2ec63e00b70feec825e919dbc1 +Author: Jia Tan +Date: 2024-02-21 00:31:54 +0800 + + Translations: Patch man pages to avoid fuzzy matches. + + This will be fixed in the next round of translations, but this avoids + having a fuzzy match or not fixing the English version. + + po4a/de.po | 2 +- + po4a/ko.po | 2 +- + po4a/ro.po | 2 +- + po4a/uk.po | 2 +- + 4 files changed, 4 insertions(+), 4 deletions(-) + +commit 898aad9fc711e03452d24d9e2c5b7f77a6f9ce64 +Author: Jia Tan +Date: 2024-02-21 00:30:43 +0800 + + xzmore: Fix typo in xzmore.1. + + Thanks to Yuri Chornoivan. + + src/scripts/xzmore.1 | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 5631aa206c8d16b4eeab85a46b8b698f4fc4cdba +Author: Jia Tan +Date: 2024-02-24 12:12:16 +0800 + + Translations: Update the Vietnamese translation. + + po/vi.po | 505 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 309 insertions(+), 196 deletions(-) + +commit a65fd7ce9d6228e87faf61dc56a35984d0088248 +Author: Jia Tan +Date: 2024-02-24 12:06:40 +0800 + + Translations: Update the Esperanto translation. + + po/eo.po | 502 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 306 insertions(+), 196 deletions(-) + +commit cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0 +Author: Jia Tan +Date: 2024-02-23 23:09:59 +0800 + + Tests: Add a few test files. + + tests/files/README | 19 +++++++++++++++++++ + tests/files/bad-3-corrupt_lzma2.xz | Bin 0 -> 484 bytes + tests/files/bad-dict_size.lzma | Bin 0 -> 41 bytes + tests/files/good-2cat.xz | Bin 0 -> 136 bytes + tests/files/good-large_compressed.lzma | Bin 0 -> 35430 bytes + tests/files/good-small_compressed.lzma | Bin 0 -> 258 bytes + 6 files changed, 19 insertions(+) + +commit 39f4a1a86ad80b2d064b812cee42668e6c8b8c73 +Author: Jia Tan +Date: 2024-02-23 20:58:36 +0800 + + Tests: Add MicroLZMA test. + + tests/Makefile.am | 4 +- + tests/test_microlzma.c | 548 +++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 551 insertions(+), 1 deletion(-) + +commit adaacafde6661496ca2814b1e94a3ba5186428cb +Author: Jia Tan +Date: 2024-02-23 20:57:59 +0800 + + Build: Define HAVE_MICROLZMA when it is configured. + + CMakeLists.txt | 4 ++++ + configure.ac | 9 +++++++-- + 2 files changed, 11 insertions(+), 2 deletions(-) + +commit eea78216d27182ca917bf00e02feaab058a4d21e +Author: Jia Tan +Date: 2024-02-23 20:27:15 +0800 + + xz: Fix Capsicum sandbox compile error. + + user_abort_pipe[] was still being used instead of the parameters. + + src/xz/sandbox.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 32b0a3ce19224f9074d01a4ffbc1655b05fcb82d +Author: Jia Tan +Date: 2024-02-23 16:12:32 +0800 + + Build: Fix ARM64 CRC32 instruction feature test. + + Old versions of Clang reported the unsupported function attribute and + __crc32d() function as warnings instead of errors, so the feature test + passed when it shouldn't have, causing a compile error at build time. + -Werror was added to this feature test to fix this. The change is not + needed for CMake because check_c_source_compiles() also performs + linking and the error is caught then. + + Thanks to Sebastian Andrzej Siewior for reporting this. + + configure.ac | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit 4c81c9611f8b2e1ad65eb7fa166afc570c58607e +Author: Lasse Collin +Date: 2024-02-22 19:16:35 +0200 + + CMake: Add LOCALEDIR to the windres workaround. + + LOCALEDIR may contain spaces like in "C:\Program Files". + + CMakeLists.txt | 16 +++++++++++----- + 1 file changed, 11 insertions(+), 5 deletions(-) + +commit de4337fd89ca7db5feb97b5c40143404f6e22986 +Author: Lasse Collin +Date: 2024-02-22 15:18:25 +0200 + + xz: Landlock: Fix error message if input file is a directory. + + If xz is given a directory, it should look like this: + + $ xz /usr/bin + xz: /usr/bin: Is a directory, skipping + + The Landlock rules didn't allow opening directories for reading: + + $ xz /usr/bin + xz: /usr/bin: Permission denied + + The simplest fix was to allow opening directories for reading. + While it's a bit silly to allow it solely for the error message, + it shouldn't make the sandbox significantly weaker. + + The single-file use case (like when called from GNU tar) is + still as strict as possible: all Landlock restrictions are + enabled before (de)compression starts. + + src/xz/sandbox.c | 15 ++++++++++++++- + 1 file changed, 14 insertions(+), 1 deletion(-) + +commit 120da10ae139ea52ca4275452adf8eda02d07cc8 +Author: Lasse Collin +Date: 2024-02-22 14:41:29 +0200 + + liblzma: Disable branchless C version in range decoder. + + Thanks to Sebastian Andrzej Siewior and Sam James for + benchmarking on various systems. + + src/liblzma/rangecoder/range_decoder.h | 13 ++++++++++--- + 1 file changed, 10 insertions(+), 3 deletions(-) + +commit 00440f52be9ac2c7438c7b0cb1082f12399632c6 +Author: Lasse Collin +Date: 2024-02-21 17:41:32 +0200 + + INSTALL: Clarify that --disable-assembler affects only 32-bit x86. + + INSTALL | 18 +++++++++--------- + 1 file changed, 9 insertions(+), 9 deletions(-) + +commit 11405be84ea294497e12d03d7219f607063f4a00 +Author: Lasse Collin +Date: 2024-02-19 18:41:37 +0200 + + Windows: build.bash: Include COPYING.0BSD in the package. + + windows/build.bash | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit c27cf64e3e27f4968431d65be7098a12a3a80d30 +Author: Lasse Collin +Date: 2024-02-18 17:59:46 +0200 + + Windows: build.bash: include liblzma-crt-mixing.txt in the package. + + windows/build.bash | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 8d38941baed05de4ff7cc775de14833108f62184 +Author: Lasse Collin +Date: 2024-02-18 17:17:43 +0200 + + Windows: Major update to Windows build instructions. + + INSTALL | 68 ++++----- + windows/INSTALL-MSVC.txt | 23 +-- + windows/INSTALL-MinGW-w64_with_Autotools.txt | 49 +++++++ + windows/INSTALL-MinGW-w64_with_CMake.txt | 203 +++++++++++++++++++++++++++ + windows/INSTALL-MinGW.txt | 138 ------------------ + windows/README-Windows.txt | 2 + + windows/build-with-cmake.bat | 35 +++++ + windows/liblzma-crt-mixing.txt | 70 +++++++++ + 8 files changed, 404 insertions(+), 184 deletions(-) + +commit 4b5b0d352348ff510ffb50a3b5b71788857d37a1 +Author: Lasse Collin +Date: 2024-02-18 15:15:04 +0200 + + Windows: Update windows/README-Windows.txt. + + It's for binary packages built with windows/build.bash. + + windows/README-Windows.txt | 104 ++++++++++++++++++--------------------------- + 1 file changed, 41 insertions(+), 63 deletions(-) + +commit 1ee716f74085223c8fbcae1d5a384e6bf53c0f6a +Author: Lasse Collin +Date: 2024-02-18 15:15:04 +0200 + + Windows: Update windows/build.bash. + + Support for the old MinGW was dropped. Only MinGW-w64 with GCC + is supported now. + + The script now supports also cross-compilation from GNU/Linux + (tests are not run). MSYS2 and also the old MSYS 1.0.11 work + for building on Windows. The i686 and x86_64 toolchains must + be in PATH to build both 32-bit and 64-bit versions. + + Parallel builds are done if "nproc" from GNU coreutils is available. + + MinGW-w64 runtime copyright information file was renamed from + COPYING-Windows.txt to COPYING.MinGW-w64-runtime.txt which + is the filename used by MinGW-w64 itself. Its existence + is now mandatory, it's checked at the beginning of the script. + + The file TODO is no longer copied to the package. + + windows/build.bash | 191 +++++++++++++++++++++++++++++++---------------------- + 1 file changed, 112 insertions(+), 79 deletions(-) + +commit 60462e42609a1d961868a1d1ebecc713c6d27e2e +Author: Jia Tan +Date: 2024-02-20 23:32:22 +0800 + + Translations: Update the Romanian man page translations. + + po4a/ro.po | 1715 +++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 875 insertions(+), 840 deletions(-) + +commit 10d733e5b8929c642e00891cfa9ead9c2cdd2e05 +Author: Jia Tan +Date: 2024-02-20 23:30:25 +0800 + + Translations: Update the Korean man page translations. + + po4a/ko.po | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 797a34b72ac6baff237d7a546fa941d8f78f2f62 +Author: Jia Tan +Date: 2024-02-20 21:03:53 +0800 + + Translations: Update the Spanish translation. + + po/es.po | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 5c3751d019f023e091df9a653e2bb1f6ea8b0d49 +Author: Jia Tan +Date: 2024-02-20 20:18:07 +0800 + + Translations: Update the Romanian translation. + + po/ro.po | 470 ++++++++++++++++++++++++++++++--------------------------------- + 1 file changed, 227 insertions(+), 243 deletions(-) + +commit e2d31154ecc750935436e8b62c6b073b2cfa84e3 +Author: Jia Tan +Date: 2024-02-20 20:15:50 +0800 + + Translations: Update the Croatian translation. + + po/hr.po | 648 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 355 insertions(+), 293 deletions(-) + +commit 704500f994d5ac271bfcfd592275c5a7da4dc8d2 +Author: Jia Tan +Date: 2024-02-20 20:05:44 +0800 + + Translations: Update the German man page translations. + + po4a/de.po | 1696 +++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 873 insertions(+), 823 deletions(-) + +commit 1cfd3dca3fef321b06db73c3c9e13f347c2e2f5f +Author: Jia Tan +Date: 2024-02-20 19:58:25 +0800 + + Translations: Update the German translation. + + po/de.po | 427 +++++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 225 insertions(+), 202 deletions(-) + +commit 28b9b3f16cc7c6e5b42e691994569c17f4561c9a +Author: Jia Tan +Date: 2024-02-20 19:56:52 +0800 + + Translations: Update the Hungarian translation. + + po/hu.po | 556 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 338 insertions(+), 218 deletions(-) + +commit 00b06cd0af6ad2ee93d3006bf80417db060c2b04 +Author: Lasse Collin +Date: 2024-02-19 16:48:05 +0200 + + CMake: Fix building of lzmainfo when translations are enabled. + + CMakeLists.txt | 2 ++ + 1 file changed, 2 insertions(+) + +commit b0d1422b6037bfea6f6723683bd82a8e6d77026c +Author: Lasse Collin +Date: 2024-02-19 13:38:42 +0200 + + CMake: Don't assume that -fvisibility=hidden is supported outside Windows. + + The original code was good enough for supporting GNU/Linux + and a few others but it wasn't very portable. + + CMake doesn't support Solaris Studio's -xldscope=hidden. + If it ever does, things should still work with this commit + as Solaris Studio supports not only its own __global but also + the GNU C __attribute__((visibility("default"))). Support for the + attribute was added in 2007 to Sun Studio 12 compiler version 5.9. + + CMakeLists.txt | 26 ++++++++++++++++++++++---- + 1 file changed, 22 insertions(+), 4 deletions(-) + +commit 2ced9d34bef4dce52ecbbf84d0903ab0aae1442c +Author: Lasse Collin +Date: 2024-02-19 12:20:59 +0200 + + CMake: Revise the component splitting. + + CMakeLists.txt | 57 +++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 31 insertions(+), 26 deletions(-) + +commit 426bdc709c169d39b31dec410016779de117ef69 +Author: Lasse Collin +Date: 2024-02-17 21:45:07 +0200 + + CMake: Update the main comment and document CMAKE_BUILD_TYPE=Release. + + CMakeLists.txt | 79 ++++++++++++++++++++++++++++++++++++++++++++++------------ + 1 file changed, 63 insertions(+), 16 deletions(-) + +commit 4430e075f7ccfc47972d6ca0aa1c3779fc265e10 +Author: Lasse Collin +Date: 2024-02-17 21:27:48 +0200 + + CMake: Use -O2 instead of -O3 in CMAKE_BUILD_TYPE=Release. + + -O3 doesn't seem useful for speed but it makes the code bigger. + CMake makes is difficult for users to simply override the + optimization level: CFLAGS / CMAKE_C_FLAGS aren't helpful because + they go before CMAKE_C_FLAGS_RELEASE. Of course, users can override + CMAKE_C_FLAGS_RELEASE directly but then they have to remember to + add also -DNDEBUG to disable assertions. + + This commit changes -O3 to -O2 in CMAKE_C_FLAGS_RELEASE if and only if + CMAKE_C_FLAGS_RELEASE cache variable doesn't already exist. So if + a custom value is passed on the command line (or reconfiguring an + already-configured build), the cache variable won't be modified. + + CMakeLists.txt | 19 +++++++++++++++++++ + 1 file changed, 19 insertions(+) + +commit 025eb6d7879e4c4e8cb29716b371e0f4c1aea660 +Author: Lasse Collin +Date: 2024-02-18 14:59:52 +0200 + + CMake: Handle symbol versioning on MicroBlaze specially. + + This is to match configure.ac. + + CMakeLists.txt | 23 +++++++++++++++++++---- + 1 file changed, 19 insertions(+), 4 deletions(-) + +commit 2edd1a35b2507d1ce68b52dbaebe23c4850a74ce +Author: Lasse Collin +Date: 2024-02-17 22:18:12 +0200 + + CMake: Keep build working even if lib/*.[ch] are removed. + + CMakeLists.txt | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit d753e2ce4715552884afadc4ed6fbf8ccca6efac +Author: Lasse Collin +Date: 2024-02-17 18:10:40 +0200 + + CMake: Install documentation. + + CMakeLists.txt | 32 ++++++++++++++++++++++++++++++++ + 1 file changed, 32 insertions(+) + +commit 7a0405bea9cb0df9318b70f779f82b2c473e98ac +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Bump maximum policy version to 3.28. + + CMP0154 doesn't affect us since we don't use FILE_SET. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit c2264ffbe3892d28930b89b0123efc369cabc143 +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Build lzmainfo. + + CMakeLists.txt | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 54 insertions(+) + +commit 998d0b29536094a89cf385a3b894e157db1ccefe +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Build lzmadec. + + CMakeLists.txt | 76 ++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 42 insertions(+), 34 deletions(-) + +commit 74e8bc7417a0f37ca7ed5ee0127d33c69b3100b9 +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Add test_scripts.sh to the tests. + + In contrast to Automake, skipping of this test when decoders + are disabled is handled at CMake side instead of test_scripts.sh + because CMake-build doesn't create config.h. + + CMakeLists.txt | 14 ++++++++++++++ + tests/test_scripts.sh | 13 ++++++++----- + 2 files changed, 22 insertions(+), 5 deletions(-) + +commit 4808f238a731befcd46c2117c62a1caaf4403989 +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Install scripts. + + Compared to the Autotools-based build, this has simpler handling + for the shell (@POSIX_SHELL@) and extra PATH entry for the scripts + (configure has --enable-path-for-scripts=PREFIX). The simpler + metho should be enough for non-ancient systems and Solaris. + + CMakeLists.txt | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++- + 1 file changed, 82 insertions(+), 1 deletion(-) + +commit 3462362ebd94d835c664e94ad8f414cfe7590ca7 +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + Scripts: Use @PACKAGE_VERSION@ instead of @VERSION@. + + PACKAGE_VERSION was already used in liblzma.pc.in. + This way only one version @foo@ is used. + + src/scripts/xzdiff.in | 2 +- + src/scripts/xzgrep.in | 2 +- + src/scripts/xzless.in | 2 +- + src/scripts/xzmore.in | 2 +- + 4 files changed, 4 insertions(+), 4 deletions(-) + +commit 67610c245ba6c68cf65991693bab9312b7dc987b +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Simplify symlink creation and install translated man pages. + + It helps that cmake_install.cmake doesn't parallelize installation + so symlinks can be created so that the target is always known to + exist (a requirement on Windows in some cases). + + This bumps the minimum CMake version from 3.13 to 3.14 to use + file(CREATE_LINK ...). It could be made to work on 3.13 by + calling "cmake -E create_symlink" but it's uglier code and + slower in "make install". 3.14 should be a reasonable version + to require nowadays, especially since the Autotools build + is still the primary build system for most OSes. + + CMakeLists.txt | 195 +++++++++++++++++++++++++++++---------------------------- + 1 file changed, 98 insertions(+), 97 deletions(-) + +commit 50cc1d8a5a8154428bf240c7e4972e32b17d99bf +Author: Lasse Collin +Date: 2024-02-17 15:35:35 +0200 + + CMake: Add support for building and installing xz with translations. + + If gettext tools are available, the .po files listed in po/LINGUAS + are converted using msgfmt. This allows building with translations + directly from xz.git without Autotools. + + If gettext tools aren't available, the Autotools-created .gmo files + in the "po" directory will be used. This allows CMake-based build + to use translations from Autotools-generated tarball. + + If translation support is found (Intl_FOUND) but both the + gettext tools and the pre-generated .gmo files are missing, + then "make" will fail. + + CMakeLists.txt | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- + 1 file changed, 66 insertions(+), 2 deletions(-) + +commit 746c471643009947f94a3494a1148f74c7381b56 +Author: Lasse Collin +Date: 2024-02-19 11:58:33 +0200 + + liblzma: Remove commented-out code. + + src/liblzma/rangecoder/range_decoder.h | 3 --- + 1 file changed, 3 deletions(-) + +commit 4ce300ce0884c6e552de2af9ae8050b47b01f0e7 +Author: Lasse Collin +Date: 2024-02-17 23:07:35 +0200 + + xz: Delete old commented-out code. + + src/xz/message.c | 19 ------------------- + 1 file changed, 19 deletions(-) + +commit cae9a5e0bf422e6c5e64180805904f7ed02dc3aa +Author: Lasse Collin +Date: 2024-02-17 23:07:35 +0200 + + xz: Use stricter pledge(2) and Landlock sandbox. + + This makes these sandboxing methods stricter when no files are + created or deleted. That is, it's a middle ground between the + initial sandbox and the strictest single-file-to-stdout sandbox: + this allows opening files for reading but output has to go to stdout. + + src/xz/main.c | 46 +++++++++++++++++++++++++++++++++------------- + src/xz/sandbox.c | 32 ++++++++++++++++++++++++++++++++ + src/xz/sandbox.h | 4 ++++ + 3 files changed, 69 insertions(+), 13 deletions(-) + +commit 02e3505991233901575b7eabc06b2c6c62a96899 +Author: Lasse Collin +Date: 2024-02-17 23:07:35 +0200 + + xz: Support Landlock ABI version 4. + + Linux 6.7 added support for ABI version 4 which restricts + TCP connections which xz won't need and thus those can be + forbidden now. Since the ABI version is handled at runtime, + supporting version 4 won't cause any compatibility issues. + + Note that new enough kernel headers are required to get + version 4 support enabled at build time. + + src/xz/sandbox.c | 25 ++++++++++++++++++++----- + 1 file changed, 20 insertions(+), 5 deletions(-) + +commit 374868d81d473ab56556a1cfd6b1b36a1fab348b +Author: Lasse Collin +Date: 2024-02-17 23:07:35 +0200 + + xz: Move sandboxing code to sandbox.c and improve Landlock sandbox. + + Landlock is now always used just like pledge(2) is: first in more + permissive mode and later (under certain common conditions) in + a strict mode that doesn't allow opening more files. + + I put pledge(2) first in sandbox.c because it's the simplest API + to use and still somewhat fine-grained for basic applications. + So it's the simplest thing to understand for anyone reading sandbox.c. + + CMakeLists.txt | 2 + + src/xz/Makefile.am | 2 + + src/xz/file_io.c | 170 +----------------------------- + src/xz/file_io.h | 6 -- + src/xz/main.c | 50 +++------ + src/xz/private.h | 6 +- + src/xz/sandbox.c | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++++ + src/xz/sandbox.h | 39 +++++++ + 8 files changed, 357 insertions(+), 213 deletions(-) + +commit 7312dfbb02197c7f990c7a3cefd027a9387d1473 +Author: Lasse Collin +Date: 2024-02-17 23:07:35 +0200 + + xz: Tweak comments. + + src/xz/main.c | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit c701a5909ad9882469fbab4fab5d2d5556d3ba78 +Author: Lasse Collin +Date: 2024-02-17 23:07:35 +0200 + + xz: Fix message_init() description. + + Also explicitly initialize progress_automatic to make it clear + that it can be read before message_init() sets it. Static variable + was initialized to false by default already so this is only for + clarity. + + src/xz/main.c | 3 ++- + src/xz/message.c | 2 +- + src/xz/message.h | 5 ++++- + 3 files changed, 7 insertions(+), 3 deletions(-) + +commit 9466306719f3b76e92fac4e55fbfd89ec92295fa +Author: Lasse Collin +Date: 2024-02-17 19:35:47 +0200 + + Build: Makefile.am: Sort EXTRA_DIST. + + Dirs first, then files in case-sensitive ASCII order. + + Makefile.am | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +commit f3440e78c9517db75bfa52e1a378fad60b073bbe +Author: Lasse Collin +Date: 2024-02-17 19:25:05 +0200 + + Build: Don't install TODO. + + Makefile.am | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit a7a3b62e2ab03c82b2bd5c78da1d1fb8b8490381 +Author: Jia Tan +Date: 2024-02-18 01:09:11 +0800 + + Translations: Update the Korean man page translations. + + po4a/ko.po | 1707 +++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 871 insertions(+), 836 deletions(-) + +commit 9b315db2d5e74700f3dc0755eb86c27947c0b393 +Author: Jia Tan +Date: 2024-02-18 01:08:32 +0800 + + Translations: Update the Korean translation. + + po/ko.po | 423 +++++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 223 insertions(+), 200 deletions(-) + +commit 56246607dff177b0410d140fcca4a42c865723dc +Author: Lasse Collin +Date: 2024-02-17 16:23:14 +0200 + + Build: Install translated lzmainfo man pages. + + All other translated man pages were being installed but + lzmainfo had been forgotten. + + src/lzmainfo/Makefile.am | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +commit f1d6b88aefcced538403c5c2606ba57065b16e70 +Author: Lasse Collin +Date: 2024-02-17 16:01:32 +0200 + + liblzma: Avoid implementation-defined behavior in the RISC-V filter. + + GCC docs promise that it works and a few other compilers do + too. Clang/LLVM is documented source code only but unsurprisingly + it behaves the same as others on x86-64 at least. But the + certainly-portable way is good enough here so use that. + + src/liblzma/simple/riscv.c | 30 ++++++++++++++++++++++-------- + 1 file changed, 22 insertions(+), 8 deletions(-) + +commit 843ddc5f617b91ae132d6bab0f2f2d9c9fcd214a +Author: Lasse Collin +Date: 2024-02-17 15:48:28 +0200 + + liblzma: Wrap a line exceeding 80 chars. + + src/liblzma/rangecoder/range_decoder.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit e9053c907250c70d98b319d95fa54cb94fc76869 +Author: Sebastian Andrzej Siewior +Date: 2024-02-16 21:50:15 +0100 + + liblzma/rangecoder: Exclude x32 from the x86-64 optimisation. + + The x32 port has a x86-64 ABI in term of all registers but uses only + 32bit pointer like x86-32. The assembly optimisation fails to compile on + x32. Given the state of x32 I suggest to exclude it from the + optimisation rather than trying to fix it. + + Signed-off-by: Sebastian Andrzej Siewior + + src/liblzma/rangecoder/range_decoder.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 3d198fb13b87f8803442e5799d465f7434a70555 +Author: Jia Tan +Date: 2024-02-17 21:05:07 +0800 + + Translations: Update the Spanish translation. + + po/es.po | 427 +++++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 226 insertions(+), 201 deletions(-) + +commit cf278bfe60a25b54b3786f06503bc61272970820 +Author: Jia Tan +Date: 2024-02-17 20:43:29 +0800 + + Translations: Update the Swedish translation. + + po/sv.po | 434 +++++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 230 insertions(+), 204 deletions(-) + +commit b0f1a41be50560cc6cb528e8e96b02b2067c52c2 +Author: Jia Tan +Date: 2024-02-17 20:41:38 +0800 + + Translations: Update the Polish translation. + + po/pl.po | 424 +++++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 224 insertions(+), 200 deletions(-) + +commit d74ed48b30c631b6a4c7e7858b06828293bf8520 +Author: Jia Tan +Date: 2024-02-17 20:41:02 +0800 + + Translations: Update the Ukrainian translation. + + po/uk.po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 711e22d5c5f3bac39ac904efb3ede874a66e2045 +Author: Lasse Collin +Date: 2024-02-16 17:53:34 +0200 + + Translations: Use the same sentence in xz.pot-header that the TP uses. + + po/xz.pot-header | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit fb5f6aaf18584672d0fee5dbe41fd30fc6bf5422 +Author: Jia Tan +Date: 2024-02-16 22:53:46 +0800 + + Fix typos discovered by codespell. + + AUTHORS | 2 +- + NEWS | 2 +- + src/liblzma/rangecoder/range_decoder.h | 4 ++-- + 3 files changed, 4 insertions(+), 4 deletions(-) + +commit c64723bbb094e29b4edd98f6fcce866e1b569b42 +Author: Jia Tan +Date: 2024-02-16 22:52:41 +0800 + + Translations: Update the Ukrainian man page translations. + + po4a/uk.po | 1710 +++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 873 insertions(+), 837 deletions(-) + +commit 2895195ed0f68b245c7bd568c126ba6e685fa1d6 +Author: Jia Tan +Date: 2024-02-16 22:51:04 +0800 + + Translations: Update the Ukrainian translation. + + po/uk.po | 466 ++++++++++++++++++++++++++++++--------------------------------- + 1 file changed, 225 insertions(+), 241 deletions(-) + +commit 4c20781f4c8f04879b64d631a4f44b4909147bde +Author: Lasse Collin +Date: 2024-02-15 22:32:52 +0200 + + Translations: Omit the generic copyright line from man page headers. + + po4a/update-po | 1 + + 1 file changed, 1 insertion(+) + +commit 4323bc3e0c1e1d2037d5e670a3bf6633e8a3031e +Author: Jia Tan +Date: 2024-02-15 22:26:43 +0800 + + Update m4/.gitignore. + + m4/.gitignore | 1 + + 1 file changed, 1 insertion(+) + +commit 5394a1665b7a108a54cb8b4ef3ebe59d3dbcca3a +Author: Lasse Collin +Date: 2024-02-14 21:11:49 +0200 + + Tests: tuktest.h: Treat Clang separately from GCC. + + Don't assume that Clang defines __GNUC__ as the extensions + are available in clang-cl as well (and possibly in some other + Clang variants?). + + tests/tuktest.h | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit cce7330b9f23485a0879422e0c3395a7065439ac +Author: Lasse Collin +Date: 2024-02-14 21:11:03 +0200 + + Tests: tuktest.h: Add a missing word to a comment. + + tests/tuktest.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 5dd8fc9452a3373cedc27379067ce638f992c741 +Author: Lasse Collin +Date: 2024-02-14 21:10:10 +0200 + + Tests: tuktest.h: Fix the comment about STest. + + tests/tuktest.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 6f1790254a03c5edf0f2976f773220f070450acd +Author: Jia Tan +Date: 2024-02-15 01:53:40 +0800 + + Bump version for 5.5.2beta. + + src/liblzma/api/lzma/version.h | 4 ++-- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 3 files changed, 4 insertions(+), 4 deletions(-) + +commit 924fdeedf48113fb1e0646d86bd89a356d21a055 +Author: Lasse Collin +Date: 2024-02-14 19:46:11 +0200 + + liblzma: Fix validate_map.sh. + + Adding the SPDX license identifier changed the line numbers. + + src/liblzma/validate_map.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 22140a2df6161b0110e6b4afa5ea0a07c5b60b01 +Author: Lasse Collin +Date: 2024-02-14 19:38:34 +0200 + + Build: Start the generated ChangeLog from around 5.4.0 instead of 5.2.0. + + Makefile.am | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 0b8cefa136c21d403a01b78517f4decb50172bdb +Author: Lasse Collin +Date: 2024-02-14 19:27:46 +0200 + + Fixed NEWS for 5.5.2beta. + + NEWS | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit a4557bad96361d93ea171ed859ac5a696fca824f +Author: Lasse Collin +Date: 2024-02-14 19:21:45 +0200 + + liblzma: Silence warnings in --enable-small build. + + src/liblzma/lzma/lzma_decoder.c | 2 ++ + src/liblzma/rangecoder/range_decoder.h | 1 + + 2 files changed, 3 insertions(+) + +commit 38edf473236d00b3e100dc4c4f0bf43a4993fed2 +Author: Lasse Collin +Date: 2024-02-14 19:15:58 +0200 + + Build: Install COPYING.0BSD as part of docs. + + Makefile.am | 1 + + 1 file changed, 1 insertion(+) + +commit b74e10bd839bcdc239afb5300ffaee195f34c217 +Author: Lasse Collin +Date: 2024-02-14 19:14:05 +0200 + + Docs: List COPYING.0BSD in README. + + README | 1 + + 1 file changed, 1 insertion(+) + +commit dfdb60ffe933a1f1497d300dbb4513ed17ec6f0e +Author: Lasse Collin +Date: 2024-02-14 19:11:48 +0200 + + Docs: Include doc/examples/11_file_info.c in tarballs. + + It was added in 2017 in c2e29f06a7d1e3ba242ac2fafc69f5d6e92f62cd + but it never got into any release tarballs because it was + forgotten to be added to Makefile.am. + + Makefile.am | 1 + + 1 file changed, 1 insertion(+) + +commit 160b6862646d95dfdbd73ab7f1031ede0f54992d +Author: Lasse Collin +Date: 2024-02-14 19:05:58 +0200 + + liblzma: Silence a warning. + + src/liblzma/rangecoder/range_decoder.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit eeedd4d0925ea417add04ceb42a6c0829244b50c +Author: Lasse Collin +Date: 2024-02-14 18:32:27 +0200 + + Add NEWS for 5.5.2beta. + + NEWS | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 60 insertions(+) + +commit 8af7db854f903068d72a9a0d21103cb0c5027fa8 +Author: Lasse Collin +Date: 2024-02-13 14:32:47 +0200 + + xz: Mention lzmainfo if trying to use 'lzma --list'. + + This kind of fixes the problem reported here: + https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/1291020 + + src/xz/list.c | 16 ++++++++++++++-- + 1 file changed, 14 insertions(+), 2 deletions(-) + +commit 0668907ff736e4cd16738c10d39a2bc9e851aefb +Author: Lasse Collin +Date: 2024-02-14 14:58:36 +0200 + + liblzma: Add comments. + + src/liblzma/lzma/lzma_decoder.c | 9 +++++++++ + src/liblzma/rangecoder/range_decoder.h | 11 +++++++++-- + 2 files changed, 18 insertions(+), 2 deletions(-) + +commit 109f1913d4824c8214d5bbd38ebebf62c37572da +Author: Lasse Collin +Date: 2024-02-13 17:00:17 +0200 + + Scripts: Add lz4 support to xzgrep and xzdiff. + + src/scripts/xzdiff.1 | 8 +++++--- + src/scripts/xzdiff.in | 14 +++++++++----- + src/scripts/xzgrep.1 | 6 ++++-- + src/scripts/xzgrep.in | 1 + + 4 files changed, 19 insertions(+), 10 deletions(-) + +commit de55485cb23af56c5adbe3239b935c957ff8ac4f +Author: Lasse Collin +Date: 2024-02-13 14:05:13 +0200 + + liblzma: Choose the range decoder variants using a bitmask macro. + + src/liblzma/rangecoder/range_decoder.h | 64 ++++++++++++++++++++++++++++------ + 1 file changed, 53 insertions(+), 11 deletions(-) + +commit 0709c2b2d7c1d8f437b003f691880fd7810e5be5 +Author: Lasse Collin +Date: 2024-02-13 11:38:10 +0200 + + xz: Fix outdated threading related info on the man page. + + src/xz/xz.1 | 22 ++++++++++++++-------- + 1 file changed, 14 insertions(+), 8 deletions(-) + +commit 3182a330c1512cc1f5c87b5c5a272578e60a5158 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Range decoder: Add x86-64 inline assembly. + + It's compatible with GCC and Clang. + + src/liblzma/rangecoder/range_decoder.h | 491 +++++++++++++++++++++++++++++++++ + 1 file changed, 491 insertions(+) + +commit cba2edc991dffba7cd4891dbc1bd26cb950cf053 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Range decoder: Add branchless C code. + + It's used only for basic bittrees and fixed-size reverse bittree + because those showed a clear benefit on x86-64 with GCC and Clang. + The other methods were more mixed and thus are commented out but + they should be tested on other archs. + + src/liblzma/rangecoder/range_decoder.h | 76 ++++++++++++++++++++++++++++++++++ + 1 file changed, 76 insertions(+) + +commit e290a72d6dee71faf3a90c9678b2f730083666a7 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Clarify a comment. + + src/liblzma/lzma/lzma_decoder.c | 9 ++++++--- + 1 file changed, 6 insertions(+), 3 deletions(-) + +commit 5e04706b91ca90d6befd4da24a588a55e631d4a9 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: LZMA decoder: Optimize loop comparison. + + But now it needs one more local variable. + + src/liblzma/lzma/lzma_decoder.c | 5 ++--- + src/liblzma/rangecoder/range_decoder.h | 10 +++++++++- + 2 files changed, 11 insertions(+), 4 deletions(-) + +commit 88276f9f2cb4871c7eb86952d93d07c1cf6caa66 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Optimize literal_subcoder() macro slightly. + + src/liblzma/lzma/lzma_common.h | 22 ++++++++++++---------- + src/liblzma/lzma/lzma_decoder.c | 12 ++++++------ + src/liblzma/lzma/lzma_encoder.c | 6 +++--- + src/liblzma/lzma/lzma_encoder_optimum_normal.c | 2 +- + src/liblzma/lzma/lzma_encoder_private.h | 4 ++-- + 5 files changed, 24 insertions(+), 22 deletions(-) + +commit 5938f6de4d8ec9656776cd69e78ddfd6c3ad84e5 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: LZ decoder: Add unlikely(). + + src/liblzma/lz/lz_decoder.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 9c252e3ed086c6b72590b2531586c42596d4a9d9 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: LZ decoder: Remove a useless unlikely(). + + src/liblzma/lz/lz_decoder.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f3872a59475456c5d365cad9f1c5be514cfa54b5 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Optimize LZ decoder slightly. + + Now extra buffer space is reserved so that repeating bytes for + any single match will never need to copy from two places (both + the beginning and the end of the buffer). This simplifies + dict_repeat() and helps a little with speed. + + This seems to reduce .lzma decompression time about 2 %, so + with .xz and CRC it could be slightly less. The small things + add up still. + + src/liblzma/lz/lz_decoder.c | 43 ++++++++++++----- + src/liblzma/lz/lz_decoder.h | 101 +++++++++++++++++++++------------------- + src/liblzma/lzma/lzma_decoder.c | 4 +- + 3 files changed, 88 insertions(+), 60 deletions(-) + +commit eb518446e578acf079abae5f1ce28db7b6e59bc1 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: LZMA decoder: Get rid of next_state[]. + + It's not completely obvious if this is better in the decoder. + It should be good if compiler can avoid creating a branch + (like using CMOV on x86). + + This also makes lzma_encoder.c use the new macros. + + src/liblzma/lzma/lzma_common.h | 14 ++++++++++++++ + src/liblzma/lzma/lzma_decoder.c | 30 ++++++++---------------------- + src/liblzma/lzma/lzma_encoder.c | 4 ++-- + 3 files changed, 24 insertions(+), 24 deletions(-) + +commit e0c0ee475c0800c08291ae45e0d66aa00d5ce604 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: LZMA decoder improvements. + + This adds macros for bittree decoding which prepares the code + for alternative C versions and inline assembly. + + src/liblzma/lzma/lzma_decoder.c | 264 ++++++++++----------------------- + src/liblzma/rangecoder/range_common.h | 4 + + src/liblzma/rangecoder/range_decoder.h | 142 ++++++++++++++++-- + 3 files changed, 210 insertions(+), 200 deletions(-) + +commit de5c5e417645ad8906ef914bc059d08c1462fc29 +Author: Jia Tan +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Creates Non-resumable and Resumable modes for lzma_decoder. + + The new decoder resumes the first decoder loop in the Resumable mode. + Then, the code executes in Non-resumable mode until it detects that it + cannot guarantee to have enough input/output to decode another symbol. + + The Resumable mode is how the decoder has always worked. Before decoding + every input bit, it checks if there is enough space and will save its + location to be resumed later. When the decoder has more input/output, + it jumps back to the correct sequence in the Resumable mode code. + + When the input/output buffers are large, the Resumable mode is much + slower than the Non-resumable because it has more branches and is harder + for the compiler to optimize since it is in a large switch block. + + Early benchmarking shows significant time improvement (8-10% on gcc and + clang x86) by using the Non-resumable code as much as possible. + + src/liblzma/lz/lz_decoder.h | 14 +- + src/liblzma/lzma/lzma_decoder.c | 720 ++++++++++++++++++++++++++++------------ + 2 files changed, 521 insertions(+), 213 deletions(-) + +commit e446ab7a18abfde18f8d1cf02a914df72b1370e3 +Author: Jia Tan +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Creates separate "safe" range decoder mode. + + The new "safe" range decoder mode is the same as old range decoder, but + now the default behavior of the range decoder will not check if there is + enough input or output to complete the operation. When the buffers are + close to fully consumed, the "safe" operations must be used instead. This + will improve speed because it will reduce the number of branches needed + for most of the range decoder operations. + + src/liblzma/lzma/lzma_decoder.c | 108 ++++++++------------------------- + src/liblzma/rangecoder/range_decoder.h | 77 +++++++++++++++++------ + 2 files changed, 82 insertions(+), 103 deletions(-) + +commit 7f6d9ca329ff3e01d4b0be7366eb4f5c93da41b9 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + doxygen/footer.html: Add missing closing tags and don't open a new tab. + + The footer template from Doxygen has the closing + as Doxygen doesn't add them otherwise. + + target="_blank" was omitted as it's not useful here but + it can be slightly annoying as one cannot just go back + in the browser history. + + Since the footer links to the license file in the same + directory and not to CC website, the rel attributes + can be omitted. + + doxygen/footer.html | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 26d1527d34d52b0f5d632d4fb636fb33d0867e92 +Author: Lasse Collin +Date: 2024-02-13 13:19:10 +0200 + + Tweak the expressions in AUTHORS. + + AUTHORS | 31 +++++++++++++++++++++++-------- + 1 file changed, 23 insertions(+), 8 deletions(-) + +commit d231d56580175fa040fdd3c6207a58243ce6217b +Author: Lasse Collin +Date: 2024-02-13 13:07:33 +0200 + + Translations: Add the man page translators into man page header comment. + + It looked odd to only have the original English authors listed + in the header comments of the translated files. + + po4a/.gitignore | 1 + + po4a/po4a.conf | 14 +++++++------- + po4a/update-po | 18 ++++++++++++++++++ + 3 files changed, 26 insertions(+), 7 deletions(-) + +commit 6d35fcb936474fca1acaebfd9502c097b6fde88e +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Translations: Translate also messages of lzmainfo. + + lzmainfo has had translation support since 2009 at least but + it was never added to po/POTFILES.in so the messages weren't + translated. It's a very rarely needed tool so it's not too bad. + + This also adds src/xz/mytime.c to po/POTFILES.in although there + are no translatable strings. It's simpler this way so that it + won't be forgotten if strings were ever added to that file. + + po/POTFILES.in | 2 ++ + 1 file changed, 2 insertions(+) + +commit a9f369dd54b05f9ac4e00ead9d765d04fc259868 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Translations: Add custom .pot header with SPDX license identifier. + + The same is used for both po/xz.pot and po4a/xz-man.pot. + + Makefile.am | 1 + + po/xz.pot-header | 7 +++++++ + po4a/update-po | 8 ++++++++ + 3 files changed, 16 insertions(+) + +commit 469cd6653bb96e83c5cf1031c204d34566b15f44 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Translations: po4a/update-po: Add copyright notice to xz-man.pot. + + All man pages are under 0BSD now so this is simple now. + + po4a/update-po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 28ce45e38fbed4b5f54f2013e38dab47d22bf699 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Update COPYING about the man pages of the scripts. + + COPYING | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit e48287bf51afd5184ea74de1dcade9e153f873f7 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + xzdiff, xzgrep, and xzmore: Rewrite the man pages. + + The main reason is a kind of silly one: + + xz-man.pot contains strings from all man pages in XZ Utils. + The man pages of xzdiff, xzgrep, and xzmore were under GPLv2 + and the rest under 0BSD. Thus xz-man.pot contained strings + under two licences. po4a creates the translated man pages + from the combined 0BSD+GPLv2 xz-man.pot. + + I haven't liked this mixing in xz-man.pot but the + Translation Project requires that all man pages must be + in the same .pot file. So a separate xz-man-gpl.pot + wasn't an option. + + Since these man pages are short, rewriting them was quick enough. + Now xz-man.pot is entirely under 0BSD and marking the per-file + licenses is simpler. + + As a bonus, some wording hopefully is now slightly better + although it's perhaps a matter of taste. + + NOTE: In xzgrep.1, the EXIT STATUS section was written by me + in the commit d796b6d7fdb8b7238b277056cf9146cce25db604 so that's + why that section could be taken as is from the old xzgrep.1. + + src/scripts/xzdiff.1 | 94 ++++++++++++++++++++++++----------------- + src/scripts/xzgrep.1 | 116 ++++++++++++++++++++++++++++++++------------------- + src/scripts/xzmore.1 | 79 ++++++++++++++++++++--------------- + 3 files changed, 173 insertions(+), 116 deletions(-) + +commit 3e551b111b8ae8150f1a1040364dbafc034f22be +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + xzless: Update man page slightly. + + The xz tool can decompress three file formats and xzless + has always supported uncompressed files too. + + src/scripts/xzless.1 | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +commit 40f36da2262d13d6e1ba8449caa855512ae626d7 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Translations: Change po/Makevars to add a copyright notice to po/xz.pot. + + po/Makevars | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 24192854e2ea5c06997431a98bda3c36c5da1497 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Translations: Update po/Makevars to use the template from gettext 0.22.4. + + Also add SPDX license identifier now that there is a known license. + + po/Makevars | 51 ++++++++++++++++++++++++++++++++++++++++++++++----- + 1 file changed, 46 insertions(+), 5 deletions(-) + +commit b94154957370116480b43bcabca25fc52deb9853 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Include the SPDX license identifier 0BSD to generated files. + + Perhaps the generated files aren't even copyrightable but + using the same license for them as for the rest of the liblzma + keeps things more consistent for tools that look for license info. + + src/liblzma/check/crc32_table_be.h | 4 +++- + src/liblzma/check/crc32_table_le.h | 4 +++- + src/liblzma/check/crc32_tablegen.c | 16 ++++++++++------ + src/liblzma/check/crc64_table_be.h | 4 +++- + src/liblzma/check/crc64_table_le.h | 4 +++- + src/liblzma/check/crc64_tablegen.c | 8 +++++--- + src/liblzma/lz/lz_encoder_hash_table.h | 4 +++- + src/liblzma/lzma/fastpos_table.c | 4 +++- + src/liblzma/lzma/fastpos_tablegen.c | 12 +++++++----- + src/liblzma/rangecoder/price_table.c | 4 +++- + src/liblzma/rangecoder/price_tablegen.c | 12 +++++++----- + 11 files changed, 50 insertions(+), 26 deletions(-) + +commit 8e4ec794836bc1701d8c9bd5e347b8ce8cc5bbb4 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + liblzma: Fix compilation of price_tablegen.c. + + It is built and run only manually so this didn't matter + unless one wanted to regenerate the price_table.c. + + src/liblzma/rangecoder/price_tablegen.c | 5 +++++ + src/liblzma/rangecoder/range_common.h | 5 ++++- + 2 files changed, 9 insertions(+), 1 deletion(-) + +commit e99bff3ffbcdf2634fd5bd13887627ec7dbfecaf +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Add SPDX license identifiers to GPL, LGPL, and FSFULLR files. + + extra/scanlzma/scanlzma.c | 2 ++ + lib/Makefile.am | 2 ++ + lib/getopt-cdefs.h | 2 ++ + lib/getopt-core.h | 2 ++ + lib/getopt-ext.h | 2 ++ + lib/getopt-pfx-core.h | 2 ++ + lib/getopt-pfx-ext.h | 2 ++ + lib/getopt.c | 2 ++ + lib/getopt.in.h | 2 ++ + lib/getopt1.c | 2 ++ + lib/getopt_int.h | 2 ++ + m4/ax_pthread.m4 | 2 ++ + m4/getopt.m4 | 2 ++ + m4/posix-shell.m4 | 2 ++ + m4/visibility.m4 | 2 ++ + src/scripts/xzdiff.1 | 3 +-- + src/scripts/xzdiff.in | 1 + + src/scripts/xzgrep.1 | 3 +-- + src/scripts/xzgrep.in | 1 + + src/scripts/xzless.in | 1 + + src/scripts/xzmore.1 | 3 +-- + src/scripts/xzmore.in | 1 + + 22 files changed, 37 insertions(+), 6 deletions(-) + +commit 22af94128b89a131f5e58ae69bee5e50227c15da +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Add SPDX license identifier into 0BSD source code files. + + .github/workflows/ci.yml | 2 ++ + .github/workflows/windows-ci.yml | 2 ++ + CMakeLists.txt | 2 ++ + Makefile.am | 3 +-- + autogen.sh | 1 + + build-aux/ci_build.sh | 3 ++- + build-aux/manconv.sh | 3 ++- + build-aux/version.sh | 3 ++- + cmake/remove-ordinals.cmake | 2 ++ + cmake/tuklib_common.cmake | 4 ++++ + cmake/tuklib_cpucores.cmake | 4 ++++ + cmake/tuklib_integer.cmake | 4 ++++ + cmake/tuklib_large_file_support.cmake | 4 ++++ + cmake/tuklib_mbstr.cmake | 4 ++++ + cmake/tuklib_physmem.cmake | 4 ++++ + cmake/tuklib_progname.cmake | 4 ++++ + configure.ac | 4 +++- + debug/Makefile.am | 3 +-- + debug/crc32.c | 2 ++ + debug/full_flush.c | 2 ++ + debug/hex2bin.c | 2 ++ + debug/known_sizes.c | 2 ++ + debug/memusage.c | 2 ++ + debug/repeat.c | 2 ++ + debug/sync_flush.c | 2 ++ + debug/translation.bash | 1 + + doc/examples/01_compress_easy.c | 2 ++ + doc/examples/02_decompress.c | 2 ++ + doc/examples/03_compress_custom.c | 2 ++ + doc/examples/04_compress_easy_mt.c | 2 ++ + doc/examples/11_file_info.c | 2 ++ + doc/examples/Makefile | 3 +-- + dos/Makefile | 2 ++ + dos/config.h | 2 ++ + doxygen/update-doxygen | 3 ++- + extra/7z2lzma/7z2lzma.bash | 3 ++- + m4/tuklib_common.m4 | 8 ++++++-- + m4/tuklib_cpucores.m4 | 8 ++++++-- + m4/tuklib_integer.m4 | 8 ++++++-- + m4/tuklib_mbstr.m4 | 8 ++++++-- + m4/tuklib_physmem.m4 | 8 ++++++-- + m4/tuklib_progname.m4 | 8 ++++++-- + po/POTFILES.in | 2 ++ + po4a/po4a.conf | 2 ++ + po4a/update-po | 3 ++- + src/Makefile.am | 3 +-- + src/common/common_w32res.rc | 2 ++ + src/common/mythread.h | 2 ++ + src/common/sysdefs.h | 2 ++ + src/common/tuklib_common.h | 2 ++ + src/common/tuklib_config.h | 2 ++ + src/common/tuklib_cpucores.c | 2 ++ + src/common/tuklib_cpucores.h | 2 ++ + src/common/tuklib_exit.c | 2 ++ + src/common/tuklib_exit.h | 2 ++ + src/common/tuklib_gettext.h | 2 ++ + src/common/tuklib_integer.h | 2 ++ + src/common/tuklib_mbstr.h | 2 ++ + src/common/tuklib_mbstr_fw.c | 2 ++ + src/common/tuklib_mbstr_width.c | 2 ++ + src/common/tuklib_open_stdxxx.c | 2 ++ + src/common/tuklib_open_stdxxx.h | 2 ++ + src/common/tuklib_physmem.c | 2 ++ + src/common/tuklib_physmem.h | 2 ++ + src/common/tuklib_progname.c | 2 ++ + src/common/tuklib_progname.h | 2 ++ + src/liblzma/Makefile.am | 3 +-- + src/liblzma/api/Makefile.am | 3 +-- + src/liblzma/api/lzma.h | 2 ++ + src/liblzma/api/lzma/base.h | 2 ++ + src/liblzma/api/lzma/bcj.h | 2 ++ + src/liblzma/api/lzma/block.h | 2 ++ + src/liblzma/api/lzma/check.h | 2 ++ + src/liblzma/api/lzma/container.h | 2 ++ + src/liblzma/api/lzma/delta.h | 2 ++ + src/liblzma/api/lzma/filter.h | 2 ++ + src/liblzma/api/lzma/hardware.h | 2 ++ + src/liblzma/api/lzma/index.h | 2 ++ + src/liblzma/api/lzma/index_hash.h | 2 ++ + src/liblzma/api/lzma/lzma12.h | 2 ++ + src/liblzma/api/lzma/stream_flags.h | 2 ++ + src/liblzma/api/lzma/version.h | 2 ++ + src/liblzma/api/lzma/vli.h | 2 ++ + src/liblzma/check/Makefile.inc | 4 ++-- + src/liblzma/check/check.c | 2 ++ + src/liblzma/check/check.h | 2 ++ + src/liblzma/check/crc32_arm64.h | 2 ++ + src/liblzma/check/crc32_fast.c | 2 ++ + src/liblzma/check/crc32_small.c | 2 ++ + src/liblzma/check/crc32_table.c | 2 ++ + src/liblzma/check/crc32_tablegen.c | 2 ++ + src/liblzma/check/crc32_x86.S | 2 ++ + src/liblzma/check/crc64_fast.c | 2 ++ + src/liblzma/check/crc64_small.c | 2 ++ + src/liblzma/check/crc64_table.c | 2 ++ + src/liblzma/check/crc64_tablegen.c | 2 ++ + src/liblzma/check/crc64_x86.S | 2 ++ + src/liblzma/check/crc_common.h | 2 ++ + src/liblzma/check/crc_x86_clmul.h | 2 ++ + src/liblzma/check/sha256.c | 2 ++ + src/liblzma/common/Makefile.inc | 3 +-- + src/liblzma/common/alone_decoder.c | 2 ++ + src/liblzma/common/alone_decoder.h | 2 ++ + src/liblzma/common/alone_encoder.c | 2 ++ + src/liblzma/common/auto_decoder.c | 2 ++ + src/liblzma/common/block_buffer_decoder.c | 2 ++ + src/liblzma/common/block_buffer_encoder.c | 2 ++ + src/liblzma/common/block_buffer_encoder.h | 2 ++ + src/liblzma/common/block_decoder.c | 2 ++ + src/liblzma/common/block_decoder.h | 2 ++ + src/liblzma/common/block_encoder.c | 2 ++ + src/liblzma/common/block_encoder.h | 2 ++ + src/liblzma/common/block_header_decoder.c | 2 ++ + src/liblzma/common/block_header_encoder.c | 2 ++ + src/liblzma/common/block_util.c | 2 ++ + src/liblzma/common/common.c | 2 ++ + src/liblzma/common/common.h | 2 ++ + src/liblzma/common/easy_buffer_encoder.c | 2 ++ + src/liblzma/common/easy_decoder_memusage.c | 2 ++ + src/liblzma/common/easy_encoder.c | 2 ++ + src/liblzma/common/easy_encoder_memusage.c | 2 ++ + src/liblzma/common/easy_preset.c | 2 ++ + src/liblzma/common/easy_preset.h | 2 ++ + src/liblzma/common/file_info.c | 2 ++ + src/liblzma/common/filter_buffer_decoder.c | 2 ++ + src/liblzma/common/filter_buffer_encoder.c | 2 ++ + src/liblzma/common/filter_common.c | 2 ++ + src/liblzma/common/filter_common.h | 2 ++ + src/liblzma/common/filter_decoder.c | 2 ++ + src/liblzma/common/filter_decoder.h | 2 ++ + src/liblzma/common/filter_encoder.c | 2 ++ + src/liblzma/common/filter_encoder.h | 2 ++ + src/liblzma/common/filter_flags_decoder.c | 2 ++ + src/liblzma/common/filter_flags_encoder.c | 2 ++ + src/liblzma/common/hardware_cputhreads.c | 2 ++ + src/liblzma/common/hardware_physmem.c | 2 ++ + src/liblzma/common/index.c | 2 ++ + src/liblzma/common/index.h | 2 ++ + src/liblzma/common/index_decoder.c | 2 ++ + src/liblzma/common/index_decoder.h | 2 ++ + src/liblzma/common/index_encoder.c | 2 ++ + src/liblzma/common/index_encoder.h | 2 ++ + src/liblzma/common/index_hash.c | 2 ++ + src/liblzma/common/lzip_decoder.c | 2 ++ + src/liblzma/common/lzip_decoder.h | 2 ++ + src/liblzma/common/memcmplen.h | 2 ++ + src/liblzma/common/microlzma_decoder.c | 2 ++ + src/liblzma/common/microlzma_encoder.c | 2 ++ + src/liblzma/common/outqueue.c | 2 ++ + src/liblzma/common/outqueue.h | 2 ++ + src/liblzma/common/stream_buffer_decoder.c | 2 ++ + src/liblzma/common/stream_buffer_encoder.c | 2 ++ + src/liblzma/common/stream_decoder.c | 2 ++ + src/liblzma/common/stream_decoder.h | 2 ++ + src/liblzma/common/stream_decoder_mt.c | 2 ++ + src/liblzma/common/stream_encoder.c | 2 ++ + src/liblzma/common/stream_encoder_mt.c | 2 ++ + src/liblzma/common/stream_flags_common.c | 2 ++ + src/liblzma/common/stream_flags_common.h | 2 ++ + src/liblzma/common/stream_flags_decoder.c | 2 ++ + src/liblzma/common/stream_flags_encoder.c | 2 ++ + src/liblzma/common/string_conversion.c | 2 ++ + src/liblzma/common/vli_decoder.c | 2 ++ + src/liblzma/common/vli_encoder.c | 2 ++ + src/liblzma/common/vli_size.c | 2 ++ + src/liblzma/delta/Makefile.inc | 3 +-- + src/liblzma/delta/delta_common.c | 2 ++ + src/liblzma/delta/delta_common.h | 2 ++ + src/liblzma/delta/delta_decoder.c | 2 ++ + src/liblzma/delta/delta_decoder.h | 2 ++ + src/liblzma/delta/delta_encoder.c | 2 ++ + src/liblzma/delta/delta_encoder.h | 2 ++ + src/liblzma/delta/delta_private.h | 2 ++ + src/liblzma/liblzma.pc.in | 3 +-- + src/liblzma/liblzma_generic.map | 2 ++ + src/liblzma/liblzma_linux.map | 2 ++ + src/liblzma/liblzma_w32res.rc | 2 ++ + src/liblzma/lz/Makefile.inc | 3 +-- + src/liblzma/lz/lz_decoder.c | 2 ++ + src/liblzma/lz/lz_decoder.h | 2 ++ + src/liblzma/lz/lz_encoder.c | 2 ++ + src/liblzma/lz/lz_encoder.h | 2 ++ + src/liblzma/lz/lz_encoder_hash.h | 2 ++ + src/liblzma/lz/lz_encoder_mf.c | 2 ++ + src/liblzma/lzma/Makefile.inc | 3 +-- + src/liblzma/lzma/fastpos.h | 2 ++ + src/liblzma/lzma/fastpos_tablegen.c | 2 ++ + src/liblzma/lzma/lzma2_decoder.c | 2 ++ + src/liblzma/lzma/lzma2_decoder.h | 2 ++ + src/liblzma/lzma/lzma2_encoder.c | 2 ++ + src/liblzma/lzma/lzma2_encoder.h | 2 ++ + src/liblzma/lzma/lzma_common.h | 2 ++ + src/liblzma/lzma/lzma_decoder.c | 2 ++ + src/liblzma/lzma/lzma_decoder.h | 2 ++ + src/liblzma/lzma/lzma_encoder.c | 2 ++ + src/liblzma/lzma/lzma_encoder.h | 2 ++ + src/liblzma/lzma/lzma_encoder_optimum_fast.c | 2 ++ + src/liblzma/lzma/lzma_encoder_optimum_normal.c | 2 ++ + src/liblzma/lzma/lzma_encoder_presets.c | 2 ++ + src/liblzma/lzma/lzma_encoder_private.h | 2 ++ + src/liblzma/rangecoder/Makefile.inc | 3 +-- + src/liblzma/rangecoder/price.h | 2 ++ + src/liblzma/rangecoder/price_tablegen.c | 2 ++ + src/liblzma/rangecoder/range_common.h | 2 ++ + src/liblzma/rangecoder/range_decoder.h | 2 ++ + src/liblzma/rangecoder/range_encoder.h | 2 ++ + src/liblzma/simple/Makefile.inc | 3 +-- + src/liblzma/simple/arm.c | 2 ++ + src/liblzma/simple/arm64.c | 2 ++ + src/liblzma/simple/armthumb.c | 2 ++ + src/liblzma/simple/ia64.c | 2 ++ + src/liblzma/simple/powerpc.c | 2 ++ + src/liblzma/simple/riscv.c | 2 ++ + src/liblzma/simple/simple_coder.c | 2 ++ + src/liblzma/simple/simple_coder.h | 2 ++ + src/liblzma/simple/simple_decoder.c | 2 ++ + src/liblzma/simple/simple_decoder.h | 2 ++ + src/liblzma/simple/simple_encoder.c | 2 ++ + src/liblzma/simple/simple_encoder.h | 2 ++ + src/liblzma/simple/simple_private.h | 2 ++ + src/liblzma/simple/sparc.c | 2 ++ + src/liblzma/simple/x86.c | 2 ++ + src/liblzma/validate_map.sh | 1 + + src/lzmainfo/Makefile.am | 3 +-- + src/lzmainfo/lzmainfo.c | 2 ++ + src/lzmainfo/lzmainfo_w32res.rc | 2 ++ + src/scripts/Makefile.am | 3 +-- + src/xz/Makefile.am | 3 +-- + src/xz/args.c | 2 ++ + src/xz/args.h | 2 ++ + src/xz/coder.c | 2 ++ + src/xz/coder.h | 2 ++ + src/xz/file_io.c | 2 ++ + src/xz/file_io.h | 2 ++ + src/xz/hardware.c | 2 ++ + src/xz/hardware.h | 2 ++ + src/xz/list.c | 2 ++ + src/xz/list.h | 2 ++ + src/xz/main.c | 2 ++ + src/xz/main.h | 2 ++ + src/xz/message.c | 2 ++ + src/xz/message.h | 2 ++ + src/xz/mytime.c | 2 ++ + src/xz/mytime.h | 2 ++ + src/xz/options.c | 2 ++ + src/xz/options.h | 2 ++ + src/xz/private.h | 2 ++ + src/xz/signals.c | 2 ++ + src/xz/signals.h | 2 ++ + src/xz/suffix.c | 2 ++ + src/xz/suffix.h | 2 ++ + src/xz/util.c | 2 ++ + src/xz/util.h | 2 ++ + src/xz/xz_w32res.rc | 2 ++ + src/xzdec/Makefile.am | 3 +-- + src/xzdec/lzmadec_w32res.rc | 2 ++ + src/xzdec/xzdec.c | 2 ++ + src/xzdec/xzdec_w32res.rc | 2 ++ + tests/Makefile.am | 3 +-- + tests/bcj_test.c | 2 ++ + tests/code_coverage.sh | 1 + + tests/create_compress_files.c | 2 ++ + tests/ossfuzz/fuzz_common.h | 2 ++ + tests/ossfuzz/fuzz_decode_alone.c | 2 ++ + tests/ossfuzz/fuzz_decode_stream.c | 2 ++ + tests/ossfuzz/fuzz_encode_stream.c | 2 ++ + tests/test_bcj_exact_size.c | 2 ++ + tests/test_block_header.c | 2 ++ + tests/test_check.c | 2 ++ + tests/test_compress.sh | 1 + + tests/test_compress_generated_abc | 1 + + tests/test_compress_generated_random | 1 + + tests/test_compress_generated_text | 1 + + tests/test_compress_prepared_bcj_sparc | 1 + + tests/test_compress_prepared_bcj_x86 | 1 + + tests/test_files.sh | 1 + + tests/test_filter_flags.c | 2 ++ + tests/test_filter_str.c | 2 ++ + tests/test_hardware.c | 2 ++ + tests/test_index.c | 2 ++ + tests/test_index_hash.c | 2 ++ + tests/test_lzip_decoder.c | 2 ++ + tests/test_memlimit.c | 2 ++ + tests/test_scripts.sh | 1 + + tests/test_stream_flags.c | 2 ++ + tests/test_suffix.sh | 1 + + tests/test_vli.c | 2 ++ + tests/tests.h | 2 ++ + tests/tuktest.h | 2 ++ + windows/build.bash | 3 ++- + 290 files changed, 588 insertions(+), 58 deletions(-) + +commit 23de53421ea258cde6a3c33a038b1e9d08f771d1 +Author: Lasse Collin +Date: 2024-02-12 23:25:54 +0200 + + liblzma: Sync the AUTHORS fix about SHA-256 to lzma.h. + + src/liblzma/api/lzma.h | 10 ++++------ + 1 file changed, 4 insertions(+), 6 deletions(-) + +commit 689e0228baeb95232430e90d628379db89583d71 +Author: Lasse Collin +Date: 2024-02-12 17:09:10 +0200 + + Change most public domain parts to 0BSD. + + Translations and doc/xz-file-format.txt and doc/lzma-file-format.txt + were not touched. + + COPYING.0BSD was added. + + .github/workflows/ci.yml | 3 - + .github/workflows/windows-ci.yml | 3 - + CMakeLists.txt | 3 - + COPYING | 112 ++++++++++++++----------- + COPYING.0BSD | 11 +++ + Makefile.am | 3 - + PACKAGERS | 11 +-- + autogen.sh | 3 - + build-aux/ci_build.sh | 3 - + build-aux/manconv.sh | 3 - + build-aux/version.sh | 3 - + cmake/remove-ordinals.cmake | 3 - + cmake/tuklib_common.cmake | 3 - + cmake/tuklib_cpucores.cmake | 3 - + cmake/tuklib_integer.cmake | 3 - + cmake/tuklib_large_file_support.cmake | 3 - + cmake/tuklib_mbstr.cmake | 3 - + cmake/tuklib_physmem.cmake | 3 - + cmake/tuklib_progname.cmake | 3 - + configure.ac | 3 - + debug/Makefile.am | 3 - + debug/crc32.c | 3 - + debug/full_flush.c | 3 - + debug/hex2bin.c | 3 - + debug/known_sizes.c | 3 - + debug/memusage.c | 3 - + debug/repeat.c | 3 - + debug/sync_flush.c | 3 - + debug/translation.bash | 3 - + doc/examples/01_compress_easy.c | 3 - + doc/examples/02_decompress.c | 3 - + doc/examples/03_compress_custom.c | 3 - + doc/examples/04_compress_easy_mt.c | 3 - + doc/examples/11_file_info.c | 3 - + doc/examples/Makefile | 3 - + dos/Makefile | 3 - + doxygen/update-doxygen | 3 - + extra/7z2lzma/7z2lzma.bash | 3 - + m4/tuklib_common.m4 | 3 - + m4/tuklib_cpucores.m4 | 3 - + m4/tuklib_integer.m4 | 3 - + m4/tuklib_mbstr.m4 | 3 - + m4/tuklib_physmem.m4 | 3 - + m4/tuklib_progname.m4 | 3 - + po4a/update-po | 3 - + src/Makefile.am | 3 - + src/common/common_w32res.rc | 3 - + src/common/mythread.h | 3 - + src/common/sysdefs.h | 3 - + src/common/tuklib_common.h | 3 - + src/common/tuklib_cpucores.c | 3 - + src/common/tuklib_cpucores.h | 3 - + src/common/tuklib_exit.c | 3 - + src/common/tuklib_exit.h | 3 - + src/common/tuklib_gettext.h | 3 - + src/common/tuklib_integer.h | 3 - + src/common/tuklib_mbstr.h | 3 - + src/common/tuklib_mbstr_fw.c | 3 - + src/common/tuklib_mbstr_width.c | 3 - + src/common/tuklib_open_stdxxx.c | 3 - + src/common/tuklib_open_stdxxx.h | 3 - + src/common/tuklib_physmem.c | 3 - + src/common/tuklib_physmem.h | 3 - + src/common/tuklib_progname.c | 3 - + src/common/tuklib_progname.h | 3 - + src/liblzma/Makefile.am | 3 - + src/liblzma/api/Makefile.am | 3 - + src/liblzma/api/lzma.h | 13 ++- + src/liblzma/api/lzma/base.h | 3 - + src/liblzma/api/lzma/bcj.h | 3 - + src/liblzma/api/lzma/block.h | 3 - + src/liblzma/api/lzma/check.h | 3 - + src/liblzma/api/lzma/container.h | 3 - + src/liblzma/api/lzma/delta.h | 3 - + src/liblzma/api/lzma/filter.h | 3 - + src/liblzma/api/lzma/hardware.h | 3 - + src/liblzma/api/lzma/index.h | 3 - + src/liblzma/api/lzma/index_hash.h | 3 - + src/liblzma/api/lzma/lzma12.h | 3 - + src/liblzma/api/lzma/stream_flags.h | 3 - + src/liblzma/api/lzma/version.h | 3 - + src/liblzma/api/lzma/vli.h | 3 - + src/liblzma/check/Makefile.inc | 3 - + src/liblzma/check/check.c | 3 - + src/liblzma/check/check.h | 3 - + src/liblzma/check/crc32_arm64.h | 3 - + src/liblzma/check/crc32_fast.c | 3 - + src/liblzma/check/crc32_small.c | 3 - + src/liblzma/check/crc32_table.c | 3 - + src/liblzma/check/crc32_tablegen.c | 3 - + src/liblzma/check/crc32_x86.S | 3 - + src/liblzma/check/crc64_fast.c | 3 - + src/liblzma/check/crc64_small.c | 3 - + src/liblzma/check/crc64_table.c | 3 - + src/liblzma/check/crc64_tablegen.c | 3 - + src/liblzma/check/crc64_x86.S | 3 - + src/liblzma/check/crc_common.h | 3 - + src/liblzma/check/crc_x86_clmul.h | 3 - + src/liblzma/check/sha256.c | 3 - + src/liblzma/common/Makefile.inc | 3 - + src/liblzma/common/alone_decoder.c | 3 - + src/liblzma/common/alone_decoder.h | 3 - + src/liblzma/common/alone_encoder.c | 3 - + src/liblzma/common/auto_decoder.c | 3 - + src/liblzma/common/block_buffer_decoder.c | 3 - + src/liblzma/common/block_buffer_encoder.c | 3 - + src/liblzma/common/block_buffer_encoder.h | 3 - + src/liblzma/common/block_decoder.c | 3 - + src/liblzma/common/block_decoder.h | 3 - + src/liblzma/common/block_encoder.c | 3 - + src/liblzma/common/block_encoder.h | 3 - + src/liblzma/common/block_header_decoder.c | 3 - + src/liblzma/common/block_header_encoder.c | 3 - + src/liblzma/common/block_util.c | 3 - + src/liblzma/common/common.c | 3 - + src/liblzma/common/common.h | 3 - + src/liblzma/common/easy_buffer_encoder.c | 3 - + src/liblzma/common/easy_decoder_memusage.c | 3 - + src/liblzma/common/easy_encoder.c | 3 - + src/liblzma/common/easy_encoder_memusage.c | 3 - + src/liblzma/common/easy_preset.c | 3 - + src/liblzma/common/easy_preset.h | 3 - + src/liblzma/common/file_info.c | 3 - + src/liblzma/common/filter_buffer_decoder.c | 3 - + src/liblzma/common/filter_buffer_encoder.c | 3 - + src/liblzma/common/filter_common.c | 3 - + src/liblzma/common/filter_common.h | 3 - + src/liblzma/common/filter_decoder.c | 3 - + src/liblzma/common/filter_decoder.h | 3 - + src/liblzma/common/filter_encoder.c | 3 - + src/liblzma/common/filter_encoder.h | 3 - + src/liblzma/common/filter_flags_decoder.c | 3 - + src/liblzma/common/filter_flags_encoder.c | 3 - + src/liblzma/common/hardware_cputhreads.c | 3 - + src/liblzma/common/hardware_physmem.c | 3 - + src/liblzma/common/index.c | 3 - + src/liblzma/common/index.h | 3 - + src/liblzma/common/index_decoder.c | 3 - + src/liblzma/common/index_decoder.h | 3 - + src/liblzma/common/index_encoder.c | 3 - + src/liblzma/common/index_encoder.h | 3 - + src/liblzma/common/index_hash.c | 3 - + src/liblzma/common/lzip_decoder.c | 3 - + src/liblzma/common/lzip_decoder.h | 3 - + src/liblzma/common/memcmplen.h | 3 - + src/liblzma/common/microlzma_decoder.c | 3 - + src/liblzma/common/microlzma_encoder.c | 3 - + src/liblzma/common/outqueue.c | 3 - + src/liblzma/common/outqueue.h | 3 - + src/liblzma/common/stream_buffer_decoder.c | 3 - + src/liblzma/common/stream_buffer_encoder.c | 3 - + src/liblzma/common/stream_decoder.c | 3 - + src/liblzma/common/stream_decoder.h | 3 - + src/liblzma/common/stream_decoder_mt.c | 3 - + src/liblzma/common/stream_encoder.c | 3 - + src/liblzma/common/stream_encoder_mt.c | 3 - + src/liblzma/common/stream_flags_common.c | 3 - + src/liblzma/common/stream_flags_common.h | 3 - + src/liblzma/common/stream_flags_decoder.c | 3 - + src/liblzma/common/stream_flags_encoder.c | 3 - + src/liblzma/common/string_conversion.c | 3 - + src/liblzma/common/vli_decoder.c | 3 - + src/liblzma/common/vli_encoder.c | 3 - + src/liblzma/common/vli_size.c | 3 - + src/liblzma/delta/Makefile.inc | 3 - + src/liblzma/delta/delta_common.c | 3 - + src/liblzma/delta/delta_common.h | 3 - + src/liblzma/delta/delta_decoder.c | 3 - + src/liblzma/delta/delta_decoder.h | 3 - + src/liblzma/delta/delta_encoder.c | 3 - + src/liblzma/delta/delta_encoder.h | 3 - + src/liblzma/delta/delta_private.h | 3 - + src/liblzma/liblzma.pc.in | 3 - + src/liblzma/liblzma_w32res.rc | 3 - + src/liblzma/lz/Makefile.inc | 3 - + src/liblzma/lz/lz_decoder.c | 3 - + src/liblzma/lz/lz_decoder.h | 3 - + src/liblzma/lz/lz_encoder.c | 3 - + src/liblzma/lz/lz_encoder.h | 3 - + src/liblzma/lz/lz_encoder_hash.h | 3 - + src/liblzma/lz/lz_encoder_mf.c | 3 - + src/liblzma/lzma/Makefile.inc | 3 - + src/liblzma/lzma/fastpos.h | 3 - + src/liblzma/lzma/fastpos_tablegen.c | 3 - + src/liblzma/lzma/lzma2_decoder.c | 3 - + src/liblzma/lzma/lzma2_decoder.h | 3 - + src/liblzma/lzma/lzma2_encoder.c | 3 - + src/liblzma/lzma/lzma2_encoder.h | 3 - + src/liblzma/lzma/lzma_common.h | 3 - + src/liblzma/lzma/lzma_decoder.c | 3 - + src/liblzma/lzma/lzma_decoder.h | 3 - + src/liblzma/lzma/lzma_encoder.c | 3 - + src/liblzma/lzma/lzma_encoder.h | 3 - + src/liblzma/lzma/lzma_encoder_optimum_fast.c | 3 - + src/liblzma/lzma/lzma_encoder_optimum_normal.c | 3 - + src/liblzma/lzma/lzma_encoder_presets.c | 3 - + src/liblzma/lzma/lzma_encoder_private.h | 3 - + src/liblzma/rangecoder/Makefile.inc | 3 - + src/liblzma/rangecoder/price.h | 3 - + src/liblzma/rangecoder/price_tablegen.c | 3 - + src/liblzma/rangecoder/range_common.h | 3 - + src/liblzma/rangecoder/range_decoder.h | 3 - + src/liblzma/rangecoder/range_encoder.h | 3 - + src/liblzma/simple/Makefile.inc | 3 - + src/liblzma/simple/arm.c | 3 - + src/liblzma/simple/arm64.c | 3 - + src/liblzma/simple/armthumb.c | 3 - + src/liblzma/simple/ia64.c | 3 - + src/liblzma/simple/powerpc.c | 3 - + src/liblzma/simple/riscv.c | 3 - + src/liblzma/simple/simple_coder.c | 3 - + src/liblzma/simple/simple_coder.h | 3 - + src/liblzma/simple/simple_decoder.c | 3 - + src/liblzma/simple/simple_decoder.h | 3 - + src/liblzma/simple/simple_encoder.c | 3 - + src/liblzma/simple/simple_encoder.h | 3 - + src/liblzma/simple/simple_private.h | 3 - + src/liblzma/simple/sparc.c | 3 - + src/liblzma/simple/x86.c | 3 - + src/liblzma/validate_map.sh | 3 - + src/lzmainfo/Makefile.am | 3 - + src/lzmainfo/lzmainfo.1 | 4 +- + src/lzmainfo/lzmainfo.c | 3 - + src/lzmainfo/lzmainfo_w32res.rc | 3 - + src/scripts/Makefile.am | 3 - + src/scripts/xzless.1 | 4 +- + src/xz/Makefile.am | 3 - + src/xz/args.c | 3 - + src/xz/args.h | 3 - + src/xz/coder.c | 3 - + src/xz/coder.h | 3 - + src/xz/file_io.c | 3 - + src/xz/file_io.h | 3 - + src/xz/hardware.c | 3 - + src/xz/hardware.h | 3 - + src/xz/list.c | 3 - + src/xz/list.h | 3 - + src/xz/main.c | 3 - + src/xz/main.h | 3 - + src/xz/message.c | 3 - + src/xz/message.h | 3 - + src/xz/mytime.c | 3 - + src/xz/mytime.h | 3 - + src/xz/options.c | 3 - + src/xz/options.h | 3 - + src/xz/private.h | 3 - + src/xz/signals.c | 3 - + src/xz/signals.h | 3 - + src/xz/suffix.c | 3 - + src/xz/suffix.h | 3 - + src/xz/util.c | 3 - + src/xz/util.h | 3 - + src/xz/xz.1 | 4 +- + src/xz/xz_w32res.rc | 3 - + src/xzdec/Makefile.am | 3 - + src/xzdec/lzmadec_w32res.rc | 3 - + src/xzdec/xzdec.1 | 4 +- + src/xzdec/xzdec.c | 3 - + src/xzdec/xzdec_w32res.rc | 3 - + tests/Makefile.am | 3 - + tests/bcj_test.c | 3 - + tests/code_coverage.sh | 3 - + tests/create_compress_files.c | 3 - + tests/files/README | 3 +- + tests/ossfuzz/fuzz_common.h | 3 - + tests/ossfuzz/fuzz_decode_alone.c | 3 - + tests/ossfuzz/fuzz_decode_stream.c | 3 - + tests/ossfuzz/fuzz_encode_stream.c | 3 - + tests/test_bcj_exact_size.c | 3 - + tests/test_block_header.c | 3 - + tests/test_check.c | 3 - + tests/test_compress.sh | 3 - + tests/test_files.sh | 3 - + tests/test_filter_flags.c | 3 - + tests/test_filter_str.c | 3 - + tests/test_hardware.c | 3 - + tests/test_index.c | 3 - + tests/test_index_hash.c | 3 - + tests/test_lzip_decoder.c | 3 - + tests/test_memlimit.c | 3 - + tests/test_scripts.sh | 3 - + tests/test_stream_flags.c | 3 - + tests/test_suffix.sh | 3 - + tests/test_vli.c | 3 - + tests/tests.h | 3 - + tests/tuktest.h | 3 - + windows/README-Windows.txt | 11 +-- + windows/build.bash | 3 - + 288 files changed, 100 insertions(+), 911 deletions(-) + +commit 76946dc4336c831fe2cc26696a035d807dd3cf13 +Author: Lasse Collin +Date: 2024-02-09 17:20:31 +0200 + + Fix SHA-256 authors. + + The initial commit 5d018dc03549c1ee4958364712fb0c94e1bf2741 + in 2007 had a comment in sha256.c that the code is based on + Crypto++ Library 5.5.1. In 2009 the Authors list in sha256.c + and the AUTHORS file was updated with information that the + code had come from Crypto++ but via 7-Zip. I know I had viewed + 7-Zip's SHA-256 code but back then the C code has been identical + enough with Crypto++, so I don't why I thought the author info + would need that extra step via 7-Zip for this single file. + + Another error is that I had mixed sha.* and shacal2.* files + when checking for author info in Crypto++. The shacal2.* files + aren't related to liblzma's sha256.c and thus Kevin Springle's + code in Crypto++ isn't either. + + AUTHORS | 6 ++---- + src/liblzma/check/sha256.c | 14 ++++---------- + 2 files changed, 6 insertions(+), 14 deletions(-) + +commit 21d9cbae9eecca28ce373d3d9464defd2cf5d851 +Author: Lasse Collin +Date: 2024-02-09 17:20:31 +0200 + + Remove macosx/build.sh. + + It was last updated in 2013. + + Makefile.am | 1 - + macosx/build.sh | 113 -------------------------------------------------------- + 2 files changed, 114 deletions(-) + +commit eac2c3c67f9113a225fb6667df862edd30366931 +Author: Lasse Collin +Date: 2024-02-09 17:20:31 +0200 + + Doc: Remove doc/examples_old. + + It was good to keep these around in parallel with the newer examples + but I think it's OK to remove the old ones at this point. + + Makefile.am | 5 -- + doc/examples_old/xz_pipe_comp.c | 127 -------------------------------------- + doc/examples_old/xz_pipe_decomp.c | 123 ------------------------------------ + 3 files changed, 255 deletions(-) + +commit 89ea1a22f4ed3685b053b7260bc5acf6c75d1664 +Author: Jia Tan +Date: 2024-02-13 22:38:58 +0800 + + Tests: Add RISC-V filter support in a few places. + + tests/test_filter_flags.c | 6 ++++++ + tests/test_filter_str.c | 6 ++++++ + 2 files changed, 12 insertions(+) + +commit 45663443eb2b377e6171529380fee312f1adcdf4 +Author: Jia Tan +Date: 2024-02-13 22:37:07 +0800 + + liblzma: Fix build error if only RISC-V BCJ filter is enabled. + + If any other BCJ filter was enabled for encoding or decoding, then this + was not a problem. + + src/liblzma/common/string_conversion.c | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 2f15597d677bc35743c777d4cf3bfa698b478681 +Author: Jia Tan +Date: 2024-02-13 22:56:24 +0800 + + Translations: Update the Korean translation. + + po/ko.po | 526 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 284 insertions(+), 242 deletions(-) + +commit df873143ad1615c6d6aaa1bf8808b1676091dfe3 +Author: Jia Tan +Date: 2024-02-13 01:55:53 +0800 + + Translations: Update the Korean man page translations. + + po4a/ko.po | 1375 ++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 770 insertions(+), 605 deletions(-) + +commit b3f415eddb150341865a1af47959c3baba076b33 +Author: Jia Tan +Date: 2024-02-13 01:53:33 +0800 + + Translations: Update the Chinese (simplified) translation. + + po/zh_CN.po | 424 ++++++++++++++++++++++++++++++++++++++---------------------- + 1 file changed, 268 insertions(+), 156 deletions(-) + +commit 9860d418d296eb3c721e5384fb367c0499b579c8 +Author: Lasse Collin +Date: 2024-02-09 23:21:01 +0200 + + xzless: Use ||- in LESSOPEN with with "less" 451 and newer. + + src/scripts/xzless.in | 9 ++++++++- + 1 file changed, 8 insertions(+), 1 deletion(-) + +commit fd0692b0525e6c26b496492be9e2c865cab734f8 +Author: Lasse Collin +Date: 2024-02-09 23:00:05 +0200 + + xzless: Use --show-preproc-errors with "less" 632 and newer. + + This makes "less" show a warning if a decompression error occurred. + + src/scripts/xzless.in | 11 +++++++++-- + 1 file changed, 9 insertions(+), 2 deletions(-) + +commit adb073da76a920b5a81e6b32254f4ddb054dc57a +Author: Jia Tan +Date: 2024-02-09 23:59:54 +0800 + + liblzma: Fix typo discovered by codespell. + + src/liblzma/check/crc32_arm64.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 55d9fc883d221cbace951a370f1fb144698f8c2e +Author: Jia Tan +Date: 2024-02-09 20:01:06 +0800 + + Translations: Update the Swedish translation. + + po/sv.po | 420 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 254 insertions(+), 166 deletions(-) + +commit 55ba4a1ea321499c805eedfa811ffde690bae311 +Author: Jia Tan +Date: 2024-02-08 20:09:04 +0800 + + Translations: Update the Spanish translation. + + po/es.po | 22 +++++++++++----------- + 1 file changed, 11 insertions(+), 11 deletions(-) + +commit 7f2293cd804a89d3c3b2d3ed573560ca9e1520ae +Author: Jia Tan +Date: 2024-02-07 21:34:35 +0800 + + Translations: Update the Spanish translation. + + po/es.po | 419 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 253 insertions(+), 166 deletions(-) + +commit f4af2036bc625739d6d33d9e1fede583a25c3828 +Author: Jia Tan +Date: 2024-02-07 21:28:32 +0800 + + Translations: Update the Polish translation. + + po/pl.po | 411 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 249 insertions(+), 162 deletions(-) + +commit e5e93bb816043c559cddf03a3b7ba13bec353ee4 +Author: Jia Tan +Date: 2024-02-07 19:40:12 +0800 + + Translations: Update the German translation. + + po/de.po | 396 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 242 insertions(+), 154 deletions(-) + +commit 28f18ff8e26902762fb007c13be235b4ac1ac071 +Author: Jia Tan +Date: 2024-02-07 19:27:25 +0800 + + Translations: Update the German man page translations. + + po4a/de.po | 1353 +++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 752 insertions(+), 601 deletions(-) + +commit cabfbc7947da05aa5dfe39bec9759e076f940e3c +Author: Jia Tan +Date: 2024-02-06 23:44:06 +0800 + + Translations: Update the Romanian translation. + + po/ro.po | 416 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 252 insertions(+), 164 deletions(-) + +commit bf20c94f5d748cea2147779f4fa7e2fd2eb8555e +Author: Jia Tan +Date: 2024-02-06 23:45:02 +0800 + + Translations: Update the Romanian man page translations. + + po4a/ro.po | 1759 +++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 966 insertions(+), 793 deletions(-) + +commit 7c25ec9feb0241e4affb7432681cc4f5696f3a96 +Author: Jia Tan +Date: 2024-02-07 20:56:57 +0800 + + Translations: Update the Ukrainian translation. + + po/uk.po | 397 ++++++++++++++++++++++++++++++++++++++------------------------- + 1 file changed, 242 insertions(+), 155 deletions(-) + +commit b3523250e9eef10b017473754c1e1c9e31f10374 +Author: Jia Tan +Date: 2024-02-06 23:30:03 +0800 + + Translations: Update the Ukrainian man page translations. + + po4a/uk.po | 1363 ++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 764 insertions(+), 599 deletions(-) + +commit a5c177f514f4c90e0d2f6045636fca6c2e80a20d +Author: Jia Tan +Date: 2024-02-02 01:39:28 +0800 + + Update AUTHORS. + + AUTHORS | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 7f68a68c19d0ae57bd0e802be0ea8f974e41299f +Author: Jia Tan +Date: 2024-02-02 01:38:51 +0800 + + liblzma: Update Authors list in crc32_arm64.h. + + src/liblzma/check/crc32_arm64.h | 1 + + 1 file changed, 1 insertion(+) + +commit 97f9ba50b84e67b3dcb5b17dd5d3e1d14f9ad1d0 +Author: Jia Tan +Date: 2024-02-01 16:07:03 +0800 + + liblzma: Check HAVE_USABLE_CLMUL before omitting CRC32 table. + + This was split from the prior commit so it could be easily applied to + the 5.4 branch. + + Closes: https://github.com/tukaani-project/xz/pull/77 + + src/liblzma/check/crc32_table.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit ca9015f4dead2c97b48f5a6933631b0a448b65b9 +Author: Jia Tan +Date: 2024-02-01 16:06:29 +0800 + + liblzma: Check HAVE_USABLE_CLMUL before omitting CRC64 table. + + If liblzma is configured with --disable-clmul-crc + CFLAGS="-msse4.1 -mpclmul", then it will fail to compile because the + generic version must be used but the CRC tables were not included. + + src/liblzma/check/crc64_table.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 2f1552a91c825e87013925e1a67a0930e7aef592 +Author: Jia Tan +Date: 2024-01-23 18:02:13 +0800 + + liblzma: Only use ifunc in crcXX_fast.c if its needed. + + The code was using HAVE_FUNC_ATTRIBUTE_IFUNC instead of CRC_USE_IFUNC. + With ARM64, ifunc is incompatible because it requires non-inline + function calls for runtime detection. + + src/liblzma/check/crc32_fast.c | 6 +++--- + src/liblzma/check/crc64_fast.c | 6 +++--- + 2 files changed, 6 insertions(+), 6 deletions(-) + +commit 30a25f3742287697bc57a1bef86c19ecf5129322 +Author: Jia Tan +Date: 2024-01-22 22:08:45 +0800 + + Docs: Add --disable-arm64-crc32 description to INSTALL. + + INSTALL | 12 +++++++++++- + 1 file changed, 11 insertions(+), 1 deletion(-) + +commit 1940f0ec28f08c0ac72c1413d9706fb82eabe6ad +Author: Jia Tan +Date: 2024-01-22 21:36:09 +0800 + + liblzma: Omit CRC tables when not needed with ARM64 optimizations. + + This is similar to the existing x86-64 CLMUL conditions to omit the + tables. They were slightly refactored to improve readability. + + src/liblzma/check/crc32_table.c | 18 +++++++++++++++--- + src/liblzma/check/crc64_table.c | 7 ++++++- + src/liblzma/check/crc_common.h | 5 ++++- + 3 files changed, 25 insertions(+), 5 deletions(-) + +commit 761f5b69a4c778c8bcb09279b845b07c28790575 +Author: Jia Tan +Date: 2024-01-22 20:54:56 +0800 + + liblzma: Rename crc32_aarch64.h to crc32_arm64.h. + + Even though the proper name for the architecture is aarch64, this + project uses ARM64 throughout. So the rename is for consistency. + + Additionally, crc32_arm64.h was slightly refactored for the following + changes: + + * Added MSVC, FreeBSD, and macOS support in + is_arch_extension_supported(). + + * crc32_arch_optimized() now checks the size when aligning the + buffer. + + * crc32_arch_optimized() loop conditions were slightly modified to + avoid both decrementing the size and incrementing the buffer + pointer. + + * Use the intrinsic wrappers defined in because GCC and + Clang name them differently. + + * Minor spacing and comment changes. + + CMakeLists.txt | 2 +- + src/liblzma/check/Makefile.inc | 2 +- + src/liblzma/check/crc32_aarch64.h | 109 ---------------------------------- + src/liblzma/check/crc32_arm64.h | 119 ++++++++++++++++++++++++++++++++++++++ + src/liblzma/check/crc32_fast.c | 3 +- + src/liblzma/check/crc64_fast.c | 3 - + 6 files changed, 122 insertions(+), 116 deletions(-) + +commit 455a08609caa3223066a717fb01bfa42c5dba47d +Author: Jia Tan +Date: 2024-01-22 20:49:30 +0800 + + liblzma: Refactor crc_common.h. + + The CRC_GENERIC is now split into CRC32_GENERIC and CRC64_GENERIC, since + the ARM64 optimizations will be different between CRC32 and CRC64. + + For the same reason, CRC_ARCH_OPTIMIZED is split into + CRC32_ARCH_OPTIMIZED and CRC64_ARCH_OPTIMIZED. + + ifunc will only be used with x86-64 CLMUL because the runtime detection + methods needed with ARM64 are not compatible with ifunc. + + src/liblzma/check/crc32_fast.c | 8 +-- + src/liblzma/check/crc64_fast.c | 8 +-- + src/liblzma/check/crc_common.h | 108 ++++++++++++++++++++++++++++------------- + 3 files changed, 82 insertions(+), 42 deletions(-) + +commit 61908e816049af7a9f43ea804a57ee8570e2e644 +Author: Jia Tan +Date: 2024-01-22 00:42:28 +0800 + + CMake: Add support for ARM64 CRC32 instruction detection. + + CMakeLists.txt | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 50 insertions(+) + +commit c5f6d79cc9515a7f22d7ea4860c6cc394b295732 +Author: Jia Tan +Date: 2024-01-22 00:36:47 +0800 + + Build: Add support for ARM64 CRC32 instruction detection. + + This adds --enable-arm64-crc32/--disable-arm64-crc32 (enabled by + default) for using the ARM64 CRC32 instruction. This can be disabled if + one knows the binary will never need to run on an ARM64 machine + with this instruction extension. + + configure.ac | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 52 insertions(+) + +commit 849d0f282a6a890c5cf5a0e0f02980b12d9ebb0f +Author: Chenxi Mao +Date: 2024-01-09 17:23:11 +0800 + + Speed up CRC32 calculation on ARM64 + + The CRC32 instructions in ARM64 can calculate the CRC32 result + for 8 bytes in a single operation, making the use of ARM64 + instructions much faster compared to the general CRC32 algorithm. + + Optimized CRC32 will be enabled if ARM64 has CRC extension + running on Linux. + + Signed-off-by: Chenxi Mao + + CMakeLists.txt | 1 + + src/liblzma/check/Makefile.inc | 3 +- + src/liblzma/check/crc32_aarch64.h | 109 ++++++++++++++++++++++++++++++++++++++ + src/liblzma/check/crc32_fast.c | 5 +- + src/liblzma/check/crc64_fast.c | 5 +- + src/liblzma/check/crc_common.h | 16 +++--- + 6 files changed, 130 insertions(+), 9 deletions(-) + +commit b43c3e48bf6097095eef36d44cdbec811074940a +Author: Jia Tan +Date: 2024-01-26 19:05:51 +0800 + + Bump version number for 5.5.1alpha. + + src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 3 files changed, 3 insertions(+), 3 deletions(-) + +commit c7a7ae1500ea90bd3c2d54533e4f433933eb598f +Author: Jia Tan +Date: 2024-01-26 19:00:52 +0800 + + Add NEWS for 5.5.1alpha + + NEWS | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 80 insertions(+) + +commit 0ef8192e8d5af4e6200d5d4aee22d1f177f7a2df +Author: Jia Tan +Date: 2024-01-26 18:54:24 +0800 + + Add NEWS for 5.4.6. + + NEWS | 22 ++++++++++++++++++++++ + 1 file changed, 22 insertions(+) + +commit 93de7e751d17731315a899264f2a7239d7d2d316 +Author: Lasse Collin +Date: 2024-01-24 20:00:57 +0200 + + Move doc/logo/xz-logo.png to "doc" and Doxygen footer to "doxygen". + + The footer isn't a complete HTML file so having it in the doxygen + directory is a tiny bit clearer. + + Makefile.am | 2 +- + doc/{logo => }/xz-logo.png | Bin + doxygen/Doxyfile | 4 ++-- + doc/logo/copyright.html => doxygen/footer.html | 0 + 4 files changed, 3 insertions(+), 3 deletions(-) + +commit 00fa01698df51c58ae2acf8c7fa4e1fb159f75a9 +Author: Jia Tan +Date: 2024-01-09 17:05:01 +0800 + + README: Add COPYING.CC-BY-SA-4.0 entry to section 1.1. + + The Overall documentation section (1.1) table spacing had to be adjusted + since the filename was very long. + + README | 38 ++++++++++++++++++++------------------ + 1 file changed, 20 insertions(+), 18 deletions(-) + +commit e280470040b27c5e58d78b25b9e2bb71fc6c3882 +Author: Jia Tan +Date: 2024-01-09 16:56:16 +0800 + + Build: Add the logo and license to the release. + + Makefile.am | 2 ++ + 1 file changed, 2 insertions(+) + +commit b1ee6cf259bb49ce91abe9f622294524e37edf4c +Author: Jia Tan +Date: 2024-01-09 16:44:42 +0800 + + COPYING: Add the license for the XZ logo. + + COPYING | 5 + + COPYING.CC-BY-SA-4.0 | 427 +++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 432 insertions(+) + +commit 31293ae7074802cc7286089a89c7b552d930c97f +Author: Jia Tan +Date: 2024-01-09 16:40:56 +0800 + + Doxygen: Added the XZ logo and copyright information. + + The PROJECT_LOGO field is now used to include the XZ logo. The footer + of each page now lists the copyright information instead of the default + footer. The license is also copied to statisfy the copyright and so the + link in the documentation can be local. + + doc/logo/copyright.html | 11 +++++++++++ + doc/logo/xz-logo.png | Bin 0 -> 6771 bytes + doxygen/Doxyfile | 6 +++--- + 3 files changed, 14 insertions(+), 3 deletions(-) + +commit 6daa4d0ea46a8441f21f609149f3633158bf4704 +Author: Lasse Collin +Date: 2024-01-23 18:29:28 +0200 + + xz: Use threaded mode by defaut (as if --threads=0 was used). + + This hopefully does more good than bad: + + + It's faster by default. + + + Only the threaded compressor creates files that + can be decompressed in threaded mode. + + - Compression ratio is worse, usually not too much though. + When it matters, -T1 must be used. + + - Memory usage increases. + + - Scripts that assume single-threaded mode but don't use -T1 will + possibly use too much resources, for example, if they run + multiple xz processes in parallel to compress multiple files. + + - Output from single-threaded and multi-threaded compressors + differ but such changes could happen for other reasons too + (they just haven't happened since 5.0.0). + + src/xz/hardware.c | 6 +++++- + src/xz/message.c | 4 ++-- + src/xz/xz.1 | 9 +++++++++ + 3 files changed, 16 insertions(+), 3 deletions(-) + +commit a2dd2dc8e5307a7280bb99868bc478560facba2c +Author: Jia Tan +Date: 2024-01-23 23:52:49 +0800 + + CI: Use RISC-V filter when building with BCJ support. + + build-aux/ci_build.sh | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 3060e1070b2421b26c0e17794c1307ec5622f11d +Author: Jia Tan +Date: 2024-01-23 23:52:14 +0800 + + Tests: Use smaller dictionary size in RISC-V test files. + + tests/files/good-1-riscv-lzma2-1.xz | Bin 7512 -> 7512 bytes + tests/files/good-1-riscv-lzma2-2.xz | Bin 7516 -> 7512 bytes + 2 files changed, 0 insertions(+), 0 deletions(-) + +commit 44ff2fa5c94dc345c4dd69195a19fc5238df60b3 +Author: Jia Tan +Date: 2024-01-23 23:50:57 +0800 + + Tests: Skip RISC-V test files if decoder was not built. + + tests/test_files.sh | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 6133a3f30049d3beaf7d22535b1e5d38e109be4e +Author: Lasse Collin +Date: 2024-01-23 16:11:54 +0200 + + xz: Man page: Add more examples of LZMA2 options with BCJ filters. + + src/xz/xz.1 | 38 +++++++++++++++++++++++++++++++------- + 1 file changed, 31 insertions(+), 7 deletions(-) + +commit 50255feeaabcc7e7db22b858a6bd64a9b5b4f16d +Author: Lasse Collin +Date: 2024-01-23 00:09:48 +0200 + + liblzma: RISC-V filter: Use byte-by-byte access. + + Not all RISC-V processors support fast unaligned access so + it's better to read only one byte in the main loop. This can + be faster even on x86-64 when compared to reading 32 bits at + a time as half the time the address is only 16-bit aligned. + + The downside is larger code size on archs that do support + fast unaligned access. + + src/liblzma/simple/riscv.c | 114 +++++++++++++++++++++++++++++++++------------ + 1 file changed, 84 insertions(+), 30 deletions(-) + +commit db5eb5f563e8baa8d912ecf576f53391ff861596 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + xz: Update xz -lvv for RISC-V filter. + + Version 5.6.0 will be shown, even though upcoming alphas and betas + will be able to support this filter. 5.6.0 looks nicer in the output and + people shouldn't be encouraged to use an unstable version in production + in any way. + + src/xz/list.c | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit e2870db5be1503e6a489fc3d47daf950d6f62723 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + Tests: Add two RISC-V Filter test files. + + These test files achieve 100% code coverage in + src/liblzma/simple/riscv.c. They contain all of the instructions that + should be filtered and a few cases that should not. + + tests/files/README | 8 ++++++++ + tests/files/good-1-riscv-lzma2-1.xz | Bin 0 -> 7512 bytes + tests/files/good-1-riscv-lzma2-2.xz | Bin 0 -> 7516 bytes + 3 files changed, 8 insertions(+) + +commit b26a89869315ece2f6d9d10d32d45f672550f245 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + xz: Update message in --long-help for RISC-V Filter. + + src/xz/message.c | 1 + + 1 file changed, 1 insertion(+) + +commit 283f778908873eca61388029fc418fa800c9d7d7 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + xz: Update the man page for the RISC-V Filter. + + A special note was added to suggest using four-byte alignment when the + compressed instruction extension is not present in a RISC-V binary. + + src/xz/xz.1 | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit ac3691ccca051d67f60b4a3b05b88e511d0b1b28 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + Tests: Add RISC-V Filter test in test_compress.sh. + + tests/test_compress.sh | 1 + + 1 file changed, 1 insertion(+) + +commit 2959dbc7358efcf421ce51bc9cd7eae8fdd8fec4 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + liblzma: Update string_conversion.c to support RISC-V Filter. + + src/liblzma/common/string_conversion.c | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 34372a5adbe5a7f6bf29498410ba3a463a720966 +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + CMake: Support RISC-V BCJ Filter for encoding and decoding. + + CMakeLists.txt | 1 + + 1 file changed, 1 insertion(+) + +commit 440a2eccb082dc13400c09e22308a58fef85146c +Author: Jia Tan +Date: 2024-01-22 23:33:39 +0800 + + liblzma: Add RISC-V BCJ filter. + + The new Filter ID is 0x0B. + + Thanks to Chien Wong for the initial version of the Filter, + the xz CLI updates, and the Autotools build system modifications. + + Thanks to Igor Pavlov for his many contributions to the design of + the filter. + + configure.ac | 4 +- + src/liblzma/api/lzma/bcj.h | 5 + + src/liblzma/common/filter_common.c | 9 + + src/liblzma/common/filter_decoder.c | 8 + + src/liblzma/common/filter_encoder.c | 10 + + src/liblzma/simple/Makefile.inc | 4 + + src/liblzma/simple/riscv.c | 688 ++++++++++++++++++++++++++++++++++++ + src/liblzma/simple/simple_coder.h | 9 + + src/xz/args.c | 7 + + 9 files changed, 742 insertions(+), 2 deletions(-) + +commit 5540f4329bbdb4deb4850d4af48b18ad074bba19 +Author: Jia Tan +Date: 2024-01-19 23:08:14 +0800 + + Docs: Update .xz file format specification to 1.2.0. + + The new RISC-V filter was added to the specification, in addition to + updating the specification URL. + + doc/xz-file-format.txt | 29 +++++++++++++++++------------ + 1 file changed, 17 insertions(+), 12 deletions(-) + +commit 22d86192f8cf00902a1f90ee2a83ca600794459b +Author: Jia Tan +Date: 2024-01-19 23:08:14 +0800 + + xz: Update website URLs in the man pages. + + src/xz/xz.1 | 6 +++--- + src/xzdec/xzdec.1 | 4 ++-- + 2 files changed, 5 insertions(+), 5 deletions(-) + +commit 6b63c4c6139fa1bb21b570521d3d2b4a608bc34d +Author: Jia Tan +Date: 2024-01-19 23:08:14 +0800 + + liblzma: Update website URL. + + dos/config.h | 2 +- + src/liblzma/api/lzma.h | 6 +++--- + 2 files changed, 4 insertions(+), 4 deletions(-) + +commit fce4758018f3a3589236f3fe7999fd9dd08c77e9 +Author: Jia Tan +Date: 2024-01-19 23:08:14 +0800 + + Docs: Update website URLs. + + .github/SECURITY.md | 2 +- + COPYING | 3 ++- + README | 4 ++-- + doc/faq.txt | 2 +- + doc/lzma-file-format.txt | 18 +++++++++--------- + windows/README-Windows.txt | 3 ++- + 6 files changed, 17 insertions(+), 15 deletions(-) + +commit c26812c5b2c8a2a47f43214afe6b0b840c73e4f5 +Author: Jia Tan +Date: 2024-01-19 23:08:14 +0800 + + Build: Update website URL. + + CMakeLists.txt | 2 +- + configure.ac | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit fbb3ce541ef79cad1710e88a27a5babb5f6f8e5b +Author: Lasse Collin +Date: 2024-01-11 15:01:50 +0200 + + liblzma: CRC: Add a comment to crc_x86_clmul.h about BUILDING_ macros. + + src/liblzma/check/crc_x86_clmul.h | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 4f518c1b6b7b7ce5dcefea81acd44d7a086a8882 +Author: Lasse Collin +Date: 2024-01-11 15:22:36 +0200 + + liblzma: CRC: Remove crc_always_inline, use lzma_always_inline instead. + + Now crc_simd_body() in crc_x86_clmul.h is only called once + in a translation unit, we no longer need to be so cautious + about ensuring the always-inline behavior. + + src/liblzma/check/crc_common.h | 20 -------------------- + src/liblzma/check/crc_x86_clmul.h | 2 +- + 2 files changed, 1 insertion(+), 21 deletions(-) + +commit 35c03ec6bf66f1b159964c9721a2dce0e2859b20 +Author: Lasse Collin +Date: 2024-01-11 14:39:46 +0200 + + liblzma: CRC: Update CLMUL comments to more generic wording. + + src/liblzma/check/crc32_fast.c | 16 ++++++++-------- + src/liblzma/check/crc64_fast.c | 10 +++++----- + 2 files changed, 13 insertions(+), 13 deletions(-) + +commit 66f080e8016129576536482ac377e2ecac7a2b90 +Author: Lasse Collin +Date: 2024-01-10 18:23:31 +0200 + + liblzma: Rename arch-specific CRC functions and macros. + + CRC_CLMUL was split to CRC_ARCH_OPTIMIZED and CRC_X86_CLMUL. + CRC_ARCH_OPTIMIZED is defined when an arch-optimized version is used. + Currently the x86 CLMUL implementations are the only arch-optimized + versions, and these also use the CRC_x86_CLMUL macro to tell when + crc_x86_clmul.h needs to be included. + + is_clmul_supported() was renamed to is_arch_extension_supported(). + crc32_clmul() and crc64_clmul() were renamed to + crc32_arch_optimized() and crc64_arch_optimized(). + This way the names make sense with arch-specific non-CLMUL + implementations as well. + + src/liblzma/check/crc32_fast.c | 13 +++++++------ + src/liblzma/check/crc64_fast.c | 13 +++++++------ + src/liblzma/check/crc_common.h | 9 ++++++--- + src/liblzma/check/crc_x86_clmul.h | 21 +++++++++++---------- + 4 files changed, 31 insertions(+), 25 deletions(-) + +commit 3dbed75b0b9c7087c76fe687acb5cf582cd57b99 +Author: Lasse Collin +Date: 2024-01-10 18:19:21 +0200 + + liblzma: Fix a comment in crc_common.h. + + src/liblzma/check/crc_common.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 419f55f9dfc2df8792902b8953d50690121afeea +Author: Lasse Collin +Date: 2023-10-20 23:35:10 +0300 + + liblzma: Avoid extern lzma_crc32_clmul() and lzma_crc64_clmul(). + + A CLMUL-only build will have the crcxx_clmul() inlined into + lzma_crcxx(). Previously a jump to the extern lzma_crcxx_clmul() + was needed. Notes about shared liblzma on ELF platforms: + + - On platforms that support ifunc and -fvisibility=hidden, this + was silly because CLMUL-only build would have that single extra + jump instruction of extra overhead. + + - On platforms that support neither -fvisibility=hidden nor linker + version script (liblzma*.map), jumping to lzma_crcxx_clmul() + would go via PLT so a few more instructions of overhead (still + not a big issue but silly nevertheless). + + There was a downside with static liblzma too: if an application only + needs lzma_crc64(), static linking would make the linker include the + CLMUL code for both CRC32 and CRC64 from crc_x86_clmul.o even though + the CRC32 code wouldn't be needed, thus increasing code size of the + executable (assuming that -ffunction-sections isn't used). + + Also, now compilers are likely to inline crc_simd_body() + even if they don't support the always_inline attribute + (or MSVC's __forceinline). Quite possibly all compilers + that build the code do support such an attribute. But now + it likely isn't a problem even if the attribute wasn't supported. + + Now all x86-specific stuff is in crc_x86_clmul.h. If other archs + The other archs can then have their own headers with their own + is_clmul_supported() and crcxx_clmul(). + + Another bonus is that the build system doesn't need to care if + crc_clmul.c is needed. + + is_clmul_supported() stays as inline function as it's not needed + when doing a CLMUL-only build (avoids a warning about unused function). + + CMakeLists.txt | 7 +- + configure.ac | 1 - + src/liblzma/check/Makefile.inc | 6 +- + src/liblzma/check/crc32_fast.c | 9 ++- + src/liblzma/check/crc64_fast.c | 9 ++- + src/liblzma/check/crc_common.h | 64 ---------------- + src/liblzma/check/{crc_clmul.c => crc_x86_clmul.h} | 86 ++++++++++++++++++---- + 7 files changed, 91 insertions(+), 91 deletions(-) + +commit e3833e297dfb5021a197bda34ba2a795e30aaf8a +Author: Lasse Collin +Date: 2023-10-21 00:06:52 +0300 + + liblzma: crc_clmul.c: Add crc_attr_target macro. + + This reduces the number of the complex #if directives. + + src/liblzma/check/crc_clmul.c | 30 ++++++++++++++++-------------- + 1 file changed, 16 insertions(+), 14 deletions(-) + +commit d164ac0e62904126f7920c25f9a2875c8cd28b97 +Author: Lasse Collin +Date: 2023-10-20 22:49:48 +0300 + + liblzma: Simplify existing cases with lzma_attr_no_sanitize_address. + + src/liblzma/check/crc_clmul.c | 12 +++--------- + 1 file changed, 3 insertions(+), 9 deletions(-) + +commit 9523c1300d22fa715765c181cf991d14d6112fb1 +Author: Lasse Collin +Date: 2023-10-20 21:53:35 +0300 + + liblzma: #define crc_attr_no_sanitize_address in crc_common.h. + + src/liblzma/check/crc_common.h | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit 93d144f0930821590524247bd174afd38003d7f0 +Author: Lasse Collin +Date: 2023-10-20 23:25:14 +0300 + + liblzma: CRC: Add empty lines. + + And remove one too. + + src/liblzma/check/crc32_fast.c | 2 ++ + src/liblzma/check/crc64_fast.c | 3 +++ + src/liblzma/check/crc_clmul.c | 1 - + 3 files changed, 5 insertions(+), 1 deletion(-) + +commit 0c7e854ffd27f1cec2e9b0e61601d6f90bfa10ae +Author: Lasse Collin +Date: 2023-10-20 23:19:33 +0300 + + liblzma: crc_clmul.c: Tidy up the location of MSVC pragma. + + It makes no difference in practice. + + src/liblzma/check/crc_clmul.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 15cf3f04f270d707a5c91cc0208b23b6db42b774 +Author: Lasse Collin +Date: 2023-12-20 21:16:24 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit cd64dd70d5665b6048829c45772d08606f44672e +Author: Lasse Collin +Date: 2023-12-20 21:15:16 +0200 + + liblzma: Use 8-byte method in memcmplen.h on ARM64. + + It requires fast unaligned access to 64-bit integers + and a fast instruction to count leading zeros in + a 64-bit integer (__builtin_ctzll()). This perhaps + should be enabled on some other archs too. + + Thanks to Chenxi Mao for the original patch: + https://github.com/tukaani-project/xz/pull/75 (the first commit) + According to the numbers there, this may improve encoding + speed by about 3-5 %. + + This enables the 8-byte method on MSVC ARM64 too which + should work but wasn't tested. + + src/liblzma/common/memcmplen.h | 18 ++++++++++-------- + 1 file changed, 10 insertions(+), 8 deletions(-) + +commit 12c90c00f05e19da3c0c91d8cd8e0d0d45965606 +Author: Lasse Collin +Date: 2023-12-20 21:01:06 +0200 + + liblzma: Check also for __clang__ in memcmplen.h. + + This change hopefully makes no practical difference as Clang + likely was detected via __GNUC__ or _MSC_VER already. + + src/liblzma/common/memcmplen.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 133c5851eb917c6d99d0b623c1689c8518e65f38 +Author: Jia Tan +Date: 2023-12-21 21:39:08 +0800 + + Translations: Update the French translation. + + po/fr.po | 632 +++++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 370 insertions(+), 262 deletions(-) + +commit 710cbc186cad0ac601c38bd6bf31167648a5581e +Author: Jia Tan +Date: 2023-12-21 16:39:53 +0800 + + xz: Add a comment to Capsicum sandbox setup. + + This comment is repeated in xzdec.c to help remind us why all the + capabilities are removed from stdin in certain situations. + + src/xz/file_io.c | 1 + + 1 file changed, 1 insertion(+) + +commit 4e1c695676bafbaecc9fb307f6ee94138ae72c12 +Author: Jia Tan +Date: 2023-12-20 22:19:19 +0800 + + Docs: Update --enable-sandbox option in INSTALL. + + xzdec now also uses the sandbox when its configured. + + INSTALL | 17 ++++++++++------- + 1 file changed, 10 insertions(+), 7 deletions(-) + +commit ebddf20214143a8e002ab897e95e880bb4c5ac44 +Author: Jia Tan +Date: 2023-12-20 22:39:13 +0800 + + CMake: Move sandbox detection outside of xz section. + + The sandbox is now enabled for xzdec as well, so it no longer belongs + in just the xz section. xz and xzdec are always built, except for older + MSVC versions, so there isn't a need to conditionally show the sandbox + configuration. CMake will do a little unecessary work on older MSVC + versions that can't build xz or xzdec, but this is a very small + downside. + + CMakeLists.txt | 178 +++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 98 insertions(+), 80 deletions(-) + +commit 5feb09266fd2928ec0a4dcb98c1dc7f053111316 +Author: Jia Tan +Date: 2023-12-20 22:43:44 +0800 + + Build: Allow sandbox to be configured for just xzdec. + + If xz is disabled, then xzdec can still use the sandbox. + + configure.ac | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit d74fb5f060b76db709b50f5fd37490394e52f975 +Author: Jia Tan +Date: 2023-12-19 21:18:28 +0800 + + xzdec: Add sandbox support for Pledge, Capsicum, and Landlock. + + A very strict sandbox is used when the last file is decompressed. The + likely most common use case of xzdec is to decompress a single file. + The Pledge sandbox is applied to the entire process with slightly more + relaxed promises, until the last file is processed. + + Thanks to Christian Weisgerber for the initial patch adding Pledge + sandboxing. + + src/xzdec/xzdec.c | 146 +++++++++++++++++++++++++++++++++++++++++++++++++++--- + 1 file changed, 139 insertions(+), 7 deletions(-) + +commit b34b6a9912d6165e34ba0db151b7f9941d2e06d5 +Author: Jia Tan +Date: 2023-12-20 21:31:34 +0800 + + liblzma: Initialize lzma_lz_encoder pointers with NULL. + + This fixes the recent change to lzma_lz_encoder that used memzero + instead of the NULL constant. On some compilers the NULL constant + (always 0) may not equal the NULL pointer (this only needs to guarentee + to not point to valid memory address). + + Later code compares the pointers to the NULL pointer so we must + initialize them with the NULL pointer instead of 0 to guarentee + code correctness. + + src/liblzma/lz/lz_encoder.c | 6 +++++- + 1 file changed, 5 insertions(+), 1 deletion(-) + +commit 183a62f0b540ff4d23cc19b2b6bc2525f0bd64df +Author: Jia Tan +Date: 2023-12-16 20:51:38 +0800 + + liblzma: Set all values in lzma_lz_encoder to NULL after allocation. + + The first member of lzma_lz_encoder doesn't necessarily need to be set + to NULL since it will always be set before anything tries to use it. + However the function pointer members must be set to NULL since other + functions rely on this NULL value to determine if this behavior is + supported or not. + + This fixes a somewhat serious bug, where the options_update() and + set_out_limit() function pointers are not set to NULL. This seems to + have been forgotten since these function pointers were added many years + after the original two (code() and end()). + + The problem is that by not setting this to NULL we are relying on the + memory allocation to zero things out if lzma_filters_update() is called + on a LZMA1 encoder. The function pointer for set_out_limit() is less + serious because there is not an API function that could call this in an + incorrect way. set_out_limit() is only called by the MicroLZMA encoder, + which must use LZMA1 where set_out_limit() is always set. Its currently + not possible to call set_out_limit() on an LZMA2 encoder at this time. + + So calling lzma_filters_update() on an LZMA1 encoder had undefined + behavior since its possible that memory could be manipulated so the + options_update member pointed to a different instruction sequence. + + This is unlikely to be a bug in an existing application since it relies + on calling lzma_filters_update() on an LZMA1 encoder in the first place. + For instance, it does not affect xz because lzma_filters_update() can + only be used when encoding to the .xz format. + + This is fixed by using memzero() to set all members of lzma_lz_encoder + to NULL after it is allocated. This ensures this mistake will not occur + here in the future if any additional function pointers are added. + + src/liblzma/lz/lz_encoder.c | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +commit 1a1bb381db7a20cf86cb45a350e5cca35224d017 +Author: Jia Tan +Date: 2023-12-16 20:30:55 +0800 + + liblzma: Tweak a comment. + + src/liblzma/lz/lz_encoder.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 55810780e04f759747b02683fb8020b8cd022a85 +Author: Jia Tan +Date: 2023-12-16 20:28:21 +0800 + + liblzma: Make parameter names in function definition match declaration. + + lzma_raw_encoder() and lzma_raw_encoder_init() used "options" as the + parameter name instead of "filters" (used by the declaration). "filters" + is more clear since the parameter represents the list of filters passed + to the raw encoder, each of which contains filter options. + + src/liblzma/common/filter_encoder.c | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +commit 5dad6f628af742bab826819760deb677597445f7 +Author: Jia Tan +Date: 2023-12-16 20:18:47 +0800 + + liblzma: Improve lzma encoder init function consistency. + + lzma_encoder_init() did not check for NULL options, but + lzma2_encoder_init() did. This is more of a code style improvement than + anything else to help make lzma_encoder_init() and lzma2_encoder_init() + more similar. + + src/liblzma/lzma/lzma_encoder.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit e1b1a9d6370b788bd6078952c6c201e12bc27cbf +Author: Jia Tan +Date: 2023-12-16 11:20:20 +0800 + + Docs: Update repository URL in Changelog. + + ChangeLog | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f9b82bc64a9405e486575c65c1729229eb0a8198 +Author: Jia Tan +Date: 2023-12-15 16:56:31 +0800 + + CI: Update Upload Artifact Action. + + .github/workflows/ci.yml | 2 +- + .github/workflows/windows-ci.yml | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit d0b24efe6cdc47db5b0fdf6306f70a2e0e63e49e +Author: Jia Tan +Date: 2023-12-07 21:48:07 +0800 + + Tests: Silence -Wsign-conversion warning on GCC version < 10. + + Since GCC version 10, GCC no longer complains about simple implicit + integer conversions with Arithmetic operators. + + For instance: + + uint8_t a = 5; + uint32_t b = a + 5; + + Give a warning on GCC 9 and earlier but this: + + uint8_t a = 5; + uint32_t b = (a + 5) * 2; + + Gives a warning with GCC 10+. + + tests/test_block_header.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 4a972a8ee3ed88ac14067c1d2f15b78988e5dae8 +Author: Jia Tan +Date: 2023-12-06 18:39:03 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit ee2f48350099201694a7586e41d7aa2f09fc74da +Author: Jia Tan +Date: 2023-12-06 18:30:25 +0800 + + Tests: Minor cleanups to OSS-Fuzz files. + + Most of these fixes are small typos and tweaks. A few were caused by bad + advice from me. Here is the summary of what is changed: + + - Author line edits + + - Small comment changes/additions + + - Using the return value in the error messages in the fuzz targets' + coder initialization code + + - Removed fuzz_encode_stream.options. This set a max length, which may + prevent some worthwhile code paths from being properly exercised. + + - Removed the max_len option from fuzz_decode_stream.options for the + same reason as fuzz_encode_stream. The alone decoder fuzz target still + has this restriction. + + - Altered the dictionary contents for fuzz_lzma.dict. Instead of keeping + the properties static and varying the dictionary size, the properties + are varied and the dictionary size is kept small. The dictionary size + doesn't have much impact on the code paths but the properties do. + + Closes: https://github.com/tukaani-project/xz/pull/73 + + tests/ossfuzz/Makefile | 3 ++ + tests/ossfuzz/config/fuzz_decode_stream.options | 1 - + tests/ossfuzz/config/fuzz_lzma.dict | 34 +++++++++++----------- + tests/ossfuzz/fuzz_common.h | 16 +++++------ + tests/ossfuzz/fuzz_decode_alone.c | 15 +++++----- + tests/ossfuzz/fuzz_decode_stream.c | 15 +++++----- + tests/ossfuzz/fuzz_encode_stream.c | 38 +++++++++++++++---------- + 7 files changed, 66 insertions(+), 56 deletions(-) + +commit 483bb90eec7c83e1c2bcd06287714afd62d8c17d +Author: Maksym Vatsyk +Date: 2023-12-05 16:31:09 +0100 + + Tests: Add fuzz_encode_stream ossfuzz target. + + This fuzz target handles .xz stream encoding. The first byte of input + is used to dynamically set the preset level in order to increase the + fuzz coverage of complex critical code paths. + + tests/ossfuzz/config/fuzz_encode_stream.options | 2 + + tests/ossfuzz/fuzz_encode_stream.c | 79 +++++++++++++++++++++++++ + 2 files changed, 81 insertions(+) + +commit 7ca8c9869df82756c3128c4fcf1058da4d18aa48 +Author: Maksym Vatsyk +Date: 2023-12-04 17:23:24 +0100 + + Tests: Add fuzz_decode_alone OSS-Fuzz target + + This fuzz target that handles LZMA alone decoding. A new fuzz + dictionary .dict was also created with common LZMA header values to + help speed up the discovery of valid headers. + + tests/ossfuzz/config/fuzz_decode_alone.options | 3 ++ + tests/ossfuzz/config/fuzz_lzma.dict | 22 ++++++++++++++ + tests/ossfuzz/fuzz_decode_alone.c | 41 ++++++++++++++++++++++++++ + 3 files changed, 66 insertions(+) + +commit 37581a77ad5a49615325b1d1925fdc402b1e1d5a +Author: Maksym Vatsyk +Date: 2023-12-04 17:21:29 +0100 + + Tests: Update OSS-Fuzz Makefile. + + All .c files can be built as separate fuzz targets. This simplifies + the Makefile by allowing us to use wildcards instead of having a + Makefile target for each fuzz target. + + tests/ossfuzz/Makefile | 13 +++++++++---- + 1 file changed, 9 insertions(+), 4 deletions(-) + +commit 28ce6a1c2a74866c51f7996a6869679c236d3c94 +Author: Maksym Vatsyk +Date: 2023-12-04 17:20:08 +0100 + + Tests: Move common OSS-Fuzz target code to .h file. + + tests/ossfuzz/fuzz_common.h | 56 ++++++++++++++++++++++++++++++++++++ + tests/ossfuzz/fuzz_decode_stream.c | 59 ++++++++++---------------------------- + 2 files changed, 71 insertions(+), 44 deletions(-) + +commit bf0521ea1591c25b9d510c1b8be86073e9d847c6 +Author: Maksym Vatsyk +Date: 2023-12-04 17:18:20 +0100 + + Tests: Rename OSS-Fuzz files. + + tests/ossfuzz/config/fuzz.options | 2 -- + tests/ossfuzz/config/fuzz_decode_stream.options | 3 +++ + tests/ossfuzz/config/{fuzz.dict => fuzz_xz.dict} | 0 + tests/ossfuzz/{fuzz.c => fuzz_decode_stream.c} | 0 + 4 files changed, 3 insertions(+), 2 deletions(-) + +commit 685094b8e1c1aa1bf934de0366ca42ef599d25f7 +Author: Jia Tan +Date: 2023-11-30 23:10:43 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 3b3023e00b0071e10f589bbc3674e0ec432b8add +Author: Kian-Meng Ang +Date: 2023-11-30 23:01:19 +0800 + + Tests: Fix typos + + tests/test_index.c | 2 +- + tests/test_lzip_decoder.c | 4 ++-- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 424d46ead8cbc0da57f406b76926ec4ed47437f5 +Author: Kian-Meng Ang +Date: 2023-11-30 22:59:47 +0800 + + xz: Fix typo + + src/xz/file_io.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 35558adf9c45e5597f2c8dbd969885dd484038d2 +Author: Jia Tan +Date: 2023-11-30 20:41:00 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit fd170e8557727bed6bec0518c16415064d972e4e +Author: Jia Tan +Date: 2023-11-22 21:20:12 +0800 + + CI: Test musl libc builds on Ubuntu runner. + + .github/workflows/ci.yml | 19 +++++++++++++++++-- + 1 file changed, 17 insertions(+), 2 deletions(-) + +commit db2b4aa068a492c0013279a4ed43803e8ff9bb3e +Author: Jia Tan +Date: 2023-11-22 21:12:15 +0800 + + CI: Allow ci_build.sh to set a different C compiler. + + build-aux/ci_build.sh | 11 ++++++++++- + 1 file changed, 10 insertions(+), 1 deletion(-) + +commit ff7badef53c2cd698d4b72b945f34dfd0835e13c +Author: Jia Tan +Date: 2023-11-24 21:19:12 +0800 + + CMake: Use consistent indentation with check_c_source_compiles(). + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit d4af167570f2c14b002ee18a39d5b1e7e5a892b1 +Author: Jia Tan +Date: 2023-11-22 20:33:36 +0800 + + CMake: Change __attribute__((__ifunc__())) detection. + + This renames ALLOW_ATTR_IFUNC to USE_ATTR_IFUNC and applies the ifunc + detection changes that were made to the Autotools build. + + Fixes: https://github.com/tukaani-project/xz/issues/70 + + CMakeLists.txt | 53 +++++++++++++++++++++++++++++++++++++++++++++-------- + 1 file changed, 45 insertions(+), 8 deletions(-) + +commit 20ecee40a0053fd16371ef0628046bf45e548d72 +Author: Jia Tan +Date: 2023-11-24 20:19:11 +0800 + + Docs: Update INSTALL for --enable_ifunc change. + + INSTALL | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit ffb456593d695d70052a2f71c7a2e6269217d194 +Author: Jia Tan +Date: 2023-11-21 20:56:55 +0800 + + Build: Change --enable-ifunc handling. + + Some compilers support __attribute__((__ifunc__())) even though the + dynamic linker does not. The compiler is able to create the binary + but it will fail on startup. So it is not enough to just test if + the attribute is supported. + + The default value for enable_ifunc is now auto, which will attempt + to compile a program using __attribute__((__ifunc__())). There are + additional checks in this program if glibc is being used or if it + is running on FreeBSD. + + Setting --enable-ifunc will skip this test and always enable + __attribute__((__ifunc__())), even if is not supported. + + configure.ac | 61 +++++++++++++++++++++++++++++++++++++++++++----------------- + 1 file changed, 44 insertions(+), 17 deletions(-) + +commit 12b89bcc9915090eb42ae638e565af44b6832a23 +Author: Lasse Collin +Date: 2023-11-23 17:39:10 +0200 + + xz: Tweak a comment. + + src/xz/util.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 2ab2e4b5a542eab93902985ce4e642719a8b7a4e +Author: Jia Tan +Date: 2023-11-23 22:13:39 +0800 + + xz: Use is_tty() in message.c. + + src/xz/message.c | 7 +------ + 1 file changed, 1 insertion(+), 6 deletions(-) + +commit 584e3a258f32d579b1d07f99b4dc6e856c10ac7e +Author: Jia Tan +Date: 2023-11-23 22:04:35 +0800 + + xz: Create separate is_tty() function. + + The new is_tty() will report if a file descriptor is a terminal or not. + On POSIX systems, it is a wrapper around isatty(). However, the native + Windows implementation of isatty() will return true for all character + devices, not just terminals. So is_tty() has a special case for Windows + so it can use alternative Windows API functions to determine if a file + descriptor is a terminal. + + This fixes a bug with MSVC and MinGW-w64 builds that refused to read from + or write to non-terminal character devices because xz thought it was a + terminal. For instance: + + xz foo -c > /dev/null + + would fail because /dev/null was assumed to be a terminal. + + src/xz/util.c | 30 +++++++++++++++++++++++------- + src/xz/util.h | 14 ++++++++++++++ + 2 files changed, 37 insertions(+), 7 deletions(-) + +commit 6b05f827f50e686537e9a23c49c5aa4c0aa6b23d +Author: Jia Tan +Date: 2023-11-22 20:39:41 +0800 + + tuklib_integer: Fix typo discovered by codespell. + + Based on internet dictionary searches, 'choise' is an outdated spelling + of 'choice'. + + src/common/tuklib_integer.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 659aca0d695807c0762d4101765189e4e33d1e2c +Author: Lasse Collin +Date: 2023-11-17 19:35:19 +0200 + + xz: Move the check for --suffix with --format=raw a few lines earlier. + + Now it reads from argv[] instead of args->arg_names. + + src/xz/args.c | 44 ++++++++++++++++++++++---------------------- + 1 file changed, 22 insertions(+), 22 deletions(-) + +commit ca278eb2b7f5a4940f5ab18955297b398d423824 +Author: Jia Tan +Date: 2023-11-17 20:35:11 +0800 + + Tests: Create test_suffix.sh. + + This tests some complicated interactions with the --suffix= option. + The suffix option must be used with --format=raw, but can optionally + be used to override the default .xz suffix. + + This test also verifies some recent bugs have been correctly solved + and to hopefully avoid further regressions in the future. + + tests/Makefile.am | 2 + + tests/test_suffix.sh | 189 +++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 191 insertions(+) + +commit 2a732aba22da1b0d4a1241cb32280ed010ba03ce +Author: Jia Tan +Date: 2023-11-17 20:19:26 +0800 + + xz: Fix a bug with --files and --files0 in raw mode without a suffix. + + The following command caused a segmentation fault: + + xz -Fraw --lzma1 --files=foo + + when foo was a valid file. The usage of --files or --files0 was not + being checked when compressing or decompressing in raw mode without a + suffix. The suffix checking code was meant to validate that all files + to be processed are "-" (if not writing to standard out), meaning the + data is only coming from standard in. In this case, there were no file + names to check since --files and --files0 store their file name in a + different place. + + Later code assumed the suffix was set and caused a segmentation fault. + Now, the above command results in an error. + + src/xz/args.c | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 299920bab9ae258a247366339264e8aefca9e3ce +Author: Jia Tan +Date: 2023-11-17 20:04:58 +0800 + + Tests: Fix typo in a comment. + + tests/test_files.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f481523baac946fa3bc13d79186ffaf0c0b818a7 +Author: Jia Tan +Date: 2023-11-15 23:40:13 +0800 + + xz: Refactor suffix test with raw format. + + The previous version set opt_stdout, but this caused an issue with + copying an input file to standard out when decompressing an unknown file + type. The following needs to result in an error: + + echo foo | xz -df + + since -c, --stdout is not used. This fixes the previous error by not + setting opt_stdout. + + src/xz/args.c | 38 +++++++++++++------------------------- + 1 file changed, 13 insertions(+), 25 deletions(-) + +commit 837ea40b1c9d4998cac4500b55171bf33e0c31a6 +Author: Jia Tan +Date: 2023-11-14 20:27:46 +0800 + + xz: Move suffix check after stdout mode is detected. + + This fixes a bug introduced in cc5aa9ab138beeecaee5a1e81197591893ee9ca0 + when the suffix check was initially moved. This caused a situation that + previously worked: + + echo foo | xz -Fraw --lzma1 | wc -c + + to fail because the old code knew that this would write to standard out + so a suffix was not needed. + + src/xz/args.c | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit d4f4a4d040ef47a5e82dffd0f067e92716606ddf +Author: Jia Tan +Date: 2023-11-14 20:27:04 +0800 + + xz: Detect when all data will be written to standard out earlier. + + If the -c, --stdout argument is not used, then we can still detect when + the data will be written to standard out if all of the provided + filenames are "-" (denoting standard in) or if no filenames are + provided. + + src/xz/args.c | 21 +++++++++++++++++++++ + 1 file changed, 21 insertions(+) + +commit 2ade7246e7ba729a91460d2fab0f4c7b89d3998b +Author: Jia Tan +Date: 2023-11-09 01:21:53 +0800 + + liblzma: Add missing comments to lz_encoder.h. + + src/liblzma/lz/lz_encoder.h | 6 +++++- + 1 file changed, 5 insertions(+), 1 deletion(-) + +commit 5fe1450603dc625340b8b7866fb4a83ff748ad06 +Author: Jia Tan +Date: 2023-11-01 20:18:30 +0800 + + Add NEWS for 5.4.5. + + NEWS | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 74 insertions(+) + +commit 46007049cd42e606543dbe650feb17bdf4469c29 +Author: Lasse Collin +Date: 2023-10-31 21:41:09 +0200 + + liblzma: Fix compilation of fastpos_tablegen.c. + + The macro lzma_attr_visibility_hidden has to be defined to make + fastpos.h usable. The visibility attribute is irrelevant to + fastpos_tablegen.c so simply #define the macro to an empty value. + + fastpos_tablegen.c is never built by the included build systems + and so the problem wasn't noticed earlier. It's just a standalone + program for generating fastpos_table.c. + + Fixes: https://github.com/tukaani-project/xz/pull/69 + Thanks to GitHub user Jamaika1. + + src/liblzma/lzma/fastpos_tablegen.c | 2 ++ + 1 file changed, 2 insertions(+) + +commit 148e20607e95781558bdfc823ecba07b7af4b590 +Author: Jia Tan +Date: 2023-10-31 21:51:40 +0800 + + Build: Fix text wrapping in an output message. + + configure.ac | 9 +++++---- + 1 file changed, 5 insertions(+), 4 deletions(-) + +commit 8c36ab79cbf23104ce7a3d533d5ac98cd492e57c +Author: Lasse Collin +Date: 2023-10-30 18:09:53 +0200 + + liblzma: Add a note why crc_always_inline exists for now. + + Solaris Studio is a possible example (not tested) which + supports the always_inline attribute but might not get + detected by the common.h #ifdefs. + + src/liblzma/check/crc_common.h | 5 +++++ + 1 file changed, 5 insertions(+) + +commit e7a86b94cd247435ac96bc79ba528b690b9ca388 +Author: Lasse Collin +Date: 2023-10-22 17:59:11 +0300 + + liblzma: Use lzma_always_inline in memcmplen.h. + + src/liblzma/common/memcmplen.h | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + +commit dcfe5632992fb7f06f921da13fcdd84f83d0d285 +Author: Lasse Collin +Date: 2023-10-30 17:43:03 +0200 + + liblzma: #define lzma_always_inline in common.h. + + src/liblzma/common/common.h | 17 +++++++++++++++++ + 1 file changed, 17 insertions(+) + +commit 41113fe30a47f6fd3e30cb4494dd538e86212edf +Author: Lasse Collin +Date: 2023-10-22 17:15:32 +0300 + + liblzma: Use lzma_attr_visibility_hidden on private extern declarations. + + These variables are internal to liblzma and not exposed in the API. + + src/liblzma/check/check.h | 7 +++++++ + src/liblzma/common/stream_flags_common.h | 3 +++ + src/liblzma/lz/lz_encoder_hash.h | 1 + + src/liblzma/lzma/fastpos.h | 1 + + src/liblzma/rangecoder/price.h | 1 + + 5 files changed, 13 insertions(+) + +commit a2f5ca706acc6f7715b8d260a8c6ed50d7717478 +Author: Lasse Collin +Date: 2023-10-22 17:08:39 +0300 + + liblzma: #define lzma_attr_visibility_hidden in common.h. + + In ELF shared libs: + + -fvisibility=hidden affects definitions of symbols but not + declarations.[*] This doesn't affect direct calls to functions + inside liblzma as a linker can replace a call to lzma_foo@plt + with a call directly to lzma_foo when -fvisibility=hidden is used. + + [*] It has to be like this because otherwise every installed + header file would need to explictly set the symbol visibility + to default. + + When accessing extern variables that aren't defined in the + same translation unit, compiler assumes that the variable has + the default visibility and thus indirection is needed. Unlike + function calls, linker cannot optimize this. + + Using __attribute__((__visibility__("hidden"))) with the extern + variable declarations tells the compiler that indirection isn't + needed because the definition is in the same shared library. + + About 15+ years ago, someone told me that it would be good if + the CRC tables would be defined in the same translation unit + as the C code of the CRC functions. While I understood that it + could help a tiny amount, I didn't want to change the code because + a separate translation unit for the CRC tables was needed for the + x86 assembly code anyway. But when visibility attributes are + supported, simply marking the extern declaration with the + hidden attribute will get identical result. When there are only + a few affected variables, this is trivial to do. I wish I had + understood this back then already. + + src/liblzma/common/common.h | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit 2c7ee92e44e1e66f0a427555233eb22c78f6c4f8 +Author: Lasse Collin +Date: 2023-09-30 22:54:28 +0300 + + liblzma: Refer to MinGW-w64 instead of MinGW in the API headers. + + MinGW (formely a MinGW.org Project, later the MinGW.OSDN Project + at ) has GCC 9.2.0 as the + most recent GCC package (released 2021-02-02). The project might + still be alive but majority of people have switched to MinGW-w64. + Thus it seems clearer to refer to MinGW-w64 in our API headers too. + Building with MinGW is likely to still work but I haven't tested it + in the recent years. + + src/liblzma/api/lzma.h | 4 ++-- + src/liblzma/api/lzma/version.h | 2 +- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 597f49b61475438a43a417236989b2acc968a686 +Author: Lasse Collin +Date: 2023-09-27 00:58:17 +0300 + + CMake: Use -D_FILE_OFFSET_BITS=64 if (and only if) needed. + + A CMake option LARGE_FILE_SUPPORT is created if and only if + -D_FILE_OFFSET_BITS=64 affects sizeof(off_t). + + This is needed on many 32-bit platforms and even with 64-bit builds + with MinGW-w64 to get support for files larger than 2 GiB. + + CMakeLists.txt | 7 ++++- + cmake/tuklib_large_file_support.cmake | 52 +++++++++++++++++++++++++++++++++++ + 2 files changed, 58 insertions(+), 1 deletion(-) + +commit 1bc548b8210366e44ba35b0b11577a8e328c1228 +Author: Lasse Collin +Date: 2023-09-30 02:14:25 +0300 + + CMake: Generate and install liblzma.pc if not using MSVC. + + Autotools based build uses -pthread and thus adds it to Libs.private + in liblzma.pc. CMake doesn't use -pthread at all if pthread functions + are available in libc so Libs.private doesn't get -pthread either. + + CMakeLists.txt | 21 +++++++++++++++++++++ + 1 file changed, 21 insertions(+) + +commit 2add71966f891d315105d6245f724ed4f43a4eff +Author: Lasse Collin +Date: 2023-09-30 01:13:13 +0300 + + CMake: Rearrange the PACKAGE_ variables. + + The windres workaround now replaces spaces with \x20 so + the package name isn't repeated. + + These changes will help with creation of liblzma.pc. + + CMakeLists.txt | 26 +++++++++++++++----------- + 1 file changed, 15 insertions(+), 11 deletions(-) + +commit a7d1b2825c49dc83f1910eeb8ba0f1dfbd886d91 +Author: Lasse Collin +Date: 2023-09-29 20:46:11 +0300 + + liblzma: Add Cflags.private to liblzma.pc.in for MSYS2. + + It properly adds -DLZMA_API_STATIC when compiling code that + will be linked against static liblzma. Having it there on + systems other than Windows does no harm. + + See: https://www.msys2.org/docs/pkgconfig/ + + src/liblzma/liblzma.pc.in | 1 + + 1 file changed, 1 insertion(+) + +commit 80e0750e3996c1c659e972ce9cf789ca2e99f702 +Author: Lasse Collin +Date: 2023-09-27 22:46:20 +0300 + + CMake: Create liblzma.def when building liblzma.dll with MinGW-w64. + + CMakeLists.txt | 20 ++++++++++++++++++++ + cmake/remove-ordinals.cmake | 26 ++++++++++++++++++++++++++ + 2 files changed, 46 insertions(+) + +commit 08d12595f486890cf601b87f36ee0ddbce57728e +Author: Lasse Collin +Date: 2023-10-26 21:44:42 +0300 + + CMake: Change one CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_LIST_DIR. + + In this case they have identical values. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit e67aaf698de75c73443a5ec786781cbf2034461d +Author: Lasse Collin +Date: 2023-10-01 19:10:57 +0300 + + CMake/Windows: Fix the import library filename. + + Both PREFIX and IMPORT_PERFIX have to be set to "" to get + liblzma.dll and liblzma.dll.a. + + CMakeLists.txt | 1 + + 1 file changed, 1 insertion(+) + +commit 88588b1246d8c26ffbc138b3e5c413c5f14c3179 +Author: Lasse Collin +Date: 2023-10-25 19:13:25 +0300 + + Build: Detect -fsanitize= in CFLAGS and incompatible build options. + + Now configure will fail if -fsanitize= is found in CFLAGS + and sanitizer-incompatible ifunc or Landlock sandboxing + would be used. These are incompatible with one or more sanitizers. + It's simpler to reject all -fsanitize= uses instead of trying to + pass those that might not cause problems. + + CMake-based build was updated similarly. It lets the configuration + finish (SEND_ERROR instead of FATAL_ERROR) so that both error + messages can be seen at once. + + CMakeLists.txt | 29 +++++++++++++++++++++++++++++ + configure.ac | 37 +++++++++++++++++++++++++++++++++---- + 2 files changed, 62 insertions(+), 4 deletions(-) + +commit 5e3d890f8862a7d4fbef5e38e11b6c9fbd98f468 +Author: Jia Tan +Date: 2023-10-24 00:50:08 +0800 + + CI: Disable sandboxing in fsanitize=address,undefined job. + + The sandboxing on Linux now supports Landlock, which restricts all + supported filesystem actions after xz opens the files it needs. The + sandbox is only enabled when one file is input and we are writing to + standard out. With fsanitize=address,undefined, the instrumentation + needs to read additional files after the sandbox is in place. This + forces all xz based test to fail, so the sandbox must instead be + disabled. + + .github/workflows/ci.yml | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit b1408987ea832e2760e478ae960a636df17a1363 +Author: Jia Tan +Date: 2023-10-24 00:15:39 +0800 + + CI: Allow disabling the sandbox in ci_build.sh. + + build-aux/ci_build.sh | 5 ++++- + 1 file changed, 4 insertions(+), 1 deletion(-) + +commit 91c435cf1c7a1e893706d4d716dfd361621ed824 +Author: Lasse Collin +Date: 2023-10-11 19:47:44 +0300 + + CMake: Don't shadow the cache entry ENABLE_THREADS with a normal variable. + + Using set(ENABLE_THREADS "posix") is confusing because it sets + a new normal variable and leaves the cache entry with the same + name unchanged. The intent wasn't to change the cache entry so + this switches to a different variable name. + + CMakeLists.txt | 10 +++++++--- + 1 file changed, 7 insertions(+), 3 deletions(-) + +commit fa1609eb9393ecd30decfed4891c907829f06710 +Author: Lasse Collin +Date: 2023-10-09 22:28:49 +0300 + + Docs: Update INSTALL about sandboxing support. + + INSTALL | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit 8276c7f41c671eee4aa3239490658b23dcfd3021 +Author: Lasse Collin +Date: 2023-10-09 22:07:52 +0300 + + xz: Support basic sandboxing with Linux Landlock (ABI versions 1-3). + + It is enabled only when decompressing one file to stdout, + similar to how Capsicum is used. + + Landlock was added in Linux 5.13. + + CMakeLists.txt | 12 +++++++++++- + configure.ac | 11 ++++++++--- + src/xz/file_io.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + src/xz/main.c | 19 +++++++++++++++++++ + src/xz/private.h | 3 ++- + 5 files changed, 98 insertions(+), 5 deletions(-) + +commit 3a1e9fd031b9320d769d63b503ef4e82e1b6ea8c +Author: Lasse Collin +Date: 2023-10-09 21:12:31 +0300 + + CMake: Edit threading related messages. + + It's mostly to change from "thread method" to "threading method". + + CMakeLists.txt | 19 ++++++++++--------- + 1 file changed, 10 insertions(+), 9 deletions(-) + +commit bf011352528ae3539ea7b780b45b96736ee57a99 +Author: Lasse Collin +Date: 2023-10-09 20:59:24 +0300 + + CMake: Use FATAL_ERROR if user-supplied options aren't understood. + + This way typos are caught quickly and compounding error messages + are avoided (a single typo could cause more than one error). + + This keeps using SEND_ERROR when the system is lacking a feature + (like threading library or sandboxing method). This way the whole + configuration log will be generated in case someone wishes to + report a problem upstream. + + CMakeLists.txt | 28 ++++++++++++++-------------- + 1 file changed, 14 insertions(+), 14 deletions(-) + +commit 3f53870c249945d657ca3d75e0993e6267d71f75 +Author: Lasse Collin +Date: 2023-10-09 18:37:32 +0300 + + CMake: Add sandboxing support. + + CMakeLists.txt | 50 +++++++++++++++++++++++++++++++++++++++++++++++++- + 1 file changed, 49 insertions(+), 1 deletion(-) + +commit 2e2cd11535ad77364cf021297e0b3f162fa3a3d0 +Author: Lasse Collin +Date: 2023-10-09 18:13:08 +0300 + + Simplify detection of Capsicum support. + + This removes support for FreeBSD 10.0 and 10.1 which used + instead of . Support for + FreeBSD 10.1 ended on 2016-12-31. So now FreeBSD >= 10.2 is + required to enable Capsicum support. + + This also removes support for Capsicum on Linux (libcaprights) + which seems to have been unmaintained since 2017 and Linux 4.11: + https://github.com/google/capsicum-linux + + configure.ac | 4 +-- + m4/ax_check_capsicum.m4 | 85 ------------------------------------------------- + src/xz/Makefile.am | 2 +- + src/xz/file_io.c | 14 +++----- + src/xz/private.h | 2 +- + 5 files changed, 9 insertions(+), 98 deletions(-) + +commit c57858b60e186d020b2dbaf7aabd9b32c71da824 +Author: Lasse Collin +Date: 2023-09-25 01:46:36 +0300 + + xz/Windows: Allow clock_gettime with POSIX threads. + + If winpthreads are used for threading, it's OK to use clock_gettime() + from winpthreads too. + + src/xz/mytime.c | 9 ++++++--- + 1 file changed, 6 insertions(+), 3 deletions(-) + +commit dd32f628bb5541ef4e8ce66966ef456a1934084c +Author: Lasse Collin +Date: 2023-09-25 01:39:26 +0300 + + mythread.h: Make MYTHREAD_POSIX compatible with MinGW-w64's winpthreads. + + This might be almost useless but it doesn't need much extra code either. + + src/common/mythread.h | 23 ++++++++++++++++++++++- + 1 file changed, 22 insertions(+), 1 deletion(-) + +commit 680e52cdd086e92691d8a0bca2c98815565f60ca +Author: Lasse Collin +Date: 2023-09-23 03:06:36 +0300 + + CMake: Check for clock_gettime() even on Windows. + + This mirrors configure.ac although currently MinGW-w64 builds + don't use clock_gettime() even if it is found. + + CMakeLists.txt | 44 +++++++++++++++++++++----------------------- + 1 file changed, 21 insertions(+), 23 deletions(-) + +commit 1c1a8c3ee4dad0064dbe63b8dbc4ac4bc679f419 +Author: Lasse Collin +Date: 2023-09-23 03:23:32 +0300 + + Build: Check for clock_gettime() even if not using POSIX threads. + + See the new comment in the code. + + This also makes the check for clock_gettime() run with MinGW-w64 + with which we don't want to use clock_gettime(). The previous + commit already took care of this situation. + + configure.ac | 31 ++++++++++++++++++------------- + 1 file changed, 18 insertions(+), 13 deletions(-) + +commit 46fd991cd2808ef62554853864c946232e7547f0 +Author: Lasse Collin +Date: 2023-09-24 22:58:53 +0300 + + xz/Windows: Ensure that clock_gettime() isn't used with MinGW-w64. + + This commit alone doesn't change anything in the real-world: + + - configure.ac currently checks for clock_gettime() only + when using pthreads. + + - CMakeLists.txt doesn't check for clock_gettime() on Windows. + + So clock_gettime() wasn't used with MinGW-w64 before either. + + clock_gettime() provides monotonic time and it's better than + gettimeofday() in this sense. But clock_gettime() is defined + in winpthreads, and liblzma or xz needs nothing else from + winpthreads. By avoiding clock_gettime(), we avoid the dependency on + libwinpthread-1.dll or the need to link against the static version. + + As a bonus, GetTickCount64() and MinGW-w64's gettimeofday() can be + faster than clock_gettime(CLOCK_MONOTONIC, &tv). The resolution + is more than good enough for the progress indicator in xz. + + src/xz/mytime.c | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +commit cdb4d91f2464b50c985ef7b9517314ea237ddda7 +Author: Lasse Collin +Date: 2023-09-24 00:21:22 +0300 + + xz/Windows: Use GetTickCount64() with MinGW-w64 if using Vista threads. + + src/xz/mytime.c | 14 +++++++++++--- + 1 file changed, 11 insertions(+), 3 deletions(-) + +commit 988e09f27b9b04a43d45d10f92782e0092ee27a9 +Author: Jia Tan +Date: 2023-10-20 19:17:46 +0800 + + liblzma: Move is_clmul_supported() back to crc_common.h. + + This partially reverts creating crc_clmul.c + (8c0f9376f58c0696d5d6719705164d35542dd891) where is_clmul_supported() + was moved, extern'ed, and renamed to lzma_is_clmul_supported(). This + caused a problem when the function call to lzma_is_clmul_supported() + results in a call through the PLT. ifunc resolvers run very early in + the dynamic loading sequence, so the PLT may not be setup properly at + this point. Whether the PLT is used or not for + lzma_is_clmul_supported() depened upon the compiler-toolchain used and + flags. + + In liblzma compiled with GCC, for instance, GCC will go through the PLT + for function calls internal to liblzma if the version scripts and + symbol visibility hiding are not used. If lazy-binding is disabled, + then it would have made any program linked with liblzma fail during + dynamic loading in the ifunc resolver. + + src/liblzma/check/crc32_fast.c | 2 +- + src/liblzma/check/crc64_fast.c | 2 +- + src/liblzma/check/crc_clmul.c | 45 ------------------------------------ + src/liblzma/check/crc_common.h | 52 +++++++++++++++++++++++++++++++++++++++--- + 4 files changed, 51 insertions(+), 50 deletions(-) + +commit 105c7ca90d4152942e0798580a37f736d02faa22 +Author: Jia Tan +Date: 2023-10-19 16:23:32 +0800 + + Build: Remove check for COND_CHECK_CRC32 in check/Makefile.inc. + + Currently crc32 is always enabled, so COND_CHECK_CRC32 must always be + set. Because of this, it makes the recent change to conditionally + compile check/crc_clmul.c appear wrong since that file has CLMUL + implementations for both CRC32 and CRC64. + + src/liblzma/check/Makefile.inc | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 139757170468f0f1fafdf0a8ffa74363d1ea1d0c +Author: Jia Tan +Date: 2023-10-19 16:09:01 +0800 + + CMake: Add ALLOW_CLMUL_CRC option to enable/disable CLMUL. + + The option is enabled by default, but will only be visible to a user + listing cache variables or using a CMake GUI application if the + immintrin.h header file is found. + + This mirrors our Autotools build --disable-clmul-crc functionality. + + CMakeLists.txt | 44 +++++++++++++++++++++++++------------------- + 1 file changed, 25 insertions(+), 19 deletions(-) + +commit c60b25569d414bb73b705977a4dd342f8f9f1965 +Author: Jia Tan +Date: 2023-10-19 00:22:50 +0800 + + liblzma: Fix -fsanitize=address failure with crc_clmul functions. + + After forcing crc_simd_body() to always be inlined it caused + -fsanitize=address to fail for lzma_crc32_clmul() and + lzma_crc64_clmul(). The __no_sanitize_address__ attribute was added + to lzma_crc32_clmul() and lzma_crc64_clmul(), but not removed from + crc_simd_body(). ASAN and inline functions behavior has changed over + the years for GCC specifically, so while strictly required we will + keep __attribute__((__no_sanitize_address__)) on crc_simd_body() in + case this becomes a requirement in the future. + + Older GCC versions refuse to inline a function with ASAN if the + caller and callee do not agree on sanitization flags + (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89124#c3). If the + function was forced to be inlined, it will not compile if the callee + function has __no_sanitize_address__ but the caller doesn't. + + src/liblzma/check/crc_clmul.c | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 9a78971261bc67622cbd7dae02f6966968ac1393 +Author: Lasse Collin +Date: 2023-10-14 20:16:13 +0300 + + tuklib_integer: Update the CMake test for fast unaligned access. + + cmake/tuklib_integer.cmake | 69 ++++++++++++++++++++++++++++++++++++---------- + 1 file changed, 54 insertions(+), 15 deletions(-) + +commit 2f81ac852bc5aafc91c8e2adc66b5114761703c4 +Author: Lasse Collin +Date: 2023-09-23 23:28:48 +0300 + + Build: Enabled unaligned access by default on PowerPC64LE and some RISC-V. + + PowerPC64LE wasn't tested but it seems like a safe change. + POWER8 supports unaligned access in little endian mode. Testing + on godbolt.org shows that GCC uses unaligned access by default. + + The RISC-V macro __riscv_misaligned_fast is very new and not + in any stable compiler release yet. + + Documentation in INSTALL was updated to match. + + Documentation about an autodetection bug when using ARM64 GCC + with -mstrict-align was added to INSTALL. + + CMake files weren't updated yet. + + INSTALL | 39 +++++++++++++++++++++++++++++++++++++-- + m4/tuklib_integer.m4 | 34 +++++++++++++++++++++++++++------- + 2 files changed, 64 insertions(+), 9 deletions(-) + +commit c8f715f1bca4c30db814fcf1fd2fe88b8992ede2 +Author: Lasse Collin +Date: 2023-10-14 17:56:59 +0300 + + tuklib_integer: Revise unaligned reads and writes on strict-align archs. + + In XZ Utils context this doesn't matter much because + unaligned reads and writes aren't used in hot code + when TUKLIB_FAST_UNALIGNED_ACCESS isn't #defined. + + src/common/tuklib_integer.h | 256 ++++++++++++++++++++++++++++++++------------ + 1 file changed, 189 insertions(+), 67 deletions(-) + +commit 6828242735cbf61b93d140383336e1e51a006f2d +Author: Lasse Collin +Date: 2023-09-23 02:21:49 +0300 + + tuklib_integer: Add missing write64be and write64le fallback functions. + + src/common/tuklib_integer.h | 34 ++++++++++++++++++++++++++++++++++ + 1 file changed, 34 insertions(+) + +commit 1c8884f0af28b3a4690bb573cdf3240a8ec73416 +Author: Jia Tan +Date: 2023-10-18 19:57:10 +0800 + + liblzma: Set the MSVC optimization fix to only cover lzma_crc64_clmul(). + + After testing a 32-bit Release build on MSVC, only lzma_crc64_clmul() + has the bug. crc_simd_body() and lzma_crc32_clmul() do not need the + optimizations disabled. + + src/liblzma/check/crc_clmul.c | 30 +++++++++++++++--------------- + 1 file changed, 15 insertions(+), 15 deletions(-) + +commit 5ce0f7a48bdf5c3b45430850a4487307afac6143 +Author: Lasse Collin +Date: 2023-10-18 14:30:00 +0300 + + liblzma: CRC_USE_GENERIC_FOR_SMALL_INPUTS cannot be used with ifunc. + + src/liblzma/check/crc_common.h | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit 27735380491bb5ce0d0f41d5244d89c1d0825f6b +Author: Lasse Collin +Date: 2023-10-17 21:53:11 +0300 + + liblzma: Include common.h in crc_common.h. + + crc_common.h depends on common.h. The headers include common.h except + when there is a reason to not do so. + + src/liblzma/check/crc_clmul.c | 1 - + src/liblzma/check/crc_common.h | 3 +++ + 2 files changed, 3 insertions(+), 1 deletion(-) + +commit e13b7947b92355c334edd594295d3a2c99c4bca1 +Author: Jia Tan +Date: 2023-10-18 01:23:26 +0800 + + liblzma: Add include guards to crc_common.h. + + src/liblzma/check/crc_common.h | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 40abd88afcc61a8157fcd12d78d491caeb8e12be +Author: Jia Tan +Date: 2023-10-18 22:50:25 +0800 + + liblzma: Add the crc_always_inline macro to crc_simd_body(). + + Forcing this to be inline has a significant speed improvement at the + cost of a few repeated instructions. The compilers tested on did not + inline this function since it is large and is used twice in the same + translation unit. + + src/liblzma/check/crc_clmul.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit a5966c276bd6fa975f0389f8a8dc61393de750b0 +Author: Jia Tan +Date: 2023-10-18 22:48:19 +0800 + + liblzma: Create crc_always_inline macro. + + This macro must be used instead of the inline keyword. On MSVC, it is + a replacement for __forceinline which is an MSVC specific keyword that + should not be used with inline (it will issue a warning if it is). + + It does not use a build system check to determine if + __attribute__((__always_inline__)) since all compilers that can use + CLMUL extensions (except the special case for MSVC) should support this + attribute. If this assumption is incorrect then it will result in a bug + report instead of silently producing slow code. + + src/liblzma/check/crc_common.h | 15 +++++++++++++++ + 1 file changed, 15 insertions(+) + +commit 96b663f67c0e738a99ba8f35d9f4ced9add74544 +Author: Jia Tan +Date: 2023-10-14 13:23:23 +0800 + + liblzma: Refactor CRC comments. + + A detailed description of the three dispatch methods was added. Also, + duplicated comments now only appear in crc32_fast.c or were removed from + both crc32_fast.c and crc64_fast.c if they appeared in crc_clmul.c. + + src/liblzma/check/crc32_fast.c | 64 +++++++++++++++++++++++++++++------------- + src/liblzma/check/crc64_fast.c | 61 ++++++---------------------------------- + 2 files changed, 53 insertions(+), 72 deletions(-) + +commit 8c0f9376f58c0696d5d6719705164d35542dd891 +Author: Jia Tan +Date: 2023-10-14 12:17:57 +0800 + + liblzma: Create crc_clmul.c. + + Both crc32_clmul() and crc64_clmul() are now exported from + crc32_clmul.c as lzma_crc32_clmul() and lzma_crc64_clmul(). This + ensures that is_clmul_supported() (now lzma_is_clmul_supported()) is + not duplicated between crc32_fast.c and crc64_fast.c. + + Also, it encapsulates the complexity of the CLMUL implementations into a + single file and reduces the complexity of crc32_fast.c and crc64_fast.c. + Before, CLMUL code was present in crc32_fast.c, crc64_fast.c, and + crc_common.h. + + During the conversion, various cleanups were applied to code (thanks to + Lasse Collin) including: + + - Require using semicolons with MASK_/L/H/LH macros. + - Variable typing and const handling improvements. + - Improvements to comments. + - Fixes to the pragmas used. + - Removed unneeded variables. + - Whitespace improvements. + - Fixed CRC_USE_GENERIC_FOR_SMALL_INPUTS handling. + - Silenced warnings and removed the need for some #pragmas + + CMakeLists.txt | 6 +- + configure.ac | 6 +- + src/liblzma/check/Makefile.inc | 3 + + src/liblzma/check/crc32_fast.c | 120 +----------- + src/liblzma/check/crc64_fast.c | 128 +------------ + src/liblzma/check/crc_clmul.c | 414 +++++++++++++++++++++++++++++++++++++++++ + src/liblzma/check/crc_common.h | 190 +------------------ + 7 files changed, 444 insertions(+), 423 deletions(-) + +commit a3ebc2c516b09616638060806c841bd4bcf7bce3 +Author: Jia Tan +Date: 2023-10-14 10:23:03 +0800 + + liblzma: Define CRC_USE_IFUNC in crc_common.h. + + When ifunc is supported, we can define a simpler macro instead of + repeating the more complex check in both crc32_fast.c and crc64_fast.c. + + src/liblzma/check/crc32_fast.c | 3 +-- + src/liblzma/check/crc64_fast.c | 3 +-- + src/liblzma/check/crc_common.h | 5 +++++ + 3 files changed, 7 insertions(+), 4 deletions(-) + +commit f1cd9d7194f005cd66ec03c6635ceae75f90ef17 +Author: Hans Jansen +Date: 2023-10-12 19:37:01 +0200 + + liblzma: Added crc32_clmul to crc32_fast.c. + + src/liblzma/check/crc32_fast.c | 247 ++++++++++++++++++++++++++++++++++++++-- + src/liblzma/check/crc32_table.c | 19 +++- + 2 files changed, 255 insertions(+), 11 deletions(-) + +commit 93e6fb08b22c7c13be2dd1e7274fe78413436254 +Author: Hans Jansen +Date: 2023-10-12 19:23:40 +0200 + + liblzma: Moved CLMUL CRC logic to crc_common.h. + + crc64_fast.c was updated to use the code from crc_common.h instead. + + src/liblzma/check/crc64_fast.c | 257 ++--------------------------------------- + src/liblzma/check/crc_common.h | 230 +++++++++++++++++++++++++++++++++++- + 2 files changed, 240 insertions(+), 247 deletions(-) + +commit 233885a437f8b55a5c8442984ebc0aaa579e92de +Author: Hans Jansen +Date: 2023-10-12 19:07:50 +0200 + + liblzma: Rename crc_macros.h to crc_common.h. + + CMakeLists.txt | 2 +- + src/liblzma/check/Makefile.inc | 2 +- + src/liblzma/check/crc32_fast.c | 2 +- + src/liblzma/check/crc64_fast.c | 2 +- + src/liblzma/check/{crc_macros.h => crc_common.h} | 2 +- + 5 files changed, 5 insertions(+), 5 deletions(-) + +commit 37947d4a7565b87e4cec8b89229d35b0a3f8d2cd +Author: Gabriela Gutierrez +Date: 2023-09-26 15:55:13 +0000 + + CI: Bump and ref actions by commit SHA in windows-ci.yml + + Referencing actions by commit SHA in GitHub workflows guarantees you are using an immutable version. Actions referenced by tags and branches are more vulnerable to attacks, such as the tag being moved to a malicious commit or a malicious commit being pushed to the branch. + + It's important to make sure the SHA's are from the original repositories and not forks. + + For reference: + + https://github.com/msys2/setup-msys2/releases/tag/v2.20.1 + https://github.com/msys2/setup-msys2/commit/27b3aa77f672cb6b3054121cfd80c3d22ceebb1d + + https://github.com/actions/checkout/releases/tag/v4.1.0 + https://github.com/actions/checkout/commit/8ade135a41bc03ea155e62e844d188df1ea18608 + + https://github.com/actions/upload-artifact/releases/tag/v3.1.3 + https://github.com/actions/upload-artifact/commit/a8a3f3ad30e3422c9c7b888a15615d19a852ae32 + + Signed-off-by: Gabriela Gutierrez + + .github/workflows/windows-ci.yml | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit f28cc9bd481ce493da11f98c18526d324211599a +Author: Gabriela Gutierrez +Date: 2023-09-26 14:35:08 +0000 + + CI: Bump and ref actions by commit SHA in ci.yml + + Referencing actions by commit SHA in GitHub workflows guarantees you are using an immutable version. Actions referenced by tags and branches are more vulnerable to attacks, such as the tag being moved to a malicious commit or a malicious commit being pushed to the branch. + + It's important to make sure the SHA's are from the original repositories and not forks. + + For reference: + + https://github.com/actions/checkout/releases/tag/v4.1.0 + https://github.com/actions/checkout/commit/8ade135a41bc03ea155e62e844d188df1ea18608 + + https://github.com/actions/upload-artifact/releases/tag/v3.1.3 + https://github.com/actions/upload-artifact/commit/a8a3f3ad30e3422c9c7b888a15615d19a852ae32 + + Signed-off-by: Gabriela Gutierrez + + .github/workflows/ci.yml | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit f74f1740067b75042497edbfa6ea457ff75484b9 +Author: Jia Tan +Date: 2023-10-12 20:12:18 +0800 + + Build: Update visibility.m4 from Gnulib. + + Updating from version 6 -> 8 from upstream. Declarations for variables + and function bodies were added to avoid unnecessary failures with + -Werror. + + m4/visibility.m4 | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +commit 5c4bca521e6fb435898a0012b3276eee70a6dadf +Author: Lasse Collin +Date: 2023-10-06 19:36:35 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit d91cb6e884c73d0b05d7e7d68ad4e6eb29f4b44b +Author: Lasse Collin +Date: 2023-10-06 18:55:57 +0300 + + CMake/Windows: Fix when the windres workaround is applied. + + CMake doesn't set WIN32 on CYGWIN but the workaround is + probably needed on Cygwin too. Same for MSYS and MSYS2. + + The workaround must not be used with Clang that is acting in + MSVC mode. This fixes it by checking for the known environments + that need the workaround instead of using "NOT MSVC". + + Thanks to Martin Storsjö. + https://github.com/tukaani-project/xz/commit/0570308ddd9c0e39e85597ebc0e31d4fc81d436f#commitcomment-129098431 + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 01e34aa1171b04f8b28960b1cc6135a903e0c13d +Author: Jia Tan +Date: 2023-09-29 22:11:54 +0800 + + CI: Disable CLANG64 MSYS2 environment until bug is resolved. + + lld 17.0.1 searches for libraries to link first in the toolchain + directories before the local directory when building. The is a problem + for us because liblzma.a is installed in MSYS2 CLANG64 by default and + xz.exe will thus use the installed library instead of the one being + built. + + This causes tests to fail when they are expecting features to be + disabled. More importantly, it will compile xz.exe with an incorrect + liblzma and could cause unexpected behavior by being unable to update + liblzma code in static builds. The CLANG64 environment can be tested + again once this is fixed. + + Link to bug: https://github.com/llvm/llvm-project/issues/67779. + + .github/workflows/windows-ci.yml | 8 +++++--- + 1 file changed, 5 insertions(+), 3 deletions(-) + +commit 30d0c35327f3639cb11224872aa58fdbf0b1526e +Author: Jia Tan +Date: 2023-09-29 20:14:39 +0800 + + CMake: Rename xz and man page symlink custom targets. + + The Ninja Generator for CMake cannot have a custom target and its + BYPRODUCTS have the same name. This has prevented Ninja builds on + Unix-like systems since the xz symlinks were introduced in + 80a1a8bb838842a2be343bd88ad1462c21c5e2c9. + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 506d03127a8565442b028ec991e1578124fd3025 +Author: Jia Tan +Date: 2023-09-29 19:58:44 +0800 + + CMake: Specify LINKER_LANGUAGE for libgnu target to fix Ninja Generator. + + CMake is unable to guess the linker language for just a header file so + it must be explicitly set. + + CMakeLists.txt | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 0570308ddd9c0e39e85597ebc0e31d4fc81d436f +Author: Lasse Collin +Date: 2023-09-27 19:54:35 +0300 + + CMake: Fix Windows build with Clang/LLVM 17. + + llvm-windres 17.0.0 has more accurate emulation of GNU windres, so + the hack for GNU windres must now be used with llvm-windres too. + + LLVM 16.0.6 has the old behavior and there likely won't be more + 16.x releases. So we can simply check for >= 17.0.0. + + See also: + https://github.com/llvm/llvm-project/commit/2bcc0fdc58a220cb9921b47ec8a32c85f2511a47 + + CMakeLists.txt | 26 ++++++++++++++------------ + 1 file changed, 14 insertions(+), 12 deletions(-) + +commit 5a9af95f85a7e5d4f9c10cb8cf737651a921f1d1 +Author: Lasse Collin +Date: 2023-09-26 21:47:13 +0300 + + liblzma: Update a comment. + + The C standards don't allow an empty translation unit which can be + avoided by declaring something, without exporting any symbols. + + When I committed f644473a211394447824ea00518d0a214ff3f7f2 I had + a feeling that some specific toolchain somewhere didn't like + empty object files (assembler or maybe "ar" complained) but + I cannot find anything to confirm this now. Quite likely I + remembered nonsense. I leave this here as a note to my future self. :-) + + src/liblzma/check/crc64_table.c | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + +commit 8ebaf3f665ddc7e4f19c613005050dde5ccbe499 +Author: Jia Tan +Date: 2023-09-27 00:02:11 +0800 + + liblzma: Avoid compiler warning without creating extra symbol. + + When the generic fast crc64 method is used, then we omit + lzma_crc64_table[][]. Similar to + d9166b52cf3458a4da3eb92224837ca8fc208d79, we can avoid compiler warnings + with -Wempty-translation-unit (Clang) or -pedantic (GCC) by creating a + never used typedef instead of an extra symbol. + + src/liblzma/check/crc64_table.c | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + +commit 092d21db2e5eea19fe079264ce48c178989c7606 +Author: Lasse Collin +Date: 2023-09-26 17:24:15 +0300 + + Build: Update the comment about -Werror usage in checks. + + configure.ac | 10 ++++++++-- + 1 file changed, 8 insertions(+), 2 deletions(-) + +commit a37a2763383e6c204fe878e1416dd35e7711d3a9 +Author: Lasse Collin +Date: 2023-09-26 15:00:43 +0300 + + Build: Fix __attribute__((ifunc(...))) detection with clang -Wall. + + Now if user-supplied CFLAGS contains -Wall -Wextra -Wpedantic + the two checks that need -Werror will still work. + + At CMake side there is add_compile_options(-Wall -Wextra) + but it didn't affect the -Werror tests. So with both Autotools + and CMake only user-supplied CFLAGS could make the checks fail + when they shouldn't. + + This is not a full fix as things like -Wunused-macros in + user-supplied CFLAGS will still cause problems with both + GCC and Clang. + + CMakeLists.txt | 8 ++++++++ + configure.ac | 8 ++++++++ + 2 files changed, 16 insertions(+) + +commit 9c42f936939b813f25d0ff4e99c3eb9c2d17a0d2 +Author: Lasse Collin +Date: 2023-09-26 13:51:31 +0300 + + Build: Fix underquoted AC_LANG_SOURCE. + + It made no practical difference in this case. + + configure.ac | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 9f1444a8a5c0e724b2c7ef83424f642f07a95982 +Author: Lasse Collin +Date: 2023-09-26 13:14:37 +0300 + + Build: Silence two Autoconf warnings. + + There were two uses of AC_COMPILE_IFELSE that didn't use + AC_LANG_SOURCE and Autoconf warned about these. The omission + had been intentional but it turned out that this didn't do + what I thought it would. + + Autoconf 2.71 manual gives an impression that AC_LANG_SOURCE + inserts all #defines that have been made with AC_DEFINE so + far (confdefs.h). The idea was that omitting AC_LANG_SOURCE + would mean that only the exact code included in the + AC_COMPILE_IFELSE call would be compiled. + + With C programs this is not true: the #defines get added without + AC_LANG_SOURCE too. There seems to be no neat way to avoid this. + Thus, with the C language at least, adding AC_LANG_SOURCE makes + no other difference than silencing a warning from Autoconf. The + generated "configure" remains identical. (Docs of AC_LANG_CONFTEST + say that the #defines have been inserted since Autoconf 2.63b and + that AC_COMPILE_IFELSE uses AC_LANG_CONFTEST. So the behavior is + documented if one also reads the docs of macros that one isn't + calling directly.) + + Any extra code, including #defines, can cause problems for + these two tests because these tests must use -Werror. + CC=clang CFLAGS=-Weverything is the most extreme example. + It enables -Wreserved-macro-identifier which warns about + #define __EXTENSIONS__ 1 because it begins with two underscores. + It's possible to write a test file that passes -Weverything but + it becomes impossible when Autoconf inserts confdefs.h. + + So this commit adds AC_LANG_SOURCE to silence Autoconf warnings. + A different solution is needed for -Werror tests. + + configure.ac | 9 ++++----- + 1 file changed, 4 insertions(+), 5 deletions(-) + +commit 519e47c2818acde571fadc79551294527fe6cc22 +Author: Jia Tan +Date: 2023-09-26 01:17:11 +0800 + + CMake: Remove accidental extra newline. + + CMakeLists.txt | 1 - + 1 file changed, 1 deletion(-) + +commit bbb42412da6a02705ba3e668e90840c2683e4e67 +Author: Jia Tan +Date: 2023-09-26 00:47:26 +0800 + + Build: Remove Gnulib dependency from tests. + + The tests do not use any Gnulib replacements so they do not need to link + libgnu.a or have /lib in the include path. + + tests/Makefile.am | 7 +------ + 1 file changed, 1 insertion(+), 6 deletions(-) + +commit d265f6b75691c6c8fa876eb5320c3ff5aed17dfa +Author: Jia Tan +Date: 2023-09-26 00:43:43 +0800 + + CMake: Remove /lib from tests include path. + + The tests never included anything from /lib, so this was not needed. + + CMakeLists.txt | 1 - + 1 file changed, 1 deletion(-) + +commit 9fb5de41f2fb654ca952d4bda15cf3777c2b720f +Author: Jia Tan +Date: 2023-09-24 22:10:41 +0800 + + Scripts: Change quoting style from `...' to '...'. + + src/scripts/xzdiff.in | 2 +- + src/scripts/xzgrep.in | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit eaebdef4d4de3c088b0905f42626b74e0d23abf3 +Author: Jia Tan +Date: 2023-09-24 22:10:18 +0800 + + xz: Change quoting style from `...' to '...'. + + src/xz/args.c | 6 +++--- + src/xz/file_io.c | 2 +- + src/xz/main.c | 4 ++-- + src/xz/message.c | 14 +++++++------- + src/xz/options.c | 2 +- + src/xz/suffix.c | 2 +- + src/xz/util.c | 6 +++--- + 7 files changed, 18 insertions(+), 18 deletions(-) + +commit f6667702bf075a05fbe336dbf3576ad1a82ec645 +Author: Jia Tan +Date: 2023-09-24 22:09:47 +0800 + + liblzma: Change quoting style from `...' to '...'. + + This was done for both internal and API headers. + + src/liblzma/api/lzma/base.h | 18 +++++++++--------- + src/liblzma/api/lzma/container.h | 10 +++++----- + src/liblzma/api/lzma/filter.h | 6 +++--- + src/liblzma/api/lzma/index.h | 8 ++++---- + src/liblzma/api/lzma/lzma12.h | 2 +- + src/liblzma/lz/lz_encoder.h | 2 +- + src/liblzma/rangecoder/range_decoder.h | 2 +- + 7 files changed, 24 insertions(+), 24 deletions(-) + +commit be012b8097a4eaee335b51357d6befa745f753ce +Author: Jia Tan +Date: 2023-09-24 22:09:16 +0800 + + Build: Change quoting style from `...' to '...'. + + configure.ac | 18 +++++++++--------- + dos/config.h | 6 +++--- + m4/getopt.m4 | 2 +- + m4/tuklib_progname.m4 | 2 +- + windows/build.bash | 2 +- + 5 files changed, 15 insertions(+), 15 deletions(-) + +commit ce162db07f03495bd333696e66883c8f36abdc1e +Author: Jia Tan +Date: 2023-09-24 22:05:02 +0800 + + Docs: Change quoting style from `...' to '...'. + + These days the ` and ' do not look symmetric. This quoting style has + been changed in various apps over the years including the GNU tools. + + INSTALL | 6 +++--- + doc/examples/01_compress_easy.c | 2 +- + doc/examples/11_file_info.c | 16 ++++++++-------- + 3 files changed, 12 insertions(+), 12 deletions(-) + +commit db17656721e43939bfa4ec13506e7c76f4b86da6 +Author: Jia Tan +Date: 2023-09-24 21:25:01 +0800 + + lib: Silence -Wsign-conversion in getopt.c. + + lib/getopt.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit a6234f677d66888f435010bc0b67de6a32fefcf6 +Author: Jia Tan +Date: 2023-09-24 20:48:52 +0800 + + Build: Update getopt.m4 from Gnulib. + + This file was modified from upstream since we do not need to replace + getopt() and can avoid complexity and feature tests. + + m4/getopt.m4 | 79 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 39 insertions(+), 40 deletions(-) + +commit 84808b68f1075e8603a8ef95d361a61fdc6a5b10 +Author: Jia Tan +Date: 2023-09-26 00:09:53 +0800 + + CMake: Add /lib to include path. + + CMakeLists.txt | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 01804a0b4b64e0f33568e947e0579263808c59d3 +Author: Jia Tan +Date: 2023-09-24 20:36:34 +0800 + + CMake: Update libgnu target with new header files. + + CMakeLists.txt | 5 +++++ + 1 file changed, 5 insertions(+) + +commit d34558388fe1d8929f6478d61dc322eb4f2900af +Author: Jia Tan +Date: 2023-09-23 00:47:52 +0800 + + lib: Update Makefile.am for new header files. + + lib/Makefile.am | 12 +++++++++++- + 1 file changed, 11 insertions(+), 1 deletion(-) + +commit 52bf644bdf536e20fcc743b712cede135e05eec5 +Author: Jia Tan +Date: 2023-09-24 20:34:03 +0800 + + lib: Update getopt1.c from Gnulib. + + The only difference was maintaining the conditional inclusion for + config.h. + + lib/getopt1.c | 56 ++++++++++++++++++++++---------------------------------- + 1 file changed, 22 insertions(+), 34 deletions(-) + +commit 7e884c00d0093c38339f17fb1d280eec493f42ca +Author: Jia Tan +Date: 2023-09-23 03:27:00 +0800 + + lib: Update getopt.in.h from Gnulib with modifications. + + We can still avoid modifying the contents of this file during + configuration to simplify the build systems. Gnulib added replacements + for inclusions guards for Cygwin. Cygwin should not need getopt_long + replacement so this feature can be omitted. + + is conditionally included to avoid MSVC since it is not + available. + + The definition for _GL_ARG_NONNULL was also copied into this file from + Gnulib since this stage is usually done during gnulib-tool. + + lib/getopt.in.h | 228 +++++++------------------------------------------------- + 1 file changed, 29 insertions(+), 199 deletions(-) + +commit cff05f82066ca3ce9425dafdb086325a8eef8de3 +Author: Jia Tan +Date: 2023-09-23 00:31:55 +0800 + + lib: Update getopt_int.h from Gnulib. + + lib/getopt_int.h | 109 ++++++++++++++++++++++++------------------------------- + 1 file changed, 48 insertions(+), 61 deletions(-) + +commit 04bd86a4b010d43c6a016a3857ecb38dc1d5b024 +Author: Jia Tan +Date: 2023-09-23 00:27:23 +0800 + + lib: Update getopt.c from Gnulib with modifications. + + The code maintains the prior modifications of conditionally including + config.h and disabling NLS support. + + _GL_UNUSED is repalced with the simple cast to void trick. _GL_UNUSED + is only used for these two parameters so its simpler than having to + define it. + + lib/getopt.c | 1134 +++++++++++++++++++--------------------------------------- + 1 file changed, 377 insertions(+), 757 deletions(-) + +commit 56b42be7287844db20b3a3bc1372c6ae8c040d63 +Author: Jia Tan +Date: 2023-09-23 00:18:56 +0800 + + lib: Add getopt-cdefs.h for getopt_long update. + + This was modified slightly from Gnulib. In Gnulib, it expects the + @HAVE_SYS_CDEFS_H@ to be replaced. Instead, we can set HAVE_SYS_CDEFS_H + on systems that have it and avoid copying another file into the build + directory. Since we are not using gnulib-tool, copying extra files + requires extra build system updates (and special handling with CMake) so + we should avoid when possible. + + lib/getopt-cdefs.h | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 70 insertions(+) + +commit 9834e591a4cf9dc2f49e42e26bf28d1d247bc196 +Author: Jia Tan +Date: 2023-09-23 00:15:25 +0800 + + lib: Copy new header files from Gnulib without modification. + + The getopt related files have changed from Gnulib by splitting up + getopt.in.h into more modular header files. We could have kept + everything in just getopt.in.h, but this will help us continue to update + in the future. + + lib/getopt-core.h | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++ + lib/getopt-ext.h | 77 +++++++++++++++++++++++++++++++++++++++++ + lib/getopt-pfx-core.h | 66 +++++++++++++++++++++++++++++++++++ + lib/getopt-pfx-ext.h | 70 +++++++++++++++++++++++++++++++++++++ + 4 files changed, 309 insertions(+) + +commit 5b7a6f06e93d99d6635a740fd2e12fab66096c93 +Author: Lasse Collin +Date: 2023-09-22 21:16:52 +0300 + + Windows: Update the version requirement comments from Win95 to W2k. + + windows/README-Windows.txt | 10 ++++------ + windows/build.bash | 6 +++--- + 2 files changed, 7 insertions(+), 9 deletions(-) + +commit e582f8e0fee46e7cd967f42f465d6bb608b73bc1 +Author: Lasse Collin +Date: 2023-09-22 21:12:54 +0300 + + tuklib_physmem: Comment out support for Windows versions older than 2000. + + src/common/tuklib_physmem.c | 20 +++++++++----------- + 1 file changed, 9 insertions(+), 11 deletions(-) + +commit 7d73d1f0e08f96c4ab7aac91b958e37a3dadf07a +Author: Lasse Collin +Date: 2023-09-24 16:32:32 +0300 + + sysdefs.h: Update the comment about __USE_MINGW_ANSI_STDIO. + + src/common/sysdefs.h | 10 +++++++++- + 1 file changed, 9 insertions(+), 1 deletion(-) + +commit 2a9929af0ab7e6c0ab725565034afe3293e51d71 +Author: Lasse Collin +Date: 2023-09-22 02:33:29 +0300 + + xz: Windows: Don't (de)compress to special files like "con" or "nul". + + Before this commit, the following writes "foo" to the + console and deletes the input file: + + echo foo | xz > con_xz + xz --suffix=_xz --decompress con_xz + + It cannot happen without --suffix because names like con.xz + are also special and so attempting to decompress con.xz + (or compress con to con.xz) will already fail when opening + the input file. + + Similar thing is possible when compressing. The following + writes to "nul" and the input file "n" is deleted. + + echo foo | xz > n + xz --suffix=ul n + + Now xz checks if the destination is a special file before + continuing. DOS/DJGPP version had a check for this but + Windows (and OS/2) didn't. + + src/xz/file_io.c | 35 ++++++++++++++++++++++++++++------- + 1 file changed, 28 insertions(+), 7 deletions(-) + +commit 01311b81f03cce1c0ce847a3d556f84dbd439343 +Author: Lasse Collin +Date: 2023-09-21 20:42:52 +0300 + + CMake: Wrap two overlong lines that are possible to wrap. + + CMakeLists.txt | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 152d0771ddd0cffcac9042ad1a66f110d228eee2 +Author: Lasse Collin +Date: 2023-09-21 20:36:31 +0300 + + CMake: Add a comment about threads on Cygwin. + + CMakeLists.txt | 1 + + 1 file changed, 1 insertion(+) + +commit 6df988cceffaa3100b428ed816fad334935b27bf +Author: Lasse Collin +Date: 2023-09-12 23:53:25 +0300 + + MSVC: Remove Visual Studio project files and update INSTALL-MSVC.txt. + + CMake is now the preferred build file generator when building + with MSVC. + + windows/INSTALL-MSVC.txt | 37 ++-- + windows/vs2013/config.h | 157 --------------- + windows/vs2013/liblzma.vcxproj | 363 --------------------------------- + windows/vs2013/liblzma_dll.vcxproj | 398 ------------------------------------ + windows/vs2013/xz_win.sln | 48 ----- + windows/vs2017/config.h | 157 --------------- + windows/vs2017/liblzma.vcxproj | 363 --------------------------------- + windows/vs2017/liblzma_dll.vcxproj | 398 ------------------------------------ + windows/vs2017/xz_win.sln | 48 ----- + windows/vs2019/config.h | 157 --------------- + windows/vs2019/liblzma.vcxproj | 364 --------------------------------- + windows/vs2019/liblzma_dll.vcxproj | 399 ------------------------------------- + windows/vs2019/xz_win.sln | 51 ----- + 13 files changed, 12 insertions(+), 2928 deletions(-) + +commit edd563daf0da1d00018684614803c77ab62efcd6 +Author: Lasse Collin +Date: 2023-09-21 19:17:40 +0300 + + CMake: Require VS2015 or later for building xzdec. + + xzdec might build with VS2013 but it hasn't been tested. + It was never supported before and VS2013 is old anyway + so for simplicity only liblzma is supported with VS2013. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit daea64d158a7151ca6c255a0e4554c6d521cd589 +Author: Lasse Collin +Date: 2023-09-12 23:43:49 +0300 + + CMake: Allow building xz with Visual Studio 2015 and later. + + Building the command line tools xz and xzdec with the combination + of CMake + Visual Studio 2015/2017/2019/2022 works now. + + VS2013 update 2 should still be able to build liblzma. + VS2013 cannot build the xz command line tool because xz + needs snprintf() that roughly conforms to C99. + VS2013 is old and no extra code will be added to support it. + + Thanks to Kelvin Lee and Jia Tan for testing. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 8c2d197c940d246849b2ec48109bb22e54036927 +Author: Lasse Collin +Date: 2023-09-12 23:34:31 +0300 + + MSVC: #define inline and restrict only when needed. + + This also drops the check for _WIN32 as that shouldn't be needed. + + src/common/sysdefs.h | 13 ++++++++----- + 1 file changed, 8 insertions(+), 5 deletions(-) + +commit af66cd585902045e5689a0418103ec81f19f1d0a +Author: Lasse Collin +Date: 2023-09-12 22:16:56 +0300 + + CMake: Add support for replacement getopt_long (lib/getopt*). + + Thanks to Jia Tan for the initial work. I added the libgnu target + and made a few related minor edits. + + CMakeLists.txt | 54 +++++++++++++++++++++++++++++++++++++++++++++++------- + 1 file changed, 47 insertions(+), 7 deletions(-) + +commit e3288fdb45c580cb849f6799cf419c4922004ae5 +Author: Lasse Collin +Date: 2023-09-12 21:12:34 +0300 + + CMake: Bump maximum policy version to 3.27. + + There are several new policies. CMP0149 may affect the Windows SDK + version that CMake will choose by default. The new behavior is more + predictable, always choosing the latest SDK version by default. + + The other new policies shouldn't affect this package. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit aff1b479c7b168652bd20305ceed4317d5db6661 +Author: Lasse Collin +Date: 2023-09-12 20:55:10 +0300 + + lib/getopt*.c: Include only HAVE_CONFIG_H is defined. + + The CMake-based build doesn't use config.h. + + Up-to-date getopt_long in Gnulib is LGPLv2 so at some + point it could be included in XZ Utils too but for now + this commit is enough to make CMake-based build possible. + + lib/getopt.c | 4 +++- + lib/getopt1.c | 4 +++- + 2 files changed, 6 insertions(+), 2 deletions(-) + +commit aa0cd585d2ed1455d35732798e0d90e3520e8ba5 +Author: Lasse Collin +Date: 2023-09-08 19:08:57 +0300 + + Doxygen: Add more C macro names to PREDEFINED. + + doxygen/Doxyfile | 7 +++++-- + 1 file changed, 5 insertions(+), 2 deletions(-) + +commit ee7709bae53637e1765ce142ef102914f1423cb5 +Author: Lasse Collin +Date: 2023-09-11 18:47:26 +0300 + + liblzma: Move a few __attribute__ uses in function declarations. + + The API headers have many attributes but these were left + as is for now. + + src/liblzma/common/common.c | 6 ++++-- + src/liblzma/common/common.h | 8 ++++---- + src/liblzma/common/memcmplen.h | 3 ++- + 3 files changed, 10 insertions(+), 7 deletions(-) + +commit 217958d88713b5dc73d366d24dd64b2b311b86fe +Author: Lasse Collin +Date: 2023-09-11 19:03:35 +0300 + + xz, xzdec, lzmainfo: Use tuklib_attr_noreturn. + + For compatibility with C23's [[noreturn]], tuklib_attr_noreturn + must be at the beginning of declaration (before "extern" or + "static", and even before any GNU C's __attribute__). + + This commit also moves all other function attributes to + the beginning of function declarations. "extern" is kept + at the beginning of a line so the attributes are listed on + separate lines before "extern" or "static". + + src/lzmainfo/lzmainfo.c | 6 ++++-- + src/xz/coder.c | 3 ++- + src/xz/hardware.h | 3 ++- + src/xz/message.h | 30 +++++++++++++++++------------- + src/xz/options.c | 3 ++- + src/xz/util.h | 8 ++++---- + src/xzdec/xzdec.c | 9 ++++++--- + 7 files changed, 37 insertions(+), 25 deletions(-) + +commit 18a66fbac031c98f9c2077fc88846e4d07849197 +Author: Lasse Collin +Date: 2023-09-11 18:53:31 +0300 + + Remove incorrect uses of __attribute__((__malloc__)). + + xrealloc() is obviously incorrect, modern GCC docs even + mention realloc() as an example where this attribute + cannot be used. + + liblzma's lzma_alloc() and lzma_alloc_zero() would be + correct uses most of the time but custom allocators + may use a memory pool or otherwise hold the pointer + so aliasing issues could happen in theory. + + The xstrdup() case likely was correct but I removed it anyway. + Now there are no __malloc__ attributes left in the code. + The allocations aren't in hot paths so this should make + no practical difference. + + src/liblzma/common/common.c | 4 ++-- + src/liblzma/common/common.h | 4 ++-- + src/xz/util.h | 4 ++-- + 3 files changed, 6 insertions(+), 6 deletions(-) + +commit 74b0e900c92d5b222b36f474f1efa431f8e262f7 +Author: Lasse Collin +Date: 2023-09-08 18:41:25 +0300 + + Build: Omit -Wc99-c11-compat since it warns about _Noreturn. + + configure.ac | 1 - + 1 file changed, 1 deletion(-) + +commit 90c94dddfd57b7d744bfad64c54e10d15778144b +Author: Lasse Collin +Date: 2023-09-08 18:19:26 +0300 + + tuklib: Update tuklib_attr_noreturn for C11/C17 and C23. + + This makes no difference for GCC or Clang as they support + GNU C's __attribute__((__noreturn__)) but this helps with MSVC: + + - VS 2019 version 16.7 and later support _Noreturn if the + options /std:c11 or /std:c17 are used. This gets handled + with the check for __STDC_VERSION__ >= 201112. + + - When MSVC isn't in C11/C17 mode, __declspec(noreturn) is used. + + C23 will deprecate _Noreturn (and ) + for [[noreturn]]. This commit anticipates that but + the final __STDC_VERSION__ value isn't known yet. + + src/common/tuklib_common.h | 22 +++++++++++++++++++++- + src/common/tuklib_exit.h | 4 ++-- + 2 files changed, 23 insertions(+), 3 deletions(-) + +commit 189f72581329ab281ad6af37f60135910cb1b146 +Author: Lasse Collin +Date: 2023-09-11 17:22:44 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 79334e7f20f2bf9e0de095835b48868f1238f584 +Author: Lasse Collin +Date: 2023-09-05 22:42:10 +0300 + + MSVC: xz: Make file_io.c and file_io.h compatible with MSVC. + + Thanks to Kelvin Lee for the original patches + and testing the modifications I made. + + src/xz/file_io.c | 26 ++++++++++++++++++++++++++ + src/xz/file_io.h | 10 ++++++++++ + 2 files changed, 36 insertions(+) + +commit c660b8d78b7bda43b12b285550d8c70e8ccec698 +Author: Lasse Collin +Date: 2023-09-05 21:33:35 +0300 + + MSVC: xz: Use GetTickCount64() to implement mytime_now(). + + It's available since Windows Vista. + + src/xz/mytime.c | 11 +++++++++-- + 1 file changed, 9 insertions(+), 2 deletions(-) + +commit 5c6f892d411670e3060f4bc309402617a209e57c +Author: Kelvin Lee +Date: 2023-09-05 15:05:09 +0300 + + MSVC: xz: Use _stricmp() instead of strcasecmp() in suffix.c. + + src/xz/suffix.c | 10 ++++++++-- + 1 file changed, 8 insertions(+), 2 deletions(-) + +commit e241051f50044259d174e8b4633dd9a1c4478408 +Author: Kelvin Lee +Date: 2023-09-05 15:01:10 +0300 + + MSVC: xz: Use _isatty() from to implement isatty(). + + src/xz/message.c | 5 +++++ + src/xz/util.c | 5 +++++ + 2 files changed, 10 insertions(+) + +commit d14bba8fc2be02a9fed8c9bcaaf61103451755f8 +Author: Kelvin Lee +Date: 2023-09-05 15:10:31 +0300 + + MSVC: xz: Use _fileno() instead of fileno(). + + src/xz/private.h | 4 ++++ + 1 file changed, 4 insertions(+) + +commit c4edd367678e6a38c42b149856159bf417da7fe1 +Author: Kelvin Lee +Date: 2023-09-05 15:00:07 +0300 + + MSVC: xzdec: Use _fileno and _setmode. + + src/xzdec/xzdec.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit cfd1054b9b539ee92524901e95d7bb5a1fe670a0 +Author: Kelvin Lee +Date: 2023-09-05 14:37:50 +0300 + + MSVC: Don't #include . + + lib/getopt.c | 4 +++- + lib/getopt.in.h | 4 +++- + src/xz/private.h | 5 ++++- + src/xzdec/xzdec.c | 5 ++++- + 4 files changed, 14 insertions(+), 4 deletions(-) + +commit adef92f23563a2cc088b31ddee9040ecc96bc996 +Author: Lasse Collin +Date: 2023-09-19 14:03:45 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 953e775941a25bfcfa353f802b13e66acb1edf2c +Author: Jia Tan +Date: 2023-09-14 21:13:23 +0800 + + CI: Enable CLMUL in address sanitization test. + + The crc64_clmul() function should be ignored by the address sanitizer + now so these builds should still pass. + + .github/workflows/ci.yml | 12 ++++-------- + 1 file changed, 4 insertions(+), 8 deletions(-) + +commit f167e79bc98f3f56af2e767b83aa81c2d2b9ed77 +Author: Lasse Collin +Date: 2023-09-14 16:35:46 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 4f44ef86758a41a8ec814096f4cb6ee6de04c82e +Author: Lasse Collin +Date: 2023-09-14 16:34:07 +0300 + + liblzma: Mark crc64_clmul() with __attribute__((__no_sanitize_address__)). + + Thanks to Agostino Sarubbo. + Fixes: https://github.com/tukaani-project/xz/issues/62 + + src/liblzma/check/crc64_fast.c | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit 7379bb3eed428c0ae734d0cc4a1fd04359d53f08 +Author: Jia Tan +Date: 2023-09-12 22:36:12 +0800 + + CMake: Fix time.h checks not running on second CMake run. + + If CMake was configured more than once, HAVE_CLOCK_GETTIME and + HAVE_CLOCK_MONOTONIC would not be set as compile definitions. The check + for librt being needed to provide HAVE_CLOCK_GETTIME was also + simplified. + + CMakeLists.txt | 18 ++++++++++-------- + 1 file changed, 10 insertions(+), 8 deletions(-) + +commit 5d691fe58286b92d704c0dc5cd0c4df22881c6c6 +Author: Jia Tan +Date: 2023-09-12 22:34:06 +0800 + + CMake: Fix unconditionally defining HAVE_CLOCK_MONOTONIC. + + If HAVE_CLOCK_GETTIME was defined, then HAVE_CLOCK_MONOTONIC was always + added as a compile definition even if the check for it failed. + + CMakeLists.txt | 8 +++----- + 1 file changed, 3 insertions(+), 5 deletions(-) + +commit eccf12866527b8d24c7d7f92f755142be8ef9b11 +Author: Lasse Collin +Date: 2023-08-31 19:50:05 +0300 + + xz: Refactor thousand separator detection and disable it on MSVC. + + Now the two variations of the format strings are created with + a macro, and the whole detection code can be easily disabled + on platforms where thousand separator formatting is known to + not work (MSVC has no support, and on DJGPP 2.05 it can have + problems in some cases). + + src/xz/util.c | 89 ++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 45 insertions(+), 44 deletions(-) + +commit f7093cd9d130477c234b40aeda613964171f8f21 +Author: Lasse Collin +Date: 2023-08-31 18:14:43 +0300 + + xz: Fix a too relaxed assertion and remove uses of SSIZE_MAX. + + SSIZE_MAX isn't readily available on MSVC. Removing it means + that there is one thing less to worry when porting to MSVC. + + src/xz/file_io.c | 5 ++--- + src/xz/file_io.h | 4 ++-- + 2 files changed, 4 insertions(+), 5 deletions(-) + +commit 74c3449d8b816a724b12ebce7417e00fb597309a +Author: Jia Tan +Date: 2023-08-28 23:14:45 +0800 + + Tests: Improve invalid unpadded size check in test_lzma_index_append(). + + This check was extended to test the code added to fix a failing assert + in ae5c07b22a6b3766b84f409f1b6b5c100469068a. + + tests/test_index.c | 26 +++++++++++++++++++++++--- + 1 file changed, 23 insertions(+), 3 deletions(-) + +commit 2544274a8b8a27f4ea6c457d2c4c32eb1e4cd336 +Author: Jia Tan +Date: 2023-08-28 21:54:41 +0800 + + Tests: Improve comments in test_index.c. + + tests/test_index.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 49be29d6380b94e6fb26e511dd2cdbd9afce0f8b +Author: Jia Tan +Date: 2023-08-28 21:52:54 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 721e3d9f7a82f59f32795d5fb97e0210d1aa839a +Author: Jia Tan +Date: 2023-08-28 21:50:16 +0800 + + liblzma: Update assert in vli_ceil4(). + + The argument to vli_ceil4() should always guarantee the return value + is also a valid lzma_vli. Thus the highest three valid lzma_vli values + are invalid arguments. All uses of the function ensure this so the + assert is updated to match this. + + src/liblzma/common/index.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ae5c07b22a6b3766b84f409f1b6b5c100469068a +Author: Jia Tan +Date: 2023-08-28 21:31:25 +0800 + + liblzma: Add overflow check for Unpadded size in lzma_index_append(). + + This was not a security bug since there was no path to overflow + UINT64_MAX in lzma_index_append() or when it calls index_file_size(). + The bug was discovered by a failing assert() in vli_ceil4() when called + from index_file_size() when unpadded_sum (the sum of the compressed size + of current Stream and the unpadded_size parameter) exceeds LZMA_VLI_MAX. + + Previously, the unpadded_size parameter was checked to be not greater + than UNPADDED_SIZE_MAX, but no check was done once compressed_base was + added. + + This could not have caused an integer overflow in index_file_size() when + called by lzma_index_append(). The calculation for file_size breaks down + into the sum of: + + - Compressed base from all previous Streams + - 2 * LZMA_STREAM_HEADER_SIZE (size of the current Streams header and + footer) + - stream_padding (can be set by lzma_index_stream_padding()) + - Compressed base from the current Stream + - Unpadded size (parameter to lzma_index_append()) + + The sum of everything except for Unpadded size must be less than + LZMA_VLI_MAX. This is guarenteed by overflow checks in the functions + that can set these values including lzma_index_stream_padding(), + lzma_index_append(), and lzma_index_cat(). The maximum value for + Unpadded size is enforced by lzma_index_append() to be less than or + equal UNPADDED_SIZE_MAX. Thus, the sum cannot exceed UINT64_MAX since + LZMA_VLI_MAX is half of UINT64_MAX. + + Thanks to Joona Kannisto for reporting this. + + src/liblzma/common/index.c | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 1057765aaabfe0f1397b8094531846655376ae38 +Author: Jia Tan +Date: 2023-08-28 22:18:29 +0800 + + Translations: Update the Esperanto translation. + + po/eo.po | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit f2e94d064f305bb8ad77ca70f91d93e55f5cf856 +Author: Jia Tan +Date: 2023-08-26 20:10:23 +0800 + + Translations: Update the Esperanto translation. + + po/eo.po | 47 +++++++++++++++++++++++++++++------------------ + 1 file changed, 29 insertions(+), 18 deletions(-) + +commit 2b871f4dbffe3801d0da3f89806b5935f758d5f3 +Author: Jia Tan +Date: 2023-08-09 20:55:36 +0800 + + Docs: Update INSTALL for --enable-threads method win95. + + The Autotools build allows win95 threads and --enable-small together now + if the compiler supports __attribute__((__constructor__)). + + INSTALL | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 356ad5b26b4196f085ce3afa1869154ca81faad8 +Author: Jia Tan +Date: 2023-08-09 20:54:15 +0800 + + CMake: Conditionally allow win95 threads and --enable-small. + + CMakeLists.txt | 28 ++++++++++++++++++++-------- + 1 file changed, 20 insertions(+), 8 deletions(-) + +commit de574404c4c2f87aca049f232c38526e3ce092aa +Author: Jia Tan +Date: 2023-08-09 20:35:16 +0800 + + Build: Conditionally allow win95 threads and --enable-small. + + When the compiler supports __attribute__((__constructor__)) + mythread_once() is never used, even with --enable-small. A configuration + with win95 threads and --enable-small will compile and be thread safe so + it can be allowed. + + This isn't a very common configuration since MSVC does not support + __attribute__((__constructor__)), but MINGW32 and CLANG32 environments + for MSYS2 can use win95 threads and have + __attribute__((__constructor__)) support. + + configure.ac | 21 +++++++++++++-------- + 1 file changed, 13 insertions(+), 8 deletions(-) + +commit 6bf33b704cd31dccf25e68480464aa22d3fcad5a +Author: Jamaika1 +Date: 2023-08-08 14:07:59 +0200 + + mythread.h: Fix typo error in Vista threads mythread_once(). + + The "once_" variable was accidentally referred to as just "once". This + prevented building with Vista threads when + HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR was not defined. + + src/common/mythread.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 80cb961e5380a3878246d41341ff91378ca59e05 +Author: Jia Tan +Date: 2023-08-04 22:17:11 +0800 + + codespell: Add .codespellrc to set default options. + + The .codespellrc allows setting default options to avoid false positive + matches, set additional dictionaries, etc. For now, codespell can be + used locally before committing doc and comment changes. + + It should help prevent silly errors and fix up commits in the future. + + .codespellrc | 24 ++++++++++++++++++++++++ + 1 file changed, 24 insertions(+) + +commit cd678a6077358935249b64a4a16fe8d17434f9c9 +Author: Jia Tan +Date: 2023-08-03 20:10:21 +0800 + + Tests: Style fixes to test_lzip_decoder.c. + + tests/test_lzip_decoder.c | 36 ++++++++++++++++++++++++------------ + 1 file changed, 24 insertions(+), 12 deletions(-) + +commit 1cac5ed4fa45c9861d745b02d80575cb2ff01d81 +Author: Jia Tan +Date: 2023-08-03 15:56:20 +0800 + + Translations: Update the Chinese (simplified) translation. + + po/zh_CN.po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 16068f6c30b888cdb873f6285af941d00f95741d +Author: Lasse Collin +Date: 2023-08-02 17:15:12 +0300 + + xz: Omit an empty paragraph on the man page. + + src/xz/xz.1 | 1 - + 1 file changed, 1 deletion(-) + +commit 9ae4371b5106189486e850ce777e40f7b6021c0b +Author: Jia Tan +Date: 2023-08-02 20:30:07 +0800 + + Add NEWS for 5.4.4. + + NEWS | 43 +++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 43 insertions(+) + +commit e8c2203b2c76466d8d3387c5212b46151de8e605 +Author: Lasse Collin +Date: 2023-08-02 15:19:43 +0300 + + build-aux/manconv.sh: Fix US-ASCII and UTF-8 output. + + groff defaults to SGR escapes. Using -P-c passes -c to grotty + which restores the old behavior. Perhaps there is a better way to + get pure plain text output but this works for now. + + build-aux/manconv.sh | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 9a706167b0d903d92fd134895acb4bc6a5e3e688 +Author: Lasse Collin +Date: 2023-08-01 19:10:43 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 33e25a0f5650754c38bed640deedefe3b4fec5ef +Author: Lasse Collin +Date: 2023-08-01 18:22:24 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 81db3b889830132334d1f2129bdc93177ac2ca7d +Author: ChanTsune <41658782+ChanTsune@users.noreply.github.com> +Date: 2023-08-01 18:17:17 +0300 + + mythread.h: Disable signal functions in builds targeting Wasm + WASI. + + signal.h in WASI SDK doesn't currently provide sigprocmask() + or sigset_t. liblzma doesn't need them so this change makes + liblzma and xzdec build against WASI SDK. xz doesn't build yet + and the tests don't either as tuktest needs setjmp() which + isn't (yet?) implemented in WASI SDK. + + Closes: https://github.com/tukaani-project/xz/pull/57 + See also: https://github.com/tukaani-project/xz/pull/56 + + (The original commit was edited a little by Lasse Collin.) + + src/common/mythread.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 71c638c611324e606d324c8189fef8fe79db6991 +Author: Jia Tan +Date: 2023-08-01 21:58:51 +0800 + + Add newline to end of .gitignore. + + Newline was accidentally removed in commit + 01cbb7f023ee7fda8ddde04bd17cf7d3c2418706. + + .gitignore | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 42df7c7aa1cca385e509eb33c65136e61890f0bf +Author: Dimitri Papadopoulos Orfanos <3234522+DimitriPapadopoulos@users.noreply.github.com> +Date: 2023-07-31 14:02:21 +0200 + + Docs: Fix typos found by codespell + + CMakeLists.txt | 4 ++-- + NEWS | 2 +- + configure.ac | 2 +- + src/liblzma/api/lzma/container.h | 4 ++-- + src/liblzma/api/lzma/filter.h | 2 +- + src/liblzma/api/lzma/lzma12.h | 4 ++-- + src/liblzma/common/block_buffer_encoder.c | 2 +- + src/liblzma/common/common.h | 2 +- + src/liblzma/common/file_info.c | 2 +- + src/liblzma/common/lzip_decoder.c | 2 +- + src/liblzma/common/stream_decoder_mt.c | 8 ++++---- + src/liblzma/common/string_conversion.c | 6 +++--- + src/liblzma/lz/lz_encoder.h | 2 +- + src/liblzma/lzma/lzma_encoder.c | 4 ++-- + src/xz/hardware.c | 4 ++-- + tests/test_filter_flags.c | 4 ++-- + tests/test_index.c | 2 +- + tests/test_vli.c | 2 +- + 18 files changed, 29 insertions(+), 29 deletions(-) + +commit 01cbb7f023ee7fda8ddde04bd17cf7d3c2418706 +Author: Jia Tan +Date: 2023-07-26 20:26:23 +0800 + + Update .gitignore. + + .gitignore | 4 ++++ + 1 file changed, 4 insertions(+) + +commit f97a1afd564c48ad9cb94682e10972a72e11fa08 +Author: Jia Tan +Date: 2023-07-28 22:03:08 +0800 + + CMake: Conditionally allow the creation of broken symlinks. + + The CMake build will try to create broken symlinks on Unix and Unix-like + platforms. Cygwin and MSYS2 are Unix-like, but may not be able to create + broken symlinks. The value of the CYGWIN or MSYS environment variables + determine if broken symlinks are valid. + + The default for MSYS2 does not allow for broken symlinks, so the CMake + build has been broken for MSYS2 since commit + 80a1a8bb838842a2be343bd88ad1462c21c5e2c9. + + CMakeLists.txt | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++----- + 1 file changed, 75 insertions(+), 7 deletions(-) + +commit 7190f4cc7c9ade5b9b3675d0cbfa3b6d6ec9cb4f +Author: Jia Tan +Date: 2023-07-28 21:56:48 +0800 + + CI: Fix windows-ci dependency installation. + + All of the MSYS2 environments need make, and it does not come with the + toolchain package. The toolchain package will install the needed + compiler toolchains since without this package CMake cannot properly + generate the Makefiles. + + .github/workflows/windows-ci.yml | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit a048f472cd9a2245265cb292853cbbcdd4f02001 +Author: Jia Tan +Date: 2023-07-28 21:54:22 +0800 + + CI: Update ci_build.sh CMake to always make Unix Makefiles. + + The default for many of the MSYS2 environments is for CMake to create + Ninja build files. This would complicate the build script since we would + need a different command to run the tests. Its simpler to always use + Unix Makefiles so that "make test" is always a usable target for + testing. + + build-aux/ci_build.sh | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 7870396a0ca945473aa0d1d790f4cbef456610bd +Author: Jia Tan +Date: 2023-07-25 20:17:23 +0800 + + CI: Test CMake builds and test framework with MSYS2. + + .github/workflows/windows-ci.yml | 32 ++++++++++++++++++++------------ + 1 file changed, 20 insertions(+), 12 deletions(-) + +commit 6497d1f8875cb7e3007f714336cc09c06fed235b +Author: Jia Tan +Date: 2023-07-25 20:14:53 +0800 + + CI: Windows CI rename system matrix variable -> msys2_env. + + Calling the MSYS2 environment "system" was a bit vague and should be + more specific. + + .github/workflows/windows-ci.yml | 15 +++++---------- + 1 file changed, 5 insertions(+), 10 deletions(-) + +commit 785e4121d9b2921ad36bd3af1cf61fa20a9265bd +Author: Jia Tan +Date: 2023-07-24 23:11:45 +0800 + + CI: Add Clang64 MSYS2 environment to Windows CI. + + .github/workflows/windows-ci.yml | 1 + + 1 file changed, 1 insertion(+) + +commit d9166b52cf3458a4da3eb92224837ca8fc208d79 +Author: Jia Tan +Date: 2023-07-24 21:43:44 +0800 + + liblzma: Prevent an empty translation unit in Windows builds. + + To workaround Automake lacking Windows resource compiler support, an + empty source file is compiled to overwrite the resource files for static + library builds. Translation units without an external declaration are + not allowed by the C standard and result in a warning when used with + -Wempty-translation-unit (Clang) or -pedantic (GCC). + + src/liblzma/Makefile.am | 6 +++++- + 1 file changed, 5 insertions(+), 1 deletion(-) + +commit db5019d691f980d622fb56fdcf383af2c3519c98 +Author: Jia Tan +Date: 2023-07-22 18:37:56 +0800 + + Translations: Update the Vietnamese translation. + + po/vi.po | 45 ++++++++++++++++++++++++++++----------------- + 1 file changed, 28 insertions(+), 17 deletions(-) + +commit f3a055f762ba5b71b746fc2d44a6ababde2c61b5 +Author: Jia Tan +Date: 2023-07-22 14:55:42 +0800 + + CI: Add Windows runner for Autotools builds with MSYS2. + + Only a subset of the tests run by the Linux and MacOS Autotools builds + are run. The most interesting tests are the ones that disable threads, + encoders, and decoders. + + The Windows runner will only be run manually since these tests will + likely take much longer than the Linux and MacOS runners. This runner + should be used before merging any large features and before releases. + + Currently the clang64 environment fails to due to a warning and + -Werror is enabled for the CI tests. This is still an early version + since the CMake build can be done for MSVC and optionally each of the + MSYS2 environments. GitHub does not allow manually running the CI tests + unless the workflow is checked on the default branch so checking in a + minimum version is a good idea. + + Thanks to Arthur S for the original proposing the original patch. + + Closes: https://github.com/tukaani-project/xz/pull/34 + + .github/workflows/windows-ci.yml | 119 +++++++++++++++++++++++++++++++++++++++ + 1 file changed, 119 insertions(+) + +commit 556536a3525df9e5ed78b8c7057991cfa9edfac8 +Author: Jia Tan +Date: 2023-07-21 22:11:01 +0800 + + CI: Add argument to ci_build.sh to pass flags to autogen.sh. + + build-aux/ci_build.sh | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +commit 39a32d36fc465c4e70f13192eea380e518ba6e8a +Author: Jia Tan +Date: 2023-07-21 18:05:44 +0800 + + Tests: Skip .lz files in test_files.sh if not configured. + + Previously if the lzip decoder was not configured then test_files.sh + would pass the lzip tests instead of skipping them. + + tests/test_files.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 194d12724b30fe42789d12a0184f9d412c449347 +Author: Jia Tan +Date: 2023-07-20 22:11:13 +0800 + + Tests: Add ARM64 filter test to test_compress.sh. + + tests/test_compress.sh | 1 + + 1 file changed, 1 insertion(+) + +commit d850365c444368102c69beaddf849ed463c33467 +Author: Jia Tan +Date: 2023-07-20 20:30:05 +0800 + + Translations: Update the Croatian translation. + + po/hr.po | 49 ++++++++++++++++++++++++++++++------------------- + 1 file changed, 30 insertions(+), 19 deletions(-) + +commit 24049eb7acf6d42a60f00efe4e7289fe8e1797fe +Author: Jia Tan +Date: 2023-07-20 20:28:32 +0800 + + Translations: Update the Korean man page translations. + + po4a/ko.po | 1255 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 629 insertions(+), 626 deletions(-) + +commit 4d4a4fa07de6cb9d913fb2f97712fddda2527b49 +Author: Jia Tan +Date: 2023-07-20 20:25:24 +0800 + + Translations: Update the Korean translation. + + po/ko.po | 45 ++++++++++++++++++++++++++++----------------- + 1 file changed, 28 insertions(+), 17 deletions(-) + +commit 237f06d9c55cf438a7538a598354bcf103f23711 +Author: Jia Tan +Date: 2023-07-20 20:24:05 +0800 + + Translations: Update the Polish translation. + + po/pl.po | 47 +++++++++++++++++++++++++++++------------------ + 1 file changed, 29 insertions(+), 18 deletions(-) + +commit 80c2c832136656d5ac7a1bca8bc42d95e13d281a +Author: Jia Tan +Date: 2023-07-20 20:22:23 +0800 + + Translations: Update the German man page translations. + + po4a/de.po | 1255 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 629 insertions(+), 626 deletions(-) + +commit fdbde14503ca03069d3649aa51926f5f796b89d8 +Author: Jia Tan +Date: 2023-07-20 20:18:44 +0800 + + Translations: Update the German translation. + + po/de.po | 47 +++++++++++++++++++++++++++++------------------ + 1 file changed, 29 insertions(+), 18 deletions(-) + +commit 9f3bf5ff5b2b5cf0b252a2bf381238ca49dc4101 +Author: Jia Tan +Date: 2023-07-20 20:17:10 +0800 + + Translations: Update the Chinese (simplified) translation. + + po/zh_CN.po | 47 +++++++++++++++++++++++++++++------------------ + 1 file changed, 29 insertions(+), 18 deletions(-) + +commit 376938c588011567c74f1d5a160c0ccce6336d46 +Author: Jia Tan +Date: 2023-07-20 20:15:47 +0800 + + Translations: Update the Swedish translation. + + po/sv.po | 47 +++++++++++++++++++++++++++++------------------ + 1 file changed, 29 insertions(+), 18 deletions(-) + +commit 26b0bc6eb82c84559936a7c7080de5c71c8276f8 +Author: Jia Tan +Date: 2023-07-20 20:14:00 +0800 + + Translations: Update the Ukrainian man page translations. + + po4a/uk.po | 1253 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 628 insertions(+), 625 deletions(-) + +commit 2d02c8b7640b54f3c5aa1c8b5990ba56f322393b +Author: Jia Tan +Date: 2023-07-20 20:09:15 +0800 + + Translations: Update the Ukrainian translation. + + po/uk.po | 45 ++++++++++++++++++++++++++++----------------- + 1 file changed, 28 insertions(+), 17 deletions(-) + +commit f881018b503fd334331c24a09075429558abbce1 +Author: Jia Tan +Date: 2023-07-20 20:06:57 +0800 + + Translations: Update the Spanish translation. + + po/es.po | 47 +++++++++++++++++++++++++++++------------------ + 1 file changed, 29 insertions(+), 18 deletions(-) + +commit 791fe6d3ffd6877fa5f852be69d9251397dfaa31 +Author: Jia Tan +Date: 2023-07-20 20:05:19 +0800 + + Translations: Update the Romanian translation. + + po/ro.po | 48 ++++++++++++++++++++++++++++++------------------ + 1 file changed, 30 insertions(+), 18 deletions(-) + +commit 8827e90704f699fe08bb5bed56b1717a2bc0eb77 +Author: Jia Tan +Date: 2023-07-20 20:02:56 +0800 + + Translations: Update the Romanian man page translations. + + po4a/ro.po | 1254 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 629 insertions(+), 625 deletions(-) + +commit 0184d344fa4f215cd345bb131db9068e077c69b8 +Author: Jia Tan +Date: 2023-07-19 23:36:00 +0800 + + liblzma: Suppress -Wunused-function warning. + + Clang 16.0.0 and earlier have a bug that the ifunc resolver function + triggers the -Wunused-function warning. The resolver function is static + and only "used" by the __attribute__((__ifunc()__)). + + At this time, the bug is still unresolved, but has been reported: + https://github.com/llvm/llvm-project/issues/63957 + + This is not a problem in GCC. + + src/liblzma/check/crc64_fast.c | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit 43845fa70fc751736c44c18f4cee42d49bfd1392 +Author: Jia Tan +Date: 2023-07-18 22:52:25 +0800 + + liblzma: Reword lzma_str_list_filters() documentation. + + This further improves the documentation from commit + f36ca7982f6bd5e9827219ed4f3c5a1fbf5d7bdf. The previous wording of + "supported options" was slightly misleading since the options that are + printed are the ones that are relevant for encoding/decoding. It is not + about which options can or must be specified. + + src/liblzma/api/lzma/filter.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 818701ba1c9dff780b7fbf28f9ab8eb11a25dd67 +Author: Jia Tan +Date: 2023-07-18 22:49:57 +0800 + + liblzma: Improve comment in string_conversion.c. + + The comment used "flag" when referring to decoder options. Just + referring to them as options is more clear and consistent. + + src/liblzma/common/string_conversion.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit b6b7d065853cd4c3f5b8d9be8aea0b6dcb0fe090 +Author: Lasse Collin +Date: 2023-07-18 17:37:33 +0300 + + xz: Translate the second "%s: " in message.c since French needs "%s : ". + + This string is used to print a filename when using "xz -v" and + stderr isn't a terminal. + + src/xz/message.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit be644042c3066d8e7a2834f989671ba74d27f749 +Author: Lasse Collin +Date: 2023-07-18 14:35:33 +0300 + + xz: Make "%s: %s" translatable because French needs "%s : %s". + + src/xz/args.c | 5 ++++- + src/xz/coder.c | 8 ++++---- + src/xz/file_io.c | 8 ++++---- + src/xz/list.c | 11 ++++++----- + 4 files changed, 18 insertions(+), 14 deletions(-) + +commit 97fd5cb669ee0afc48d2087675ab166aff89eaa2 +Author: Lasse Collin +Date: 2023-07-18 13:57:54 +0300 + + liblzma: Tweak #if condition in memcmplen.h. + + Maybe ICC always #defines _MSC_VER on Windows but now + it's very clear which code will get used. + + src/liblzma/common/memcmplen.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 40392c19f71985852d75997f109dea97177d6f3f +Author: Lasse Collin +Date: 2023-07-18 13:49:43 +0300 + + liblzma: Omit unnecessary parenthesis in a preprocessor directive. + + src/liblzma/common/memcmplen.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit abc1d5601b7e419ebc28a1ab4b268613b52e6f98 +Author: Jia Tan +Date: 2023-07-18 00:51:48 +0800 + + xz: Update Authors list in a few files. + + src/xz/args.c | 3 ++- + src/xz/args.h | 3 ++- + src/xz/coder.c | 3 ++- + src/xz/coder.h | 3 ++- + src/xz/message.c | 3 ++- + 5 files changed, 10 insertions(+), 5 deletions(-) + +commit 289034a168878baa9df6ff6e159110aade69cba5 +Author: Jia Tan +Date: 2023-07-14 23:20:33 +0800 + + Docs: Add a new section to INSTALL for Tests. + + The new Tests section describes basic information about the tests, how + to run them, and important details when cross compiling. We have had a + few questions about how to compile the tests without running them, so + hopefully this information will help others with the same question in the + future. + + Fixes: https://github.com/tukaani-project/xz/issues/54 + + INSTALL | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++-------------- + 1 file changed, 64 insertions(+), 17 deletions(-) + +commit 1119e5f5a519b0ab71c81fc4dc84c0cc72abe513 +Author: Jia Tan +Date: 2023-07-14 21:10:27 +0800 + + Docs: Update README. + + This adds an entry to "Other implementations of the .xz format" for + XZ for Java. + + README | 4 ++++ + 1 file changed, 4 insertions(+) + +commit f99e2e4e53b7ea89e4eef32ddd4882e0416357c9 +Author: Jia Tan +Date: 2023-07-13 23:32:10 +0800 + + xz: Fix typo in man page. + + The Memory limit information section described three output + columns when it actually has six. This was reworded to + "multiple" to make it more future proof. + + src/xz/xz.1 | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f907705eb1f6c5edaafc9668a34c51a989932f1d +Author: Jia Tan +Date: 2023-07-13 21:46:12 +0800 + + xz: Minor clean up for coder.c + + * Moved max_block_list_size from a global to local variable. + * Reworded error message in validate_block_list_filter(). + * Removed helper function filter_chain_error(). + * Changed 1 << X to 1U << X in many places + + src/xz/coder.c | 53 +++++++++++++++++++++-------------------------------- + 1 file changed, 21 insertions(+), 32 deletions(-) + +commit 9adc9e56157ecbf2948e5036df8567809b9ae177 +Author: Jia Tan +Date: 2023-07-13 21:26:47 +0800 + + xz: Update man page Authors and date. + + src/xz/xz.1 | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit c12e429f2635da8d8f5749e5f733f451baca6945 +Author: Jia Tan +Date: 2023-06-20 20:32:59 +0800 + + xz: Add a section to man page for robot mode --filters-help. + + src/xz/xz.1 | 32 ++++++++++++++++++++++++++++++-- + 1 file changed, 30 insertions(+), 2 deletions(-) + +commit e10f2db5d10300c16fa482a136ed31c1aa6e8e8d +Author: Jia Tan +Date: 2023-06-19 23:11:41 +0800 + + xz: Slight reword in xz man page for consistency. + + Changed will print => prints in xz --robot --version description to + match --robot --info-memory description. + + src/xz/xz.1 | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit f5dc172a402fa946f3c45a16929d7fe14c9f5e81 +Author: Jia Tan +Date: 2023-06-19 23:07:10 +0800 + + xz: Reorder robot mode subsections in the man page. + + The order is now consistent with the order the command line arguments + are documented earlier in the man page. The new order is: + 1. --list + 2. --info-memory + 3. --version + + Instead of the previous order: + 1. --version + 2. --info-memory + 3. --list + + src/xz/xz.1 | 192 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 96 insertions(+), 96 deletions(-) + +commit 9628be23aef2784249fd9f3199799d785d2ec5cc +Author: Jia Tan +Date: 2023-05-13 00:46:50 +0800 + + xz: Update man page for new --filters-help option. + + src/xz/xz.1 | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit a165d7df1964121eb9df715e6f836a31c865beef +Author: Jia Tan +Date: 2023-05-13 00:44:41 +0800 + + xz: Add a new --filters-help option. + + The --filters-help can be used to help create filter chains with the + --filters and --filtersX options. The message in --long-help is too + short to fully explain the syntax to construct complex filter chains. + + In --robot mode, xz will only print the output from liblzma function + lzma_str_list_filters. + + src/xz/args.c | 8 ++++++++ + src/xz/message.c | 30 ++++++++++++++++++++++++++++++ + src/xz/message.h | 5 +++++ + 3 files changed, 43 insertions(+) + +commit 95f1a414b156ee35d3e71862a14915fdd138f913 +Author: Jia Tan +Date: 2023-04-21 20:28:11 +0800 + + xz: Update the man page for --block-list and --filtersX + + The --block-list option description needed updating since the new + --filtersX option changes how it can be used. The new entry for + --filters1=FILTERS ... --filter9=FILTERS was created right after + the --filters option. + + src/xz/xz.1 | 106 +++++++++++++++++++++++++++++++++++++++++++++--------------- + 1 file changed, 80 insertions(+), 26 deletions(-) + +commit 47a63cad2aa778280e0c1926b7159427ea028cb1 +Author: Jia Tan +Date: 2023-04-21 19:50:14 +0800 + + xz: Update --long-help for the new --filtersX option. + + src/xz/message.c | 12 ++++++++++-- + 1 file changed, 10 insertions(+), 2 deletions(-) + +commit 8b9913a13daca2550d02dfdcdc9be15f55ca4d13 +Author: Jia Tan +Date: 2023-06-17 20:46:21 +0800 + + xz: Ignore filter chains that are set but never used in --block-list. + + If a filter chain is set but not used in --block-list, it introduced + unexpected behavior such as requiring an unneeded amount of memory to + compress, reducing the number of threads in multi-threaded encoding, and + printing an incorrect amount of memory needed to decompress. + + This also renames filters_init_mask => filters_used_mask. A filter is + assumed to be used if it is specified in --filtersX until + coder_set_compression_settings() determines which filters are referenced + in --block-list. + + src/xz/coder.c | 66 ++++++++++++++++++++++++++++++++++++++++++---------------- + 1 file changed, 48 insertions(+), 18 deletions(-) + +commit 183819bfd9efac8c184d9bf123325719b7eee30f +Author: Jia Tan +Date: 2023-05-13 20:11:13 +0800 + + xz: Set the Block size for mt encoding correctly. + + When opt_block_size is not used, the Block size for mt encoder is + derived from the minimum of the largest Block specified by + --block-list and the recommended Block size on all filter chains + calculated by lzma_mt_block_size(). This avoids using unnecessary + memory and ensures that all Blocks are large enough for the most memory + needy filter chain. + + src/xz/coder.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++- + 1 file changed, 67 insertions(+), 1 deletion(-) + +commit afb2dbec3d857b026486b75e42a4728e12d234cb +Author: Jia Tan +Date: 2023-05-11 00:09:41 +0800 + + xz: Validate --flush-timeout for all specified filter chains. + + src/xz/coder.c | 24 ++++++++++++++++-------- + 1 file changed, 16 insertions(+), 8 deletions(-) + +commit 5f0c5a04388f8334962c70bc37a8c2ff8f605e0a +Author: Jia Tan +Date: 2023-05-13 19:54:33 +0800 + + xz: Allows --block-list filters to scale down memory usage. + + Previously, only the default filter chain could have its memory usage + adjusted. The filter chains specified with --filtersX were not checked + for memory usage. Now, all used filter chains will be adjusted if + necessary. + + src/xz/coder.c | 269 +++++++++++++++++++++++++++++++++++++++++++++------------ + 1 file changed, 214 insertions(+), 55 deletions(-) + +commit 479fd58d60622331fcbe48fddf756927b9f80d9a +Author: Jia Tan +Date: 2023-05-10 21:50:33 +0800 + + xz: Do not include block splitting if encoders are disabled. + + The block splitting logic and split_block() function are not needed if + encoders are disabled. This will help slightly reduce the binary size + when built without encoders and allow split_block() to use functions + that require encoders being enabled. + + src/xz/coder.c | 29 ++++++++++++++++++++--------- + 1 file changed, 20 insertions(+), 9 deletions(-) + +commit f86ede22500f7ae024ec3ec3f3489ab5a857a3b3 +Author: Jia Tan +Date: 2023-05-10 22:38:59 +0800 + + xz: Free filters[] in debug mode. + + This will only free filter chains created with --filters1-9 since the + default filter chain may be set from a static function variable. The + complexity to free the default filter chain is not worth the burden on + code maintenance. + + src/xz/coder.c | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit f281cd0d692ac0c70fc7669b80dddb863ea947e1 +Author: Jia Tan +Date: 2023-05-13 19:28:23 +0800 + + xz: Add a message if --block-list is used outside of xz compresssion. + + --block-list is only supported with compression in xz format. This avoids + silently ignoring when --block-list is unused. + + src/xz/args.c | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit d6af7f347077b22403133239592e478931307759 +Author: Jia Tan +Date: 2023-04-18 20:29:09 +0800 + + xz: Create command line options for filters[1-9]. + + The new command line options are meant to be combined with --block-list. + They work as an optional extension to --block-list to specify a custom + filter chain for each block listed. The new options allow the creation + of up to 9 reusable filter chains. For instance: + + xz --block-list=1:10MiB,3:5MiB,,2:5MiB,1:0 --filters1=delta--lzma2 \ + --filters2=x86--lzma2 --filters3=arm64--lzma2 + + Will create the following blocks: + 1. A block of size 10 MiB with filter chain delta, lzma2. + 2. A block of size 5 MiB with filter chain arm64, lzma2. + 3. A block of size 5 MiB with filter chain arm64, lzma2. + 4. A block of size 5 MiB with filter chain x86, lzma2. + 5. A block containing the rest of the file contents with filter chain + delta, lzma2. + + src/xz/args.c | 82 ++++++++++++++++++++++--- + src/xz/coder.c | 188 ++++++++++++++++++++++++++++++++++++++++++--------------- + src/xz/coder.h | 20 +++++- + 3 files changed, 230 insertions(+), 60 deletions(-) + +commit 072d29250113268536719ad0e040ab8a66fb6435 +Author: Jia Tan +Date: 2023-05-13 19:36:09 +0800 + + xz: Use lzma_filters_free() in forget_filter_chain(). + + This is a little cleaner than the previous implementation of + forget_filter_chain(). It is also more consistent since + lzma_str_to_filters() will always terminate the filter chain so there + is no need to terminate it later in coder_set_compression_settings(). + + src/xz/coder.c | 18 ++++++++++-------- + 1 file changed, 10 insertions(+), 8 deletions(-) + +commit 3d21da5cff4b511633cb6e0d8a1090485c0c1059 +Author: Jia Tan +Date: 2023-04-17 22:22:45 +0800 + + xz: Separate string to filter conversion into a helper function. + + Converting from string to filter will also need to be done for block + specific filter chains. + + src/xz/coder.c | 33 ++++++++++++++++++++------------- + 1 file changed, 20 insertions(+), 13 deletions(-) + +commit a6583726e5f950278f96abcf79c04f1056810be6 +Author: Jia Tan +Date: 2023-01-06 00:03:35 +0800 + + Tests: Use new --filters option in test_compress.sh + + tests/test_compress.sh | 20 ++++++++++---------- + 1 file changed, 10 insertions(+), 10 deletions(-) + +commit 5f3b898d07cc9b7160c7c88b3120b7edabb8a5b0 +Author: Jia Tan +Date: 2023-01-06 00:03:06 +0800 + + xz: Update --long-help and man page for new --filters option. + + src/xz/message.c | 6 ++++++ + src/xz/xz.1 | 41 ++++++++++++++++++++++++++++++++++++----- + 2 files changed, 42 insertions(+), 5 deletions(-) + +commit 9ded880a0221f4d1256845fc4ab957ffd377c760 +Author: Jia Tan +Date: 2023-01-06 00:02:29 +0800 + + xz: Add --filters option to CLI. + + The --filters option uses the new lzma_str_to_filters() function + to convert a string into a full filter chain. Using this option + will reset all previous filters set by --preset, --[filter], or + --filters. + + src/xz/args.c | 9 +++++++-- + src/xz/coder.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++-- + src/xz/coder.h | 3 +++ + 3 files changed, 58 insertions(+), 4 deletions(-) + +commit 2c189bb00af73dc7ba1a67a9d274d5be03ee3a88 +Author: Jia Tan +Date: 2023-07-14 21:30:25 +0800 + + Tests: Improve feature testing for skipping. + + Fixed a bug where test_compress_* would all fail if arm64 or armthumb + filters were enabled for compression but arm was disabled. Since the + grep tests only checked for "define HAVE_ENCODER_ARM", this would match + on HAVE_ENCODER_ARM64 or HAVE_ENCODER_ARMTHUMB. + + Now the config.h feature test requires " 1" at the end to prevent the + prefix problem. have_feature() was also updated for this even though + there were known current bugs affecting it. This is just in case future + features have a similar prefix problem. + + tests/test_compress.sh | 4 ++-- + tests/test_files.sh | 2 +- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 80a6b9bcad016c99c9ba3f3eeb4a619fcadfd357 +Author: Jia Tan +Date: 2023-07-10 20:56:28 +0800 + + Translations: Update the Chinese (traditional) translation. + + po/zh_TW.po | 659 ++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 377 insertions(+), 282 deletions(-) + +commit 17f8844e6fc355abf997d77637a7447c4f7bbcbd +Author: Jia Tan +Date: 2023-07-08 21:24:19 +0800 + + liblzma: Remove non-portable empty initializer. + + Commit 78704f36e74205857c898a351c757719a6c8b666 added an empty + initializer {} to prevent a warning. The empty initializer is a GNU + extension and results in a build failure on MSVC. The -wpedantic flag + warns about empty initializers. + + src/liblzma/common/stream_encoder_mt.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 3aca4f629cd577f0c54f594d5d88722edf0b0413 +Author: Jia Tan +Date: 2023-07-08 20:03:59 +0800 + + Translations: Update the Vietnamese translation. + + po/vi.po | 620 +++++++++++++++++++++++++++++++++++---------------------------- + 1 file changed, 349 insertions(+), 271 deletions(-) + +commit 66bdcfa85fef2911cc80f5f30fed3f9610faccb4 +Author: Jia Tan +Date: 2023-06-28 20:46:31 +0800 + + Tests: Fix memory leaks in test_index. + + Several tests were missing calls to lzma_index_end() to clean up the + lzma_index structs. The memory leaks were discovered by using + -fsanitize=address with GCC. + + tests/test_index.c | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit fe3bd438fb119f9bad3f08dc29d331e4956196e1 +Author: Jia Tan +Date: 2023-06-28 20:43:29 +0800 + + Tests: Fix memory leaks in test_block_header. + + test_block_header was not properly freeing the filter options between + calls to lzma_block_header_decode(). The memory leaks were discovered by + using -fsanitize=address with GCC. + + tests/test_block_header.c | 38 ++++++++++++++++++++++---------------- + 1 file changed, 22 insertions(+), 16 deletions(-) + +commit 78704f36e74205857c898a351c757719a6c8b666 +Author: Jia Tan +Date: 2023-06-28 20:31:11 +0800 + + liblzma: Prevent uninitialzed warning in mt stream encoder. + + This change only impacts the compiler warning since it was impossible + for the wait_abs struct in stream_encode_mt() to be used before it was + initialized since mythread_condtime_set() will always be called before + mythread_cond_timedwait(). + + Since the mythread.h code is different between the POSIX and + Windows versions, this warning was only present on Windows builds. + + Thanks to Arthur S for reporting the warning and providing an initial + patch. + + src/liblzma/common/stream_encoder_mt.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit e3356a204c5ae02db3ec4552b6c1be354e9b6142 +Author: Jia Tan +Date: 2023-06-28 20:22:38 +0800 + + liblzma: Prevent warning for MSYS2 Windows build. + + In lzma_memcmplen(), the header file is only included if + _MSC_VER and _M_X64 are both defined but _BitScanForward64() was + previously used if _M_X64 was defined. GCC for MSYS2 defines _M_X64 but + not _MSC_VER so _BitScanForward64() was used without including + . + + Now, lzma_memcmplen() will use __builtin_ctzll() for MSYS2 GCC builds as + expected. + + src/liblzma/common/memcmplen.h | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +commit 45e250a9e9f3c3e8e8af2983366b170bf54f890e +Author: Jia Tan +Date: 2023-06-28 21:01:22 +0800 + + CI: Add test with -fsanitize=address,undefined. + + ci_build.sh was updated to accept disabling of __attribute__ ifunc + and CLMUL. This will allow -fsanitize=address to pass because ifunc + is incompatible with -fsanitize=address. The CLMUL implementation has + optimizations that potentially read past the buffer and mask out the + unwanted bytes. + + This test will only run on Autotools Linux. + + .github/workflows/ci.yml | 23 +++++++++++++++++++---- + build-aux/ci_build.sh | 8 +++++++- + 2 files changed, 26 insertions(+), 5 deletions(-) + +commit 596ee722cd7ddf0afae584fc06365adc0e735977 +Author: Jia Tan +Date: 2023-06-28 20:16:04 +0800 + + CI: Upgrade checkout action from v2 to v3. + + .github/workflows/ci.yml | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 86118ea320f867e09e98a8682cc08cbbdfd640e2 +Author: Jia Tan +Date: 2023-06-27 23:38:32 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 3d1fdddf92321b516d55651888b9c669e254634e +Author: Jia Tan +Date: 2023-06-27 17:27:09 +0300 + + Docs: Document the configure option --disable-ifunc in INSTALL. + + INSTALL | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit b4cf7a2822e8d30eb2b12a1a07fd04383b10ade3 +Author: Lasse Collin +Date: 2023-06-27 17:24:49 +0300 + + Minor tweaks to style and comments. + + CMakeLists.txt | 8 ++++---- + configure.ac | 9 +++++---- + 2 files changed, 9 insertions(+), 8 deletions(-) + +commit 23fb9e3a329117c2968c1e7388b6ef07c782dba1 +Author: Lasse Collin +Date: 2023-06-27 17:19:49 +0300 + + CMake: Rename CHECK_ATTR_IFUNC to ALLOW_ATTR_IFUNC. + + It's so that there's a clear difference in wording compared + to liblzma's integrity check types. + + CMakeLists.txt | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit ee44863ae88e377a5df10db007ba9bfadde3d314 +Author: Lasse Collin +Date: 2023-06-27 17:05:23 +0300 + + liblzma: Add ifunc implementation to crc64_fast.c. + + The ifunc method avoids indirection via the function pointer + crc64_func. This works on GNU/Linux and probably on FreeBSD too. + The previous __attribute((__constructor__)) method is kept for + compatibility with ELF platforms which do support ifunc. + + The ifunc method has some limitations, for example, building + liblzma with -fsanitize=address will result in segfaults. + The configure option --disable-ifunc must be used for such builds. + + Thanks to Hans Jansen for the original patch. + Closes: https://github.com/tukaani-project/xz/pull/53 + + src/liblzma/check/crc64_fast.c | 35 ++++++++++++++++++++++++++--------- + 1 file changed, 26 insertions(+), 9 deletions(-) + +commit b72d21202402a603db6d512fb9271cfa83249639 +Author: Hans Jansen +Date: 2023-06-22 19:49:30 +0200 + + Add ifunc check to CMakeLists.txt + + CMake build system will now verify if __attribute__((__ifunc__())) can be + used in the build system. If so, HAVE_FUNC_ATTRIBUTE_IFUNC will be + defined to 1. + + CMakeLists.txt | 19 +++++++++++++++++++ + 1 file changed, 19 insertions(+) + +commit 23b5c36fb71904bfbe16bb20f976da38dadf6c3b +Author: Hans Jansen +Date: 2023-06-22 19:46:55 +0200 + + Add ifunc check to configure.ac + + configure.ac will now verify if __attribute__((__ifunc__())) can be used in + the build system. If so, HAVE_FUNC_ATTRIBUTE_IFUNC will be defined to 1. + + configure.ac | 28 ++++++++++++++++++++++++++++ + 1 file changed, 28 insertions(+) + +commit dbb3a536ed9873ffa0870321f6873e564c6a9da8 +Author: Jia Tan +Date: 2023-06-07 00:18:30 +0800 + + CI: Add apt update command before installing dependencies. + + Without the extra command, all of the CI tests were automatically + failing because the Ubuntu servers could not be reached properly. + + .github/workflows/ci.yml | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit 6bcd516812331de42b347922913230895bebad34 +Author: Jia Tan +Date: 2023-06-07 00:10:38 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 0d94ba69220d894d2a86081821d2d7a89df5a10b +Author: Benjamin Buch +Date: 2023-06-06 15:32:45 +0200 + + CMake: Protects against double find_package + + Boost iostream uses `find_package` in quiet mode and then again uses + `find_package` with required. This second call triggers a + `add_library cannot create imported target "ZLIB::ZLIB" because another + target with the same name already exists.` + + This can simply be fixed by skipping the alias part on secondary + `find_package` runs. + + CMakeLists.txt | 16 +++++++++------- + 1 file changed, 9 insertions(+), 7 deletions(-) + +commit 045d7aae286ecd2ce163be9e0d9041343a03f89a +Author: Jia Tan +Date: 2023-05-31 20:26:42 +0800 + + Translations: Update the Esperanto translation. + + po/eo.po | 185 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 92 insertions(+), 93 deletions(-) + +commit b0cc7c2dcefe4cbc4e1e697598c14fb687ed0b78 +Author: Jia Tan +Date: 2023-05-31 20:25:00 +0800 + + Translations: Update the Croatian translation. + + po/hr.po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit af045ef6f848f02cd14c9ad195a5f87bb0c02dce +Author: Jia Tan +Date: 2023-05-31 20:15:53 +0800 + + Translations: Update the Chinese (simplified) translation. + + po/zh_CN.po | 317 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 157 insertions(+), 160 deletions(-) + +commit e6b92d5817fe91ad27a0f7f57bd0f2144311e383 +Author: Jia Tan +Date: 2023-05-17 23:12:13 +0800 + + Translations: Update German translation of man pages. + + po4a/de.po | 52 ++++++++++++---------------------------------------- + 1 file changed, 12 insertions(+), 40 deletions(-) + +commit 592961ccdbba39c7d60fe37e36764232feb57c60 +Author: Jia Tan +Date: 2023-05-17 23:09:18 +0800 + + Translations: Update the German translation. + + po/de.po | 189 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 94 insertions(+), 95 deletions(-) + +commit 13572cb2c391f5b7503e333c6e05b20bd5bbb524 +Author: Jia Tan +Date: 2023-05-17 20:30:01 +0800 + + Translations: Update the Croatian translation. + + po/hr.po | 187 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 93 insertions(+), 94 deletions(-) + +commit 4e6e425ea8f097c6fb43e69cc9540294dca3680d +Author: Jia Tan +Date: 2023-05-17 20:26:54 +0800 + + Translations: Update Korean translation of man pages. + + po4a/ko.po | 3015 ++++++++++++------------------------------------------------ + 1 file changed, 568 insertions(+), 2447 deletions(-) + +commit d5ef1f6faf7c270f60093629257150085ecf19ca +Author: Jia Tan +Date: 2023-05-17 20:13:01 +0800 + + Translations: Update the Korean translation. + + po/ko.po | 319 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 158 insertions(+), 161 deletions(-) + +commit e22d0b0f2e301e7906d0106689d967ed84362028 +Author: Jia Tan +Date: 2023-05-16 23:49:09 +0800 + + Translations: Update the Spanish translation. + + po/es.po | 319 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 158 insertions(+), 161 deletions(-) + +commit f50da74d52d01f6cfd826a921249e289cf671678 +Author: Jia Tan +Date: 2023-05-16 23:47:23 +0800 + + Translations: Update the Romanian translation. + + po/ro.po | 195 ++++++++++++++++++++++++++++++++------------------------------- + 1 file changed, 98 insertions(+), 97 deletions(-) + +commit 4b9ad60a7305e9841b7cb4ea611bdf5fa7271696 +Author: Jia Tan +Date: 2023-05-16 23:45:43 +0800 + + Translations: Update Romanian translation of man pages. + + po4a/ro.po | 19 ++++++++++--------- + 1 file changed, 10 insertions(+), 9 deletions(-) + +commit cb6fd57f889c5d9fab36ae8c9e10083a5fe32dea +Author: Jia Tan +Date: 2023-05-16 23:43:51 +0800 + + Translations: Update Ukrainian translation of man pages. + + po4a/uk.po | 12 ++++++------ + 1 file changed, 6 insertions(+), 6 deletions(-) + +commit c3e8fcbc2db4861f92ad15606c995bd255803c52 +Author: Jia Tan +Date: 2023-05-16 23:37:54 +0800 + + Translations: Update the Ukrainian translation. + + po/uk.po | 321 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 159 insertions(+), 162 deletions(-) + +commit 27b81b84fcedbc55aa6e6b21004c44070b15b038 +Author: Jia Tan +Date: 2023-05-16 23:07:35 +0800 + + Translations: Update the Polish translation. + + po/pl.po | 316 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 155 insertions(+), 161 deletions(-) + +commit 8024ad636a65ed6ea95c94d57255be4c6724d6ed +Author: Jia Tan +Date: 2023-05-16 22:52:14 +0800 + + Translations: Update the Swedish translation. + + po/sv.po | 319 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 158 insertions(+), 161 deletions(-) + +commit 6699a29673f227c4664826db485ed9f7596320d2 +Author: Jia Tan +Date: 2023-05-16 21:21:38 +0800 + + Translations: Update the Esperanto translation. + + po/eo.po | 34 +++++++++++++++++----------------- + 1 file changed, 17 insertions(+), 17 deletions(-) + +commit f36ca7982f6bd5e9827219ed4f3c5a1fbf5d7bdf +Author: Jia Tan +Date: 2023-05-13 21:21:54 +0800 + + liblzma: Slightly rewords lzma_str_list_filters() documentation. + + Reword "options required" to "supported options". The previous may have + suggested that the options listed were all required anytime a filter is + used for encoding or decoding. The reword makes this more clear that + adjusting the options is optional. + + src/liblzma/api/lzma/filter.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 3374a5359e52f1671d8f831d65827d5020fe2595 +Author: Jia Tan +Date: 2023-05-11 23:49:23 +0800 + + liblzma: Adds lzma_nothrow to MicroLZMA API functions. + + None of the liblzma functions may throw an exception, so this + attribute should be applied to all liblzma API functions. + + src/liblzma/api/lzma/container.h | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit 8f236574986e7c414c0ea059f441982d1387e6a4 +Author: Jia Tan +Date: 2023-05-09 20:20:06 +0800 + + liblzma: Exports lzma_mt_block_size() as an API function. + + The lzma_mt_block_size() was previously just an internal function for + the multithreaded .xz encoder. It is used to provide a recommended Block + size for a given filter chain. + + This function is helpful to determine the maximum Block size for the + multithreaded .xz encoder when one wants to change the filters between + blocks. Then, this determined Block size can be provided to + lzma_stream_encoder_mt() in the lzma_mt options parameter when + intializing the coder. This requires one to know all the filter chains + they are using before starting to encode (or at least the filter chain + that will need the largest Block size), but that isn't a bad limitation. + + src/liblzma/api/lzma/container.h | 28 ++++++++++++++++++++++++++++ + src/liblzma/common/filter_encoder.c | 16 ++++++++++------ + src/liblzma/common/filter_encoder.h | 6 +----- + src/liblzma/common/stream_encoder_mt.c | 20 +++++++++----------- + src/liblzma/liblzma_generic.map | 5 +++++ + src/liblzma/liblzma_linux.map | 5 +++++ + src/liblzma/lzma/lzma2_encoder.c | 3 +++ + 7 files changed, 61 insertions(+), 22 deletions(-) + +commit d0f33d672a4da7985ebb5ba8d829f885de49c171 +Author: Jia Tan +Date: 2023-05-08 22:58:09 +0800 + + liblzma: Creates IS_ENC_DICT_SIZE_VALID() macro. + + This creates an internal liblzma macro to test if the dictionary size + is valid for encoding. + + src/liblzma/lz/lz_encoder.c | 4 +--- + src/liblzma/lz/lz_encoder.h | 8 ++++++++ + 2 files changed, 9 insertions(+), 3 deletions(-) + +commit c247d06e1f6cada9a76f4f6225cbd97ea760f52f +Author: Jia Tan +Date: 2023-05-02 20:39:56 +0800 + + Add NEWS for 5.4.3. + + NEWS | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit 77050b78364ffb6b0f129e742b7c31602d725c08 +Author: Jia Tan +Date: 2023-05-02 20:39:37 +0800 + + Add NEWS for 5.2.12. + + NEWS | 14 ++++++++++++++ + 1 file changed, 14 insertions(+) + +commit 713e15e43eb6279a7ab4bbad3d1325ebfdcf09a0 +Author: Jia Tan +Date: 2023-05-04 20:38:52 +0800 + + Translations: Update the Croatian translation. + + po/hr.po | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 9ad64bdf309844b6ca6c3e8a4dfb6dbaedda0ca9 +Author: Jia Tan +Date: 2023-05-04 20:30:25 +0800 + + tuklib_integer.h: Reverts previous commit. + + Previous commit 6be460dde07113fe3f08f814b61ddc3264125a96 would cause an + error if the integer size was 32 bit. + + src/common/tuklib_integer.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 6be460dde07113fe3f08f814b61ddc3264125a96 +Author: Jia Tan +Date: 2023-05-04 19:25:20 +0800 + + tuklib_integer.h: Changes two other UINT_MAX == UINT32_MAX to >=. + + src/common/tuklib_integer.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 44c0c5eae990a22ef04e9b88c1a15838a0d00878 +Author: Lasse Collin +Date: 2023-05-03 22:46:42 +0300 + + tuklib_integer.h: Fix a recent copypaste error in Clang detection. + + Wrong line was changed in 7062348bf35c1e4cbfee00ad9fffb4a21aa6eff7. + Also, this has >= instead of == since ints larger than 32 bits would + work too even if not relevant in practice. + + src/common/tuklib_integer.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 2cf5ae5b5b279b0b2e69ca4724e7bd705865fe68 +Author: Jia Tan +Date: 2023-04-25 20:06:15 +0800 + + CI: Adds a build and test for small configuration. + + .github/workflows/ci.yml | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 16b81a057a87c2f18e6ed6447f003af0cbdcfe43 +Author: Jia Tan +Date: 2023-04-25 20:05:26 +0800 + + CI: ci_build.sh allows configuring small build. + + build-aux/ci_build.sh | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit 78ccd93951f9e988d447bcdd70b24f6df5448d1d +Author: Jia Tan +Date: 2023-04-20 20:15:00 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit f41df2ac2fed347d3f107f3533e76e000d29c6cb +Author: Jia Tan +Date: 2023-04-19 22:22:16 +0800 + + Windows: Include when needed. + + Legacy Windows did not need to #include to use the MSVC + intrinsics. Newer versions likely just issue a warning, but the MSVC + documentation says to include the header file for the intrinsics we use. + + GCC and Clang can "pretend" to be MSVC on Windows, so extra checks are + needed in tuklib_integer.h to only include when it will is + actually needed. + + src/common/tuklib_integer.h | 6 ++++++ + src/liblzma/common/memcmplen.h | 10 ++++++++++ + 2 files changed, 16 insertions(+) + +commit 7062348bf35c1e4cbfee00ad9fffb4a21aa6eff7 +Author: Jia Tan +Date: 2023-04-19 21:59:03 +0800 + + tuklib_integer: Use __builtin_clz() with Clang. + + Clang has support for __builtin_clz(), but previously Clang would + fallback to either the MSVC intrinsic or the regular C code. This was + discovered due to a bug where a new version of Clang required the + header file in order to use the MSVC intrinsics. + + Thanks to Anton Kochkov for notifying us about the bug. + + src/common/tuklib_integer.h | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 3938718ce3773c90755785c0df8777f133b7ae29 +Author: Lasse Collin +Date: 2023-04-14 18:42:33 +0300 + + liblzma: Update project maintainers in lzma.h. + + AUTHORS was updated earlier, lzma.h was simply forgotten. + + src/liblzma/api/lzma.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2a89670ab295e377f8b44f5bda6d198deb8ea285 +Author: Jia Tan +Date: 2023-04-13 20:45:19 +0800 + + liblzma: Cleans up old commented out code. + + src/liblzma/common/alone_encoder.c | 11 ----------- + 1 file changed, 11 deletions(-) + +commit 0fbb2b87a7b5a1dd9d0f4a5e84ac7919557dbe81 +Author: Jia Tan +Date: 2023-04-07 20:46:41 +0800 + + Docs: Add missing word to SECURITY.md. + + .github/SECURITY.md | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit fb9c50f38a17bf37581de4034b36c8df8ec90a87 +Author: Jia Tan +Date: 2023-04-07 20:43:22 +0800 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 537c6cd8a9db0dd6b13683e64ddac2943190d715 +Author: Jia Tan +Date: 2023-04-07 20:42:12 +0800 + + Docs: Minor edits to SECURITY.md. + + .github/SECURITY.md | 25 ++++++++++++++++++++----- + 1 file changed, 20 insertions(+), 5 deletions(-) + +commit 6549df8dd53f358345957e232648fdb699930074 +Author: Gabriela Gutierrez +Date: 2023-04-07 12:08:30 +0000 + + Docs: Create SECURITY.md + + Signed-off-by: Gabriela Gutierrez + + .github/SECURITY.md | 14 ++++++++++++++ + 1 file changed, 14 insertions(+) + +commit d0faa85df5a5d253a4625d45313cf5e9277e6cd2 +Author: Jia Tan +Date: 2023-03-28 22:48:24 +0800 + + CI: Tests for disabling threading on CMake builds. + + .github/workflows/ci.yml | 3 --- + build-aux/ci_build.sh | 4 ++-- + 2 files changed, 2 insertions(+), 5 deletions(-) + +commit 8be5cc3b1359d88b4b30a39067466c0ae0bfbc4d +Author: Jia Tan +Date: 2023-03-28 22:45:42 +0800 + + CI: Removes CMakeCache.txt between builds. + + If the cache file is not removed, CMake will not reset configurations + back to their default values. In order to make the tests independent, it + is simplest to purge the cache. Unfortunatly, this will slow down the + tests a little and repeat some checks. + + build-aux/ci_build.sh | 2 ++ + 1 file changed, 2 insertions(+) + +commit 2cb6028fc31de082b7f927632363bb1426b61aaa +Author: Jia Tan +Date: 2023-03-28 22:32:40 +0800 + + CMake: Update liblzma-config.cmake generation. + + Now that the threading is configurable, the liblzma CMake package only + needs the threading library when using POSIX threads. + + CMakeLists.txt | 33 ++++++++++++++++++++++----------- + 1 file changed, 22 insertions(+), 11 deletions(-) + +commit 4d7fac0b07cc722825ba8d7838c558827e635611 +Author: Jia Tan +Date: 2023-03-28 22:25:33 +0800 + + CMake: Allows setting thread method. + + The thread method is now configurable for the CMake build. It matches + the Autotools build by allowing ON (pick the best threading method), + OFF (no threading), posix, win95, and vista. If both Windows and + posix threading are both available, then ON will choose Windows + threading. Windows threading will also not use: + + target_link_libraries(liblzma Threads::Threads) + + since on systems like MinGW-w64 it would link the posix threads + without purpose. + + CMakeLists.txt | 144 +++++++++++++++++++++++++++++++++++++++++---------------- + 1 file changed, 104 insertions(+), 40 deletions(-) + +commit 20cd905d898c1494dee42b78530769bb9c9f8076 +Author: Jia Tan +Date: 2023-03-24 23:05:48 +0800 + + CI: Runs CMake feature tests. + + Now, CMake will run similar feature disable tests that the Autotools + version did before. In order to do this without repeating lines in + ci.yml, it now makes sense to use the GitHub Workflow matrix to create + a loop. + + .github/workflows/ci.yml | 169 +++++++++++++++-------------------------------- + 1 file changed, 55 insertions(+), 114 deletions(-) + +commit 4fabdb269f1fc5624b3b94a170c4efb329d1d229 +Author: Jia Tan +Date: 2023-03-24 20:35:11 +0800 + + CI: ci_build.sh allows CMake features to be configured. + + Also included various clean ups for style and helper functions for + repeated work. + + build-aux/ci_build.sh | 233 +++++++++++++++++++++++++++++++------------------- + 1 file changed, 143 insertions(+), 90 deletions(-) + +commit cf3d1f130e50cf63da4bb1031771605f6f443b6a +Author: Jia Tan +Date: 2023-03-24 20:06:33 +0800 + + CI: Change ci_build.sh to use bash instead of sh. + + This script is only meant to be run as part of the CI build/test process + on machines that are known to have bash (Ubuntu and MacOS). If this + assumption changes in the future, then the bash specific commands will + need to be replaced with a more portable option. For now, it is + convenient to use bash commands. + + build-aux/ci_build.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ddfe164368e779c40d061aa4ccc376129e92f8e1 +Author: Jia Tan +Date: 2023-03-24 20:05:59 +0800 + + CMake: Only build xzdec if decoders are enabled. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 116e81f002c503d3c3cd12726db8f9116e58ef25 +Author: Jia Tan +Date: 2023-03-22 15:42:04 +0800 + + Build: Removes redundant check for LZMA1 filter support. + + src/liblzma/lzma/Makefile.inc | 5 +---- + 1 file changed, 1 insertion(+), 4 deletions(-) + +commit 0ba234f692772595329d225462d391fe2c199d0a +Author: Lasse Collin +Date: 2023-03-23 15:14:29 +0200 + + CMake: Bump maximum policy version to 3.26. + + It adds only one new policy related to FOLDERS which we don't use. + This makes it clear that the code is compatible with the policies + up to 3.26. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b0891684b4436aed31510fddcbb218d513bd5489 +Author: Jia Tan +Date: 2023-03-21 23:36:00 +0800 + + CMake: Conditionally build xz list.* files if decoders are enabled. + + CMakeLists.txt | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +commit 2c1a830efb61d9d65906a09c9ee3ce27c2c49227 +Author: Jia Tan +Date: 2023-02-25 11:46:50 +0800 + + CMake: Allow configuring features as cache variables. + + This allows users to change the features they build either in + CMakeCache.txt or by using a CMake GUI. The sources built for + liblzma are affected by this too, so only the necessary files + will be compiled. + + CMakeLists.txt | 528 ++++++++++++++++++++++++++++++++++++++++++--------------- + 1 file changed, 391 insertions(+), 137 deletions(-) + +commit 8be136f667aaeb8f9e16fbd57a83cb282f0c27ff +Author: Lasse Collin +Date: 2023-03-21 14:07:51 +0200 + + Build: Add a comment that AC_PROG_CC_C99 is needed for Autoconf 2.69. + + It's obsolete in Autoconf >= 2.70 and just an alias for AC_PROG_CC + but Autoconf 2.69 requires AC_PROG_CC_C99 to get a C99 compiler. + + configure.ac | 3 +++ + 1 file changed, 3 insertions(+) + +commit 53cc475f2652d9e390ca002018dfd0af0626ef80 +Author: Lasse Collin +Date: 2023-03-21 14:04:37 +0200 + + Build: configure.ac: Use AS_IF and AS_CASE where required. + + This makes no functional difference in the generated configure + (at least with the Autotools versions I have installed) but this + change might prevent future bugs like the one that was just + fixed in the commit 5a5bd7f871818029d5ccbe189f087f591258c294. + + configure.ac | 30 +++++++++++++++--------------- + 1 file changed, 15 insertions(+), 15 deletions(-) + +commit 3b8890a40233b6c783bb101ec14405e786871775 +Author: Lasse Collin +Date: 2023-03-21 13:12:03 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 5a5bd7f871818029d5ccbe189f087f591258c294 +Author: Lasse Collin +Date: 2023-03-21 13:11:49 +0200 + + Build: Fix --disable-threads breaking the building of shared libs. + + This is broken in the releases 5.2.6 to 5.4.2. A workaround + for these releases is to pass EGREP='grep -E' as an argument + to configure in addition to --disable-threads. + + The problem appeared when m4/ax_pthread.m4 was updated in + the commit 6629ed929cc7d45a11e385f357ab58ec15e7e4ad which + introduced the use of AC_EGREP_CPP. AC_EGREP_CPP calls + AC_REQUIRE([AC_PROG_EGREP]) to set the shell variable EGREP + but this was only executed if POSIX threads were enabled. + Libtool code also has AC_REQUIRE([AC_PROG_EGREP]) but Autoconf + omits it as AC_PROG_EGREP has already been required earlier. + Thus, if not using POSIX threads, the shell variable EGREP + would be undefined in the Libtool code in configure. + + ax_pthread.m4 is fine. The bug was in configure.ac which called + AX_PTHREAD conditionally in an incorrect way. Using AS_CASE + ensures that all AC_REQUIREs get always run. + + Thanks to Frank Busse for reporting the bug. + Fixes: https://github.com/tukaani-project/xz/issues/45 + + configure.ac | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit dfe1710784c0a3c3a90c17b80c9e1fe19b5fce06 +Author: Lasse Collin +Date: 2023-03-19 22:45:59 +0200 + + liblzma: Silence -Wsign-conversion in SSE2 code in memcmplen.h. + + Thanks to Christian Hesse for reporting the issue. + Fixes: https://github.com/tukaani-project/xz/issues/44 + + src/liblzma/common/memcmplen.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit f0c580c5fc38bf49a184b48d76c1d8c057d499ce +Author: Jia Tan +Date: 2023-03-18 22:10:57 +0800 + + Add NEWS for 5.4.2. + + NEWS | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 48 insertions(+) + +commit af4925e6043113ec9b5f9c0cf13abf2a18ccb1f6 +Author: Jia Tan +Date: 2023-03-18 22:10:12 +0800 + + Add NEWS for 5.2.11. + + NEWS | 27 +++++++++++++++++++++++++++ + 1 file changed, 27 insertions(+) + +commit 5a7b930efa7f9849d8da8397e8e5d8638f92be40 +Author: Lasse Collin +Date: 2023-03-18 16:00:54 +0200 + + Update the copy of GNU GPLv3 from gnu.org to COPYING.GPLv3. + + COPYING.GPLv3 | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +commit b473a92891f7e991398a3b5eff305f6f2b6d7293 +Author: Lasse Collin +Date: 2023-03-18 15:51:57 +0200 + + Change a few HTTP URLs to HTTPS. + + The xz man page timestamp was intentionally left unchanged. + + INSTALL | 2 +- + README | 8 ++++---- + configure.ac | 2 +- + dos/INSTALL.txt | 4 ++-- + src/liblzma/api/lzma.h | 8 ++++---- + src/liblzma/check/sha256.c | 2 +- + src/xz/xz.1 | 2 +- + windows/INSTALL-MinGW.txt | 10 +++++----- + 8 files changed, 19 insertions(+), 19 deletions(-) + +commit 8b2f6001b4f412c259a7883427f2f2c8cea98ea8 +Author: Jia Tan +Date: 2023-03-18 00:40:28 +0800 + + CMake: Fix typo in a comment. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 76e2315e14c399c15cc90e7930fd4d3d086b0227 +Author: Lasse Collin +Date: 2023-03-17 18:36:22 +0200 + + Windows: build.bash: Copy liblzma API docs to the output package. + + windows/build.bash | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 133cf55edc5ce92952d2709abd992e48ef1f45ee +Author: Lasse Collin +Date: 2023-03-17 08:53:38 +0200 + + Windows: Add microlzma_*.c to the VS project files. + + These should have been included in 5.3.2alpha already. + + windows/vs2013/liblzma.vcxproj | 2 ++ + windows/vs2013/liblzma_dll.vcxproj | 2 ++ + windows/vs2017/liblzma.vcxproj | 2 ++ + windows/vs2017/liblzma_dll.vcxproj | 2 ++ + windows/vs2019/liblzma.vcxproj | 2 ++ + windows/vs2019/liblzma_dll.vcxproj | 2 ++ + 6 files changed, 12 insertions(+) + +commit 75c9ca450fab6982fda9286b168081c9d54126cd +Author: Lasse Collin +Date: 2023-03-17 08:43:51 +0200 + + CMake: Add microlzma_*.c to the build. + + These should have been included in 5.3.2alpha already. + + CMakeLists.txt | 2 ++ + 1 file changed, 2 insertions(+) + +commit 0cc3313bd4e569c51e686e5aab8c40c35241d34b +Author: Lasse Collin +Date: 2023-03-17 08:41:36 +0200 + + Build: Update comments about unaligned access to mention 64-bit. + + cmake/tuklib_integer.cmake | 7 +++---- + m4/tuklib_integer.m4 | 4 ++-- + 2 files changed, 5 insertions(+), 6 deletions(-) + +commit 5e57e3301319f20c35f8111dea73fa58403b96b1 +Author: Lasse Collin +Date: 2023-03-17 00:02:30 +0200 + + Tests: Update .gitignore. + + .gitignore | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 0007394d54e21bf30abb9a5e09cbc1e8d44a73ac +Author: Lasse Collin +Date: 2023-03-14 20:04:03 +0200 + + po4a/update-po: Display the script name consistently in error messages. + + po4a/update-po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 509157c80c500426ec853bd992d684ebafc8500c +Author: Jia Tan +Date: 2023-03-17 01:30:36 +0800 + + Doc: Rename Doxygen HTML doc directory name liblzma => api. + + When the docs are installed, calling the directory "liblzma" is + confusing since multiple other files in the doc directory are for + liblzma. This should also make it more natural for distros when they + package the documentation. + + .gitignore | 2 +- + Makefile.am | 18 +++++++++--------- + PACKAGERS | 4 ++-- + doxygen/Doxyfile | 2 +- + doxygen/update-doxygen | 18 +++++++++--------- + 5 files changed, 22 insertions(+), 22 deletions(-) + +commit fd90e2f4c29180b44e33c7ef726f94e4eae54ed3 +Author: Jia Tan +Date: 2023-03-16 22:07:15 +0800 + + liblzma: Remove note from lzma_options_bcj about the ARM64 exception. + + This was left in by mistake since an early version of the ARM64 filter + used a different struct for its options. + + src/liblzma/api/lzma/bcj.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 4f50763b981f9056c5f1763dfb26cfa4a26a181d +Author: Jia Tan +Date: 2023-03-16 21:44:02 +0800 + + CI: Add doxygen as a dependency. + + Autogen now requires --no-doxygen or having doxygen installed to run + without errors. + + .github/workflows/ci.yml | 5 ++--- + 1 file changed, 2 insertions(+), 3 deletions(-) + +commit f68f4b27f62f53fdac570885a1f4f23367ce6599 +Author: Lasse Collin +Date: 2023-03-15 19:19:13 +0200 + + COPYING: Add a note about the included Doxygen-generated HTML. + + COPYING | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit 8979308528c1f45cb9ee52d511f05232b4ad90a1 +Author: Jia Tan +Date: 2023-03-16 21:41:09 +0800 + + Doc: Update PACKAGERS with details about liblzma API docs install. + + PACKAGERS | 22 ++++++++++++++++------ + 1 file changed, 16 insertions(+), 6 deletions(-) + +commit 55ba6e93004842ae0a0792214a23504267ad8f43 +Author: Jia Tan +Date: 2023-03-16 21:38:32 +0800 + + liblzma: Add set lzma.h as the main page for Doxygen documentation. + + The \mainpage command is used in the first block of comments in lzma.h. + This changes the previously nearly empty index.html to use the first + comment block in lzma.h for its contents. + + lzma.h is no longer documented separately, but this is for the better + since lzma.h only defined a few macros that users do not need to use. + The individual API header files all have a disclaimer that they should + not be #included directly, so there should be no confusion on the fact + that lzma.h should be the only header used by applications. + + Additionally, the note "See ../lzma.h for information about liblzma as + a whole." was removed since lzma.h is now the main page of the + generated HTML and does not have its own page anymore. So it would be + confusing in the HTML version and was only a "nice to have" when + browsing the source files. + + src/liblzma/api/lzma.h | 1 + + src/liblzma/api/lzma/base.h | 2 -- + src/liblzma/api/lzma/bcj.h | 2 -- + src/liblzma/api/lzma/block.h | 2 -- + src/liblzma/api/lzma/check.h | 2 -- + src/liblzma/api/lzma/container.h | 2 -- + src/liblzma/api/lzma/delta.h | 2 -- + src/liblzma/api/lzma/filter.h | 2 -- + src/liblzma/api/lzma/hardware.h | 2 -- + src/liblzma/api/lzma/index.h | 2 -- + src/liblzma/api/lzma/index_hash.h | 4 +--- + src/liblzma/api/lzma/lzma12.h | 2 -- + src/liblzma/api/lzma/stream_flags.h | 2 -- + src/liblzma/api/lzma/version.h | 2 -- + src/liblzma/api/lzma/vli.h | 2 -- + 15 files changed, 2 insertions(+), 29 deletions(-) + +commit 16f21255597f6a57e5692780f962cdc090f62b8c +Author: Jia Tan +Date: 2023-03-16 21:37:32 +0800 + + Build: Generate doxygen documentation in autogen.sh. + + Another command line option (--no-doxygen) was added to disable + creating the doxygen documenation in cases where it not wanted or + if the doxygen tool is not installed. + + autogen.sh | 35 +++++++++++++++++++++++++++++------ + 1 file changed, 29 insertions(+), 6 deletions(-) + +commit 1321852a3be7196bd7fcfd146221a5669e46407c +Author: Jia Tan +Date: 2023-03-16 21:35:55 +0800 + + Build: Create doxygen/update-doxygen script. + + This is a helper script to generate the Doxygen documentation. It can be + run in 'liblzma' or 'internal' mode by setting the first argument. It + will default to 'liblzma' mode and only generate documentation for the + liblzma API header files. + + The helper script will be run during the custom mydist hook when we + create releases. This hook already alters the source directory, so its + fine to do it here too. This way, we can include the Doxygen generated + files in the distrubtion and when installing. + + In 'liblzma' mode, the JavaScript is stripped from the .html files and + the .js files are removed. This avoids license hassle from jQuery and + other libraries that Doxygen 1.9.6 puts into jquery.js in minified form. + + Makefile.am | 1 + + doxygen/update-doxygen | 111 +++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 112 insertions(+) + +commit b1216a7772952d2fe7fe9c6acfcbd98d30abbc7b +Author: Jia Tan +Date: 2023-03-16 21:34:36 +0800 + + Build: Install Doxygen docs and include in distribution if generated. + + Added a install-data-local target to install the Doxygen documentation + only when it has been generated. In order to correctly remove the docs, + a corresponding uninstall-local target was added. + + If the doxygen docs exist in the source tree, they will also be included + in the distribution now too. + + Makefile.am | 18 ++++++++++++++++++ + 1 file changed, 18 insertions(+) + +commit c97d12f300b2a94c9f54a44c8931c8bc08cf0a73 +Author: Lasse Collin +Date: 2023-03-16 21:23:48 +0800 + + Doxygen: Refactor Doxyfile.in to doxygen/Doxyfile. + + Instead of having Doxyfile.in configured by Autoconf, the Doxyfile + can have the tags that need to be configured piped into the doxygen + command through stdin with the overrides after Doxyfile's contents. + + Going forward, the documentation should be generated in two different + modes: liblzma or internal. + + liblzma is useful for most users. It is the documentation for just + the liblzma API header files. This is the default. + + internal is for people who want to understand how xz and liblzma work. + It might be useful for people who want to contribute to the project. + + .gitignore | 3 +- + Makefile.am | 1 - + configure.ac | 40 --- + Doxyfile.in => doxygen/Doxyfile | 721 +++++++++++++++++++++++++--------------- + 4 files changed, 456 insertions(+), 309 deletions(-) + +commit 1b7661faa4bbf4a54c6b75900b5059835c382a0f +Author: Jia Tan +Date: 2023-02-28 23:22:36 +0800 + + Tests: Remove unused macros and functions. + + tests/tests.h | 75 ----------------------------------------------------------- + 1 file changed, 75 deletions(-) + +commit af55191102f01e76de658c881299f0909ca0feda +Author: Jia Tan +Date: 2022-12-29 21:52:15 +0800 + + liblzma: Defines masks for return values from lzma_index_checks(). + + src/liblzma/api/lzma/index.h | 23 +++++++++++++++++++++++ + tests/test_index.c | 22 +++++++++++----------- + 2 files changed, 34 insertions(+), 11 deletions(-) + +commit 8f38cdd9ab71e2a9d5a9787550222b7578243b73 +Author: Jia Tan +Date: 2023-01-12 22:29:07 +0800 + + Tests: Refactors existing lzma_index tests. + + Converts the existing lzma_index tests into tuktests and covers every + API function from index.h except for lzma_file_info_decoder, which can + be tested in the future. + + tests/test_index.c | 2036 ++++++++++++++++++++++++++++++++++++++-------------- + 1 file changed, 1492 insertions(+), 544 deletions(-) + +commit 717aa3651ce582807f379d8654c2516e1594df77 +Author: Lasse Collin +Date: 2023-03-11 18:42:08 +0200 + + xz: Simplify the error-label in Capsicum sandbox code. + + Also remove unneeded "sandbox_allowed = false;" as this code + will never be run more than once (making it work with multiple + input files isn't trivial). + + src/xz/file_io.c | 27 ++++++++++++--------------- + 1 file changed, 12 insertions(+), 15 deletions(-) + +commit a0eecc235d3ba8ad3453da98b46c7bc3e644de75 +Author: Lasse Collin +Date: 2023-03-07 19:59:23 +0200 + + xz: Make Capsicum sandbox more strict with stdin and stdout. + + src/xz/file_io.c | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit 916448d624aaf55cef0fc3e53754affb8c4f309d +Author: Jia Tan +Date: 2023-03-08 23:08:46 +0800 + + Revert: "Add warning if Capsicum sandbox system calls are unsupported." + + The warning causes the exit status to be 2, so this will cause problems + for many scripted use cases for xz. The sandbox usage is already very + limited already, so silently disabling this allows it to be more usable. + + src/xz/file_io.c | 10 ++++------ + 1 file changed, 4 insertions(+), 6 deletions(-) + +commit 01587dda2a8f13fef7e12fd624e6d05da5f9624f +Author: Jia Tan +Date: 2023-03-07 20:02:22 +0800 + + xz: Fix -Wunused-label in io_sandbox_enter(). + + Thanks to Xin Li for recommending the fix. + + src/xz/file_io.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 5fb936786601a1cd013a5d436adde65982b1e13c +Author: Jia Tan +Date: 2023-03-06 21:37:45 +0800 + + xz: Add warning if Capsicum sandbox system calls are unsupported. + + The warning is only used when errno == ENOSYS. Otherwise, xz still + issues a fatal error. + + src/xz/file_io.c | 2 ++ + 1 file changed, 2 insertions(+) + +commit 61ee82cb1232a402c82282bbae42821f2b952b0d +Author: Jia Tan +Date: 2023-03-06 21:27:53 +0800 + + xz: Skip Capsicum sandbox system calls when they are unsupported. + + If a system has the Capsicum header files but does not actually + implement the system calls, then this would render xz unusable. Instead, + we can check if errno == ENOSYS and not issue a fatal error. + + src/xz/file_io.c | 22 +++++++++++++++++----- + 1 file changed, 17 insertions(+), 5 deletions(-) + +commit f070722b57ba975a0dff36492d766f03026b1d21 +Author: Jia Tan +Date: 2023-03-06 21:08:26 +0800 + + xz: Reorder cap_enter() to beginning of capsicum sandbox code. + + cap_enter() puts the process into the sandbox. If later calls to + cap_rights_limit() fail, then the process can still have some extra + protections. + + src/xz/file_io.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit f1ab1f6b339d16a53ac53efeb97779ecd2bae70f +Author: Jia Tan +Date: 2023-02-24 23:46:23 +0800 + + liblzma: Clarify lzma_lzma_preset() documentation in lzma12.h. + + lzma_lzma_preset() does not guarentee that the lzma_options_lzma are + usable in an encoder even if it returns false (success). If liblzma + is built with default configurations, then the options will always be + usable. However if the match finders hc3, hc4, or bt4 are disabled, then + the options may not be usable depending on the preset level requested. + + The documentation was updated to reflect this complexity, since this + behavior was unclear before. + + src/liblzma/api/lzma/lzma12.h | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 4b7fb3bf41a0ca4c97fad3799949a2aa61b13b99 +Author: Lasse Collin +Date: 2023-02-27 18:38:35 +0200 + + CMake: Require that the C compiler supports C99 or a newer standard. + + Thanks to autoantwort for reporting the issue and suggesting + a different patch: + https://github.com/tukaani-project/xz/pull/42 + + CMakeLists.txt | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit 9aa7fdeb04c486d2700967090956af88fdccab7e +Author: Jia Tan +Date: 2023-02-24 18:10:37 +0800 + + Tests: Small tweak to test-vli.c. + + The static global variables can be disabled if encoders and decoders + are not built. If they are not disabled and -Werror is used, it will + cause an usused warning as an error. + + tests/test_vli.c | 2 ++ + 1 file changed, 2 insertions(+) + +commit 3cf72c4bcba5370f07477c9b9b62ae33069ef9a9 +Author: Jia Tan +Date: 2023-02-06 21:46:43 +0800 + + liblzma: Replace '\n' -> newline in filter.h documentation. + + The '\n' renders as a newline when the comments are converted to html + by Doxygen. + + src/liblzma/api/lzma/filter.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 002006be62d77c706565fa6ec828bea64be302da +Author: Jia Tan +Date: 2023-02-06 21:45:37 +0800 + + liblzma: Shorten return description for two functions in filter.h. + + Shorten the description for lzma_raw_encoder_memusage() and + lzma_raw_decoder_memusage(). + + src/liblzma/api/lzma/filter.h | 8 ++------ + 1 file changed, 2 insertions(+), 6 deletions(-) + +commit 463d9359b8595f01d44ada1739d75aeb87f36524 +Author: Jia Tan +Date: 2023-02-06 21:44:45 +0800 + + liblzma: Reword a few lines in filter.h + + src/liblzma/api/lzma/filter.h | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit 01441df92c0fd6a6c02fe5ac27982a54ce887cc0 +Author: Jia Tan +Date: 2023-02-06 21:35:06 +0800 + + liblzma: Improve documentation in filter.h. + + All functions now explicitly specify parameter and return values. + The notes and code annotations were moved before the parameter and + return value descriptions for consistency. + + Also, the description above lzma_filter_encoder_is_supported() about + not being able to list available filters was removed since + lzma_str_list_filters() will do this. + + src/liblzma/api/lzma/filter.h | 226 ++++++++++++++++++++++++++---------------- + 1 file changed, 143 insertions(+), 83 deletions(-) + +commit 805b45cd60bfd5da3d3d89077de3789df179b324 +Author: Lasse Collin +Date: 2023-02-23 20:46:16 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 30e95bb44c36ae26b2ab12a94343b215fec285e7 +Author: Lasse Collin +Date: 2023-02-21 22:57:10 +0200 + + liblzma: Avoid null pointer + 0 (undefined behavior in C). + + In the C99 and C17 standards, section 6.5.6 paragraph 8 means that + adding 0 to a null pointer is undefined behavior. As of writing, + "clang -fsanitize=undefined" (Clang 15) diagnoses this. However, + I'm not aware of any compiler that would take advantage of this + when optimizing (Clang 15 included). It's good to avoid this anyway + since compilers might some day infer that pointer arithmetic implies + that the pointer is not NULL. That is, the following foo() would then + unconditionally return 0, even for foo(NULL, 0): + + void bar(char *a, char *b); + + int foo(char *a, size_t n) + { + bar(a, a + n); + return a == NULL; + } + + In contrast to C, C++ explicitly allows null pointer + 0. So if + the above is compiled as C++ then there is no undefined behavior + in the foo(NULL, 0) call. + + To me it seems that changing the C standard would be the sane + thing to do (just add one sentence) as it would ensure that a huge + amount of old code won't break in the future. Based on web searches + it seems that a large number of codebases (where null pointer + 0 + occurs) are being fixed instead to be future-proof in case compilers + will some day optimize based on it (like making the above foo(NULL, 0) + return 0) which in the worst case will cause security bugs. + + Some projects don't plan to change it. For example, gnulib and thus + many GNU tools currently require that null pointer + 0 is defined: + + https://lists.gnu.org/archive/html/bug-gnulib/2021-11/msg00000.html + + https://www.gnu.org/software/gnulib/manual/html_node/Other-portability-assumptions.html + + In XZ Utils null pointer + 0 issue should be fixed after this + commit. This adds a few if-statements and thus branches to avoid + null pointer + 0. These check for size > 0 instead of ptr != NULL + because this way bugs where size > 0 && ptr == NULL will likely + get caught quickly. None of them are in hot spots so it shouldn't + matter for performance. + + A little less readable version would be replacing + + ptr + offset + + with + + offset != 0 ? ptr + offset : ptr + + or creating a macro for it: + + #define my_ptr_add(ptr, offset) \ + ((offset) != 0 ? ((ptr) + (offset)) : (ptr)) + + Checking for offset != 0 instead of ptr != NULL allows GCC >= 8.1, + Clang >= 7, and Clang-based ICX to optimize it to the very same code + as ptr + offset. That is, it won't create a branch. So for hot code + this could be a good solution to avoid null pointer + 0. Unfortunately + other compilers like ICC 2021 or MSVC 19.33 (VS2022) will create a + branch from my_ptr_add(). + + Thanks to Marcin Kowalczyk for reporting the problem: + https://github.com/tukaani-project/xz/issues/36 + + src/liblzma/common/block_decoder.c | 5 ++++- + src/liblzma/common/block_encoder.c | 7 +++++-- + src/liblzma/common/common.c | 20 ++++++++++++++------ + src/liblzma/common/index_decoder.c | 13 ++++++++++--- + src/liblzma/common/index_encoder.c | 11 +++++++++-- + src/liblzma/common/index_hash.c | 13 ++++++++++--- + src/liblzma/common/lzip_decoder.c | 6 +++++- + src/liblzma/delta/delta_decoder.c | 7 ++++++- + src/liblzma/delta/delta_encoder.c | 12 ++++++++++-- + src/liblzma/simple/simple_coder.c | 6 ++++-- + 10 files changed, 77 insertions(+), 23 deletions(-) + +commit fa9065fac54194fe0407fc7f0cc9633fdce13c21 +Author: Jia Tan +Date: 2023-02-07 00:00:44 +0800 + + liblzma: Adjust container.h for consistency with filter.h. + + src/liblzma/api/lzma/container.h | 20 +++++++++----------- + 1 file changed, 9 insertions(+), 11 deletions(-) + +commit 00a721b63d82dfb658dca8d8cb599d8a245c663f +Author: Jia Tan +Date: 2023-02-07 00:00:09 +0800 + + liblzma: Fix small typos and reword a few things in filter.h. + + src/liblzma/api/lzma/container.h | 13 ++++++------- + 1 file changed, 6 insertions(+), 7 deletions(-) + +commit 5b1c171d4ffe89ef18fa31509bb0185d6fd11d39 +Author: Jia Tan +Date: 2023-02-06 23:42:08 +0800 + + liblzma: Convert list of flags in lzma_mt to bulleted list. + + src/liblzma/api/lzma/container.h | 9 ++++++--- + 1 file changed, 6 insertions(+), 3 deletions(-) + +commit dbd47622eb99fefb3538a22baec3def002aa56f5 +Author: Jia Tan +Date: 2023-01-26 23:17:41 +0800 + + liblzma: Fix typo in documentation in container.h + + lzma_microlzma_decoder -> lzma_microlzma_encoder + + src/liblzma/api/lzma/container.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 14cd30806d69e55906073745bcce3ee50e0ec942 +Author: Jia Tan +Date: 2023-01-26 23:16:34 +0800 + + liblzma: Improve documentation for container.h + + Standardizing each function to always specify parameters and return + values. Also moved the parameters and return values to the end of each + function description. + + src/liblzma/api/lzma/container.h | 146 +++++++++++++++++++++++++-------------- + 1 file changed, 93 insertions(+), 53 deletions(-) + +commit c9c8bfae3502842dcead85eeb2b951b437c2cd88 +Author: Jia Tan +Date: 2023-02-22 20:59:41 +0800 + + CMake: Add LZIP decoder test to list of tests. + + CMakeLists.txt | 1 + + 1 file changed, 1 insertion(+) + +commit b9f171dd00a3cc32b6d41ea8e082cf545640ec2a +Author: Lasse Collin +Date: 2023-02-17 20:56:49 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 2ee86d20e49985b903b78ebcfa3fa672e73e93aa +Author: Lasse Collin +Date: 2023-02-17 20:48:28 +0200 + + Build: Use only the generic symbol versioning on MicroBlaze. + + On MicroBlaze, GCC 12 is broken in sense that + __has_attribute(__symver__) returns true but it still doesn't + support the __symver__ attribute even though the platform is ELF + and symbol versioning is supported if using the traditional + __asm__(".symver ...") method. Avoiding the traditional method is + good because it breaks LTO (-flto) builds with GCC. + + See also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101766 + + For now the only extra symbols in liblzma_linux.map are the + compatibility symbols with the patch that spread from RHEL/CentOS 7. + These require the use of __symver__ attribute or __asm__(".symver ...") + in the C code. Compatibility with the patch from CentOS 7 doesn't + seem valuable on MicroBlaze so use liblzma_generic.map on MicroBlaze + instead. It doesn't require anything special in the C code and thus + no LTO issues either. + + An alternative would be to detect support for __symver__ + attribute in configure.ac and CMakeLists.txt and fall back + to __asm__(".symver ...") but then LTO would be silently broken + on MicroBlaze. It sounds likely that MicroBlaze is a special + case so let's treat it as a such because that is simpler. If + a similar issue exists on some other platform too then hopefully + someone will report it and this can be reconsidered. + + (This doesn't do the same fix in CMakeLists.txt. Perhaps it should + but perhaps CMake build of liblzma doesn't matter much on MicroBlaze. + The problem breaks the build so it's easy to notice and can be fixed + later.) + + Thanks to Vincent Fazio for reporting the problem and proposing + a patch (in the end that solution wasn't used): + https://github.com/tukaani-project/xz/pull/32 + + configure.ac | 12 ++++++++++-- + 1 file changed, 10 insertions(+), 2 deletions(-) + +commit d831072cceca458d94d2d5da201862f6d43a417b +Author: Lasse Collin +Date: 2023-02-16 21:09:00 +0200 + + liblzma: Very minor API doc tweaks. + + Use "member" to refer to struct members as that's the term used + by the C standard. + + Use lzma_options_delta.dist and such in docs so that in Doxygen's + HTML output they will link to the doc of the struct member. + + Clean up a few trailing white spaces too. + + src/liblzma/api/lzma/block.h | 6 +++--- + src/liblzma/api/lzma/delta.h | 6 +++--- + src/liblzma/api/lzma/index.h | 10 +++++----- + src/liblzma/api/lzma/stream_flags.h | 6 +++--- + 4 files changed, 14 insertions(+), 14 deletions(-) + +commit f029daea39c215fd7d5cb6b6798818b055cf5b22 +Author: Jia Tan +Date: 2023-02-17 00:54:33 +0800 + + liblzma: Adjust spacing in doc headers in bcj.h. + + src/liblzma/api/lzma/bcj.h | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +commit a5de68bac2bb7e1b9119e6cea7d761a22ea73e9c +Author: Jia Tan +Date: 2023-02-17 00:44:44 +0800 + + liblzma: Adjust documentation in bcj.h for consistent style. + + src/liblzma/api/lzma/bcj.h | 43 ++++++++++++++++++++++--------------------- + 1 file changed, 22 insertions(+), 21 deletions(-) + +commit efa498c13b883810497e0ea8a169efd6f48f5026 +Author: Jia Tan +Date: 2023-02-17 00:36:05 +0800 + + liblzma: Rename field => member in documentation. + + Also adjusted preset value => preset level. + + src/liblzma/api/lzma/base.h | 18 +++++++-------- + src/liblzma/api/lzma/block.h | 44 ++++++++++++++++++------------------- + src/liblzma/api/lzma/container.h | 26 +++++++++++----------- + src/liblzma/api/lzma/delta.h | 12 +++++----- + src/liblzma/api/lzma/index.h | 30 ++++++++++++------------- + src/liblzma/api/lzma/lzma12.h | 28 +++++++++++------------ + src/liblzma/api/lzma/stream_flags.h | 32 +++++++++++++-------------- + 7 files changed, 95 insertions(+), 95 deletions(-) + +commit 718b22a6c5e3ee5de123323ea798872381f9320e +Author: Lasse Collin +Date: 2023-02-16 17:59:50 +0200 + + liblzma: Silence a warning from MSVC. + + It gives C4146 here since unary minus with unsigned integer + is still unsigned (which is the intention here). Doing it + with substraction makes it clearer and avoids the warning. + + Thanks to Nathan Moinvaziri for reporting this. + + src/liblzma/check/crc64_fast.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 87c53553fa7d50f777b4edfa99f2083628f590fe +Author: Jia Tan +Date: 2023-02-16 21:04:54 +0800 + + liblzma: Improve documentation for stream_flags.h + + Standardizing each function to always specify parameters and return + values. Also moved the parameters and return values to the end of each + function description. + + A few small things were reworded and long sentences broken up. + + src/liblzma/api/lzma/stream_flags.h | 76 ++++++++++++++++++++++--------------- + 1 file changed, 46 insertions(+), 30 deletions(-) + +commit 13d99e75a543e9e5f8633cc241eae55b91a3b242 +Author: Jia Tan +Date: 2023-02-14 21:50:16 +0800 + + liblzma: Improve documentation in lzma12.h. + + All functions now explicitly specify parameter and return values. + + src/liblzma/api/lzma/lzma12.h | 32 +++++++++++++++++++++++--------- + 1 file changed, 23 insertions(+), 9 deletions(-) + +commit 43ec344c868f930e96879eb9e49212cce92a9884 +Author: Jia Tan +Date: 2023-01-27 22:44:06 +0800 + + liblzma: Improve documentation in check.h. + + All functions now explicitly specify parameter and return values. + Also moved the note about SHA-256 functions not being exported to the + top of the file. + + src/liblzma/api/lzma/check.h | 41 ++++++++++++++++++++++++++++------------- + 1 file changed, 28 insertions(+), 13 deletions(-) + +commit 9c71db4e884fd49aea3d1e711036bff45ca66487 +Author: Jia Tan +Date: 2023-02-08 21:33:52 +0800 + + liblzma: Improve documentation in index.h + + All functions now explicitly specify parameter and return values. + + src/liblzma/api/lzma/index.h | 177 ++++++++++++++++++++++++++++++------------- + 1 file changed, 126 insertions(+), 51 deletions(-) + +commit 421f2f2e160720f6009e3b6a125cafe2feaa9419 +Author: Jia Tan +Date: 2023-02-08 20:35:32 +0800 + + liblzma: Reword a comment in index.h. + + src/liblzma/api/lzma/index.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit b67539484981351d501b68de5e925425e50c59b1 +Author: Jia Tan +Date: 2023-02-08 20:30:23 +0800 + + liblzma: Omit lzma_index_iter's internal field from Doxygen docs. + + Add \private above this field and its sub-fields since it is not meant + to be modified by users. + + src/liblzma/api/lzma/index.h | 9 ++++++++- + 1 file changed, 8 insertions(+), 1 deletion(-) + +commit 0c9e4fc2ad6d88d54f299240fcc5a2ce7d695d96 +Author: Jia Tan +Date: 2023-01-21 21:32:03 +0800 + + liblzma: Fix documentation for LZMA_MEMLIMIT_ERROR. + + LZMA_MEMLIMIT_ERROR was missing the "<" character needed to put + documentation after a member. + + src/liblzma/api/lzma/base.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 816fec125aa74bcef46512c73acc6d9e5a700d15 +Author: Jia Tan +Date: 2023-01-21 00:29:38 +0800 + + liblzma: Improve documentation for base.h. + + Standardizing each function to always specify params and return values. + Also fixed a small grammar mistake. + + src/liblzma/api/lzma/base.h | 30 +++++++++++++++++++++++++----- + 1 file changed, 25 insertions(+), 5 deletions(-) + +commit 862dacef1a4e7e1b28d465956fa4244ed01df154 +Author: Jia Tan +Date: 2023-02-14 00:12:34 +0800 + + liblzma: Add one more missing [out] annotation in vli.h + + src/liblzma/api/lzma/vli.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 867b08ae4254bf55dd1f7fd502cc618231b92f75 +Author: Jia Tan +Date: 2023-02-14 00:08:33 +0800 + + liblzma: Minor improvements to vli.h. + + Added [out] annotations to parameters that are pointers and can have + their value changed. Also added a clarification to lzma_vli_is_valid. + + src/liblzma/api/lzma/vli.h | 13 +++++++------ + 1 file changed, 7 insertions(+), 6 deletions(-) + +commit 90d0e628ff11e5030bcc4fc000bca056adda6603 +Author: Jia Tan +Date: 2023-02-10 21:38:02 +0800 + + liblzma: Add comments for macros in delta.h. + + Document LZMA_DELTA_DIST_MIN and LZMA_DELTA_DIST_MAX for completeness + and to avoid Doxygen warnings. + + src/liblzma/api/lzma/delta.h | 8 ++++++++ + 1 file changed, 8 insertions(+) + +commit 9255fffdb13e59874bf7f95c370c410ad3a7e114 +Author: Jia Tan +Date: 2023-02-10 21:35:23 +0800 + + liblzma: Improve documentation in index_hash.h. + + All functions now explicitly specify parameter and return values. + Also reworded the description of lzma_index_hash_init() for readability. + + src/liblzma/api/lzma/index_hash.h | 36 +++++++++++++++++++++++++++--------- + 1 file changed, 27 insertions(+), 9 deletions(-) + +commit 1dbe12b90cff79bb51923733ac0840747b4b4131 +Author: Lasse Collin +Date: 2023-02-07 19:07:45 +0200 + + xz: Improve the comment about start_time in mytime.c. + + start_time is relative to an arbitary point in time, it's not + time of day, so using it for anything else than time differences + wouldn't make sense. + + src/xz/mytime.c | 15 ++++++++++----- + 1 file changed, 10 insertions(+), 5 deletions(-) + +commit 7673ef5aa80c1af7fb693360dd82f527b46c2c56 +Author: Jia Tan +Date: 2023-02-04 21:06:35 +0800 + + Build: Adjust CMake version search regex. + + Now, the LZMA_VERSION_MAJOR, LZMA_VERSION_MINOR, and LZMA_VERSION_PATCH + macros do not need to be on consecutive lines in version.h. They can be + separated by more whitespace, comments, or even other content, as long + as they appear in the proper order (major, minor, patch). + + CMakeLists.txt | 2 ++ + 1 file changed, 2 insertions(+) + +commit b8bce89be7fb5bffe5fef4a2782ca9b2b107eaac +Author: Jia Tan +Date: 2023-02-04 12:01:23 +0800 + + xz: Add a comment clarifying the use of start_time in mytime.c. + + src/xz/mytime.c | 5 +++++ + 1 file changed, 5 insertions(+) + +commit 912af91b10a18fb9bb3167247ecaaefca8248ee9 +Author: Jia Tan +Date: 2023-01-26 09:50:21 +0800 + + liblzma: Improve documentation for version.h. + + Specified parameter and return values for API functions and documented + a few more of the macros. + + src/liblzma/api/lzma/version.h | 29 ++++++++++++++++++++++------- + 1 file changed, 22 insertions(+), 7 deletions(-) + +commit 850adec171203cd22b57d016084d713f72ae5307 +Author: Jia Tan +Date: 2023-02-03 22:52:55 +0800 + + Docs: Omit SIGTSTP not handled from TODO. + + TODO | 4 ---- + 1 file changed, 4 deletions(-) + +commit 2c78a83c6faec70154d9eb78022a618ed62cdcb3 +Author: Jia Tan +Date: 2023-02-03 00:33:32 +0800 + + liblzma: Fix bug in lzma_str_from_filters() not checking filters[] length. + + The bug is only a problem in applications that do not properly terminate + the filters[] array with LZMA_VLI_UNKNOWN or have more than + LZMA_FILTERS_MAX filters. This bug does not affect xz. + + src/liblzma/common/string_conversion.c | 7 +++++++ + 1 file changed, 7 insertions(+) + +commit e01f01b9af1c074463b92694a16ecc16a31907c0 +Author: Jia Tan +Date: 2023-02-03 00:32:47 +0800 + + Tests: Create test_filter_str.c. + + Tests lzma_str_to_filters(), lzma_str_from_filters(), and + lzma_str_list_filters() API functions. + + CMakeLists.txt | 1 + + tests/Makefile.am | 2 + + tests/test_filter_str.c | 593 ++++++++++++++++++++++++++++++++++++++++++++++++ + 3 files changed, 596 insertions(+) + +commit 8dfc029e7a4ce45809c30313dc0e502f0d22be26 +Author: Jia Tan +Date: 2023-01-22 08:49:00 +0800 + + liblzma: Fix typos in comments in string_conversion.c. + + src/liblzma/common/string_conversion.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 54ad83c1ae2180dcc0cb2445b181dc1e9732a5d6 +Author: Jia Tan +Date: 2023-02-03 00:20:20 +0800 + + liblzma: Clarify block encoder and decoder documentation. + + Added a few sentences to the description for lzma_block_encoder() and + lzma_block_decoder() to highlight that the Block Header must be coded + before calling these functions. + + src/liblzma/api/lzma/block.h | 15 +++++++++++---- + 1 file changed, 11 insertions(+), 4 deletions(-) + +commit f680e771b3eb2a46310fe85b3e000ac3a1a0640f +Author: Jia Tan +Date: 2023-02-03 00:12:24 +0800 + + Update lzma_block documentation for lzma_block_uncomp_encode(). + + src/liblzma/api/lzma/block.h | 3 +++ + 1 file changed, 3 insertions(+) + +commit 504cf4af895fd45aad0c56eb3b49d90acd54465b +Author: Jia Tan +Date: 2023-02-03 00:11:37 +0800 + + liblzma: Minor edits to lzma_block header_size documentation. + + src/liblzma/api/lzma/block.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 115b720fb521f99aa832d06b2c12b7f8c6c50680 +Author: Jia Tan +Date: 2023-02-03 00:11:07 +0800 + + liblzma: Enumerate functions that read version in lzma_block. + + src/liblzma/api/lzma/block.h | 13 +++++++++++-- + 1 file changed, 11 insertions(+), 2 deletions(-) + +commit 85ea0979adcf808a3830aefbe7a4ec884e542ea1 +Author: Jia Tan +Date: 2023-02-03 00:10:34 +0800 + + liblzma: Clarify comment in block.h. + + src/liblzma/api/lzma/block.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 1f7ab90d9ce224230a04de6b921ad6e2029023a8 +Author: Jia Tan +Date: 2023-02-03 00:07:23 +0800 + + liblzma: Improve documentation for block.h. + + Standardizing each function to always specify params and return values. + Output pointer parameters are also marked with doxygen style [out] to + make it clear. Any note sections were also moved above the parameter and + return sections for consistency. + + src/liblzma/api/lzma/block.h | 96 ++++++++++++++++++++++++++++++++++---------- + 1 file changed, 75 insertions(+), 21 deletions(-) + +commit c563a4bc554a96bd0b6aab3c139715b7ec8f6ca3 +Author: Jia Tan +Date: 2023-02-01 23:38:30 +0800 + + liblzma: Clarify a comment about LZMA_STR_NO_VALIDATION. + + The flag description for LZMA_STR_NO_VALIDATION was previously confusing + about the treatment for filters than cannot be used with .xz format + (lzma1) without using LZMA_STR_ALL_FILTERS. Now, it is clear that + LZMA_STR_NO_VALIDATION is not a super set of LZMA_STR_ALL_FILTERS. + + src/liblzma/api/lzma/filter.h | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit 315c64c7e18acc59a745b68148188a73e998252b +Author: Jia Tan +Date: 2023-02-01 21:43:33 +0800 + + CI: Update .gitignore for artifacts directory in build-aux. + + The workflow action for our CI pipeline can only reference artifacts in + the source directory, so we should ignore these files if the ci_build.sh + is run locally. + + .gitignore | 1 + + 1 file changed, 1 insertion(+) + +commit 2c1341f4fa06e7f487d61142aa354c433e17ec7f +Author: Jia Tan +Date: 2023-02-01 21:36:46 +0800 + + CI: Add quotes around variables in a few places. + + build-aux/ci_build.sh | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 3a401b0e0c7a2658af7801dd0690256ef24149e0 +Author: Jia Tan +Date: 2023-02-01 21:36:22 +0800 + + CI: Upload test logs as artifacts if a test fails. + + .github/workflows/ci.yml | 60 ++++++++++++++++++++++++++++++++++-------------- + build-aux/ci_build.sh | 31 ++++++++++++++++++++----- + 2 files changed, 68 insertions(+), 23 deletions(-) + +commit 610dde15a88f12cc540424eb3eb3ed61f3876f74 +Author: Lasse Collin +Date: 2023-01-27 20:02:49 +0200 + + xz: Use clock_gettime() even if CLOCK_MONOTONIC isn't available. + + mythread.h and thus liblzma already does it. + + src/xz/mytime.c | 11 ++++++++--- + src/xz/private.h | 3 +-- + 2 files changed, 9 insertions(+), 5 deletions(-) + +commit 2e02877288f6576cd4595e9ac7684f867cd47d68 +Author: Lasse Collin +Date: 2023-01-27 19:41:19 +0200 + + po4a/po4a.conf: Sort the language identifiers in alphabetical order. + + po4a/po4a.conf | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ff592c616eda274215b485cf1b8d34f060c9f3be +Author: Lasse Collin +Date: 2023-01-26 18:29:17 +0200 + + xz: Add SIGTSTP handler for progress indicator time keeping. + + This way, if xz is stopped the elapsed time and estimated time + remaining won't get confused by the amount of time spent in + the stopped state. + + This raises SIGSTOP. It's not clear to me if this is the correct way. + POSIX and glibc docs say that SIGTSTP shouldn't stop the process if + it is orphaned but this commit doesn't attempt to handle that. + + Search for SIGTSTP in section 2.4.3: + + https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html + + src/xz/mytime.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- + src/xz/mytime.h | 6 ++++++ + src/xz/private.h | 12 ++++++++++++ + src/xz/signals.c | 17 ++++++++++++++++- + 4 files changed, 89 insertions(+), 2 deletions(-) + +commit 3b1c8ac8d1d553cbb1fb22b545d2b1424c752b76 +Author: Jia Tan +Date: 2023-01-27 20:14:51 +0800 + + Translations: Add Brazilian Portuguese translation of man pages. + + Thanks to Rafael Fontenelle. + + po4a/po4a.conf | 2 +- + po4a/pt_BR.po | 3677 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 3678 insertions(+), 1 deletion(-) + +commit a15a7552f9f67c4e402f5d2967324e0ccfd6fccc +Author: Lasse Collin +Date: 2023-01-26 17:51:06 +0200 + + Build: Avoid different quoting style in --enable-doxygen doc. + + configure.ac | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit af5a4bd5afc089d9697756dded38feafaa987ae4 +Author: Lasse Collin +Date: 2023-01-26 17:39:46 +0200 + + tuklib_physmem: Check for __has_warning before GCC version. + + Clang can be configured to fake a too high GCC version so + this way it's more robust. + + src/common/tuklib_physmem.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit f35d98e20609e0be6a04ae2604bfb7cb9d5bd5e4 +Author: Jia Tan +Date: 2023-01-24 20:48:50 +0800 + + liblzma: Fix documentation in filter.h for lzma_str_to_filters() + + The previous documentation for lzma_str_to_filters() was technically + correct, but misleading. lzma_str_to_filters() returns NULL on success, + which is in practice always defined to 0. This is the same value as + LZMA_OK, but lzma_str_to_filters() does not return lzma_ret so we should + be more clear. + + src/liblzma/api/lzma/filter.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2f78ecc5939b3d97ddfc2a6bd31b50108a28d0a2 +Author: Lasse Collin +Date: 2023-01-23 23:44:58 +0200 + + Revert "tuklib_common: Define __has_warning if it is not defined." + + This reverts commit 82e3c968bfa10e3ff13333bd9cbbadb5988d6766. + + Macros in the reserved namespace (_foo or __foo) shouldn't be #defined + without a very good reason. Here the alternative would have been + to #define tuklib_has_warning(str) to an approriate value. + + Also the tuklib_* files should stay namespace clean if possible. + + src/common/tuklib_common.h | 7 ------- + 1 file changed, 7 deletions(-) + +commit 8366cf8738e8b7bb74c967d07bf0fd2a1878e575 +Author: Lasse Collin +Date: 2023-01-23 23:38:34 +0200 + + tuklib_physmem: Clean up the way -Wcast-function-type is silenced on Windows. + + __has_warning and other __has_foo macros are meant to become + compiler-agnostic so it's not good to check for __clang__ with it. + + This also relied on tuklib_common.h for #defining __has_warning + which was confusing as #defining reserved macros is generally + not a good idea. + + src/common/tuklib_physmem.c | 17 +++++++++++++---- + 1 file changed, 13 insertions(+), 4 deletions(-) + +commit 683a3c7e2fcd922200c31078e5c9dd1348e90941 +Author: Lasse Collin +Date: 2023-01-24 00:05:38 +0200 + + xz: Flip the return value of suffix_is_set to match the documentation. + + Also edit style to match the existing coding style in the project. + + src/xz/args.c | 6 +++--- + src/xz/suffix.c | 2 +- + src/xz/suffix.h | 1 + + 3 files changed, 5 insertions(+), 4 deletions(-) + +commit cc5aa9ab138beeecaee5a1e81197591893ee9ca0 +Author: Jia Tan +Date: 2023-01-07 21:55:06 +0800 + + xz: Refactor duplicated check for custom suffix when using --format=raw + + src/xz/args.c | 8 ++++++++ + src/xz/suffix.c | 26 ++++++++------------------ + src/xz/suffix.h | 7 +++++++ + 3 files changed, 23 insertions(+), 18 deletions(-) + +commit 9663141274e01592a281a7f2df5d7a31a1dac8bf +Author: Jia Tan +Date: 2023-01-20 21:53:14 +0800 + + liblzma: Set documentation on all reserved fields to private. + + This prevents the reserved fields from being part of the generated + Doxygen documentation. + + src/liblzma/api/lzma/base.h | 17 +++++++++++++++ + src/liblzma/api/lzma/block.h | 43 +++++++++++++++++++++++++++++++++++++ + src/liblzma/api/lzma/container.h | 24 +++++++++++++++++++++ + src/liblzma/api/lzma/delta.h | 12 +++++++++++ + src/liblzma/api/lzma/index.h | 27 +++++++++++++++++++++++ + src/liblzma/api/lzma/lzma12.h | 22 +++++++++++++++++++ + src/liblzma/api/lzma/stream_flags.h | 28 ++++++++++++++++++++++++ + 7 files changed, 173 insertions(+) + +commit 6327a045f34d48fc5afc58ba0d32a82c94403049 +Author: Jia Tan +Date: 2022-12-20 21:39:59 +0800 + + Doxygen: Update Doxyfile.in from 1.4.7 to 1.8.17. + + A few Doxygen tags were obsolete from 1.4.7. Version 1.8.17 released + in 2019, so this should be compatible with resonable modern distros. + The purpose of Doxygen these days is for docs on the website, so it + doesn't necessarily have to work for everyone. Just when the maintainers + want to update the docs. + + Doxyfile.in | 2523 ++++++++++++++++++++++++++++++++++++++++++++--------------- + 1 file changed, 1893 insertions(+), 630 deletions(-) + +commit bbf71b69ebf9d0d62a0af150a5c37d193b8159ad +Author: Jia Tan +Date: 2023-01-03 20:37:30 +0800 + + Doxygen: Make Doxygen only produce liblzma API documentation by default. + + Doxygen is now configurable in autotools only with + --enable-doxygen=[api|all]. The default is "api", which will only + generate HTML output for liblzma API functions. The LaTex documentation + output was also disabled. + + Doxyfile.in | 18 +++++++++--------- + configure.ac | 39 +++++++++++++++++++++++++++++++++++++++ + 2 files changed, 48 insertions(+), 9 deletions(-) + +commit 6fcf4671b6047113c583a0919fc850987a4ec5f4 +Author: Jia Tan +Date: 2022-12-21 23:59:43 +0800 + + liblzma: Highlight liblzma API headers should not be included directly. + + This improves the generated Doxygen HTML files to better highlight + how to properly use the liblzma API header files. + + src/liblzma/api/lzma/base.h | 5 +++-- + src/liblzma/api/lzma/bcj.h | 5 +++-- + src/liblzma/api/lzma/block.h | 5 +++-- + src/liblzma/api/lzma/check.h | 5 +++-- + src/liblzma/api/lzma/container.h | 5 +++-- + src/liblzma/api/lzma/delta.h | 5 +++-- + src/liblzma/api/lzma/filter.h | 5 +++-- + src/liblzma/api/lzma/hardware.h | 5 +++-- + src/liblzma/api/lzma/index.h | 5 +++-- + src/liblzma/api/lzma/index_hash.h | 5 +++-- + src/liblzma/api/lzma/lzma12.h | 5 +++-- + src/liblzma/api/lzma/stream_flags.h | 5 +++-- + src/liblzma/api/lzma/version.h | 5 +++-- + src/liblzma/api/lzma/vli.h | 5 +++-- + 14 files changed, 42 insertions(+), 28 deletions(-) + +commit b43ff180fb2e372adce876bfa155fc9bcf0c3db4 +Author: Jia Tan +Date: 2023-01-19 20:35:09 +0800 + + tuklib_physmem: Silence warning from -Wcast-function-type on MinGW-w64. + + tuklib_physmem depends on GetProcAddress() for both MSVC and MinGW-w64 + to retrieve a function address. The proper way to do this is to cast the + return value to the type of function pointer retrieved. Unfortunately, + this causes a cast-function-type warning, so the best solution is to + simply ignore the warning. + + src/common/tuklib_physmem.c | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit 82e3c968bfa10e3ff13333bd9cbbadb5988d6766 +Author: Jia Tan +Date: 2023-01-19 20:32:40 +0800 + + tuklib_common: Define __has_warning if it is not defined. + + clang supports the __has_warning macro to determine if the version of + clang compiling the code supports a given warning. If we do not define + it for other compilers, it may cause a preprocessor error. + + src/common/tuklib_common.h | 7 +++++++ + 1 file changed, 7 insertions(+) + +commit b2ba1a489df451cdcd93b2334e319dd06778de19 +Author: Jia Tan +Date: 2023-01-18 22:11:05 +0800 + + CI: Reorder 32-bit build first for Linux autotool builds. + + The 32-bit build needs to be first so the configure cache only needs to + be reset one time. The 32-bit build sets the CFLAGS env variable, so any + build using that flag after will fail unless the cache is reset. + + .github/workflows/ci.yml | 17 ++++++++++++----- + 1 file changed, 12 insertions(+), 5 deletions(-) + +commit dd1c1135741057c91e8d018be9ec4d43968b0e64 +Author: Jia Tan +Date: 2023-01-18 21:51:43 +0800 + + CI: Enable --config-cache in autotool builds. + + If CFLAGS are set in a build, the cache must be cleared with + "make distclean", or by deleting the cache file. + + build-aux/ci_build.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit d3e11477053764c003eec2daa5198c747d70ff69 +Author: Jia Tan +Date: 2023-01-16 21:35:45 +0800 + + xz: Add missing comment for coder_set_compression_settings() + + src/xz/coder.h | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 123255b6ed15f4428b2aa92e4962015a5362f6bf +Author: Jia Tan +Date: 2023-01-16 20:55:10 +0800 + + xz: Do not set compression settings with raw format in list mode. + + Calling coder_set_compression_settings() in list mode with verbose mode + on caused the filter chain and memory requirements to print. This was + unnecessary since the command results in an error and not consistent + with other formats like lzma and alone. + + src/xz/args.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 571919c47b9ff5171ede84378620ed0a9aeb98c0 +Author: Jia Tan +Date: 2023-01-13 20:37:06 +0800 + + Translations: Update the Brazilian Portuguese translation. + + po/pt_BR.po | 603 ++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 344 insertions(+), 259 deletions(-) + +commit 81cb02e2c22bbc036cdfaa2d2c4176f6bd60d3cf +Author: Jia Tan +Date: 2023-01-12 23:43:06 +0800 + + CI: Disable shared and nls from various jobs in autotool runners. + + Disabling shared library generation and linking should help speed up the + runners. The shared library is still being tested in the 32 bit build + and the full feature. + + Disabling nls is to check for any unexpected warnings or errors. + + .github/workflows/ci.yml | 56 ++++++++++++++++++++++++------------------------ + 1 file changed, 28 insertions(+), 28 deletions(-) + +commit 58a052198a7bcaf6e958f87fad72e69e19a2579b +Author: Jia Tan +Date: 2023-01-12 23:39:19 +0800 + + CI: Reorder the 32-bit job in the Ubuntu runner. + + Run the 32 bit job sooner since this is a more interesting test than + some of the later jobs. + + .github/workflows/ci.yml | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit 4110a998b83459fe2bc9bc1bec30ad68afa8f797 +Author: Jia Tan +Date: 2023-01-12 23:09:03 +0800 + + CI: Allow disabling Native Language Support. + + build-aux/ci_build.sh | 9 ++++++++- + 1 file changed, 8 insertions(+), 1 deletion(-) + +commit 0dec634e705b5bf89a37c5d62d71e8511d480058 +Author: Jia Tan +Date: 2023-01-12 23:02:20 +0800 + + CI: Only run autogen.sh if it has not already run. + + build-aux/ci_build.sh | 11 ++++++++--- + 1 file changed, 8 insertions(+), 3 deletions(-) + +commit 32287dc8def94df4546e903495d14c132bd54cc4 +Author: Jia Tan +Date: 2023-01-12 22:58:36 +0800 + + CI: Allow disabling shared library in autotools builds. + + build-aux/ci_build.sh | 9 ++++++++- + 1 file changed, 8 insertions(+), 1 deletion(-) + +commit 77d1ebcc99ddd82a300d1838f608150221931dcd +Author: Jia Tan +Date: 2023-01-12 22:44:18 +0800 + + CI: Improve Usage readability and add -h option. + + build-aux/ci_build.sh | 15 +++++++++++++-- + 1 file changed, 13 insertions(+), 2 deletions(-) + +commit a8bb8358d10b059274f3cf993d9b8f490bafb268 +Author: Lasse Collin +Date: 2023-01-12 13:04:05 +0200 + + Build: Omit -Wmissing-noreturn from the default warnings. + + It's not that important. It can be annoying in builds that + disable many features since in those cases the tests programs + will correctly trigger this warning with Clang. + + configure.ac | 1 - + 1 file changed, 1 deletion(-) + +commit 52dc033d0bde0d19e3912303c6c74bae559d6498 +Author: Lasse Collin +Date: 2023-01-12 06:05:58 +0200 + + xz: Use ssize_t for the to-be-ignored return value from write(fd, ptr, 1). + + It makes no difference here as the return value fits into an int + too and it then gets ignored but this looks better. + + src/xz/file_io.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit b1a6d180a363d57b2b1c89526ff3f0782bf863d3 +Author: Lasse Collin +Date: 2023-01-12 06:01:12 +0200 + + xz: Silence warnings from -Wsign-conversion in a 32-bit build. + + src/common/tuklib_mbstr_fw.c | 2 +- + src/xz/list.c | 4 ++-- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 31c21c734b7c7d7428a3da7402a2cb7bc2587339 +Author: Lasse Collin +Date: 2023-01-12 05:38:48 +0200 + + liblzma: Silence another warning from -Wsign-conversion in a 32-bit build. + + It doesn't warn on a 64-bit system because truncating + a ptrdiff_t (signed long) to uint32_t is diagnosed under + -Wconversion by GCC and -Wshorten-64-to-32 by Clang. + + src/liblzma/lz/lz_encoder_mf.c | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +commit 37fbdfb7263522c11c7ad2685413d6295532581d +Author: Lasse Collin +Date: 2023-01-12 04:46:45 +0200 + + liblzma: Silence a warning from -Wsign-conversion in a 32-bit build. + + src/common/mythread.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 5ce6ddc221d0bfb57d810d845bb65fb0aac0b008 +Author: Lasse Collin +Date: 2023-01-12 04:17:24 +0200 + + Build: Make configure add more warning flags for GCC and Clang. + + -Wstrict-aliasing was removed from the list since it is enabled + by -Wall already. + + A normal build is clean with these on GNU/Linux x86-64 with + GCC 12.2.0 and Clang 14.0.6. + + configure.ac | 36 +++++++++++++++++++++++++++++++----- + 1 file changed, 31 insertions(+), 5 deletions(-) + +commit bfc3a0a8ac16de90049c1b1ba1445a7626d0230c +Author: Lasse Collin +Date: 2023-01-12 04:14:18 +0200 + + Tests: Fix warnings from clang --Wassign-enum. + + Explicitly casting the integer to lzma_check silences the warning. + Since such an invalid value is needed in multiple tests, a constant + INVALID_LZMA_CHECK_ID was added to tests.h. + + The use of 0x1000 for lzma_block.check wasn't optimal as if + the underlying type is a char then 0x1000 will be truncated to 0. + However, in these test cases the value is ignored, thus even with + such truncation the test would have passed. + + tests/test_block_header.c | 6 +++--- + tests/test_check.c | 2 +- + tests/test_stream_flags.c | 8 ++++---- + tests/tests.h | 9 +++++++++ + 4 files changed, 17 insertions(+), 8 deletions(-) + +commit 49245bb31e215ad455a1ab85e4ed6783152dc522 +Author: Lasse Collin +Date: 2023-01-12 03:51:07 +0200 + + Tests: Silence warnings from -Wsign-conversion. + + Note that assigning an unsigned int to lzma_check doesn't warn + on GNU/Linux x86-64 since the enum type is unsigned on that + platform. The enum can be signed on some other platform though + so it's best to use enumeration type lzma_check in these situations. + + tests/test_check.c | 6 +++--- + tests/test_stream_flags.c | 10 +++++----- + 2 files changed, 8 insertions(+), 8 deletions(-) + +commit 3f13bf6b9e8624cbe6d6e3e82d6c98a3ed1ad571 +Author: Lasse Collin +Date: 2023-01-12 03:19:59 +0200 + + liblzma: Silence warnings from clang -Wconditional-uninitialized. + + This is similar to 2ce4f36f179a81d0c6e182a409f363df759d1ad0. + The actual initialization of the variables is done inside + mythread_sync() macro. Clang doesn't seem to see that + the initialization code inside the macro is always executed. + + src/liblzma/common/stream_decoder_mt.c | 8 +++++--- + src/liblzma/common/stream_encoder_mt.c | 2 +- + 2 files changed, 6 insertions(+), 4 deletions(-) + +commit 6c886cc5b3c90c6a75e6be8b1278ec2261e452a6 +Author: Lasse Collin +Date: 2023-01-12 03:11:40 +0200 + + Fix warnings from clang -Wdocumentation. + + src/liblzma/check/check.h | 4 ---- + src/liblzma/lz/lz_encoder_mf.c | 4 ++-- + src/xz/options.c | 4 ++-- + 3 files changed, 4 insertions(+), 8 deletions(-) + +commit a0e7fb1c1ea658b67f30517f5d1975efd0226dba +Author: Lasse Collin +Date: 2023-01-12 03:04:28 +0200 + + Tests: test_lzip_decoder: Remove trailing white-space. + + tests/test_lzip_decoder.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit c0f8d6782f29e219fd496dd23f6a033270509d5c +Author: Lasse Collin +Date: 2023-01-12 03:03:55 +0200 + + Tests: test_lzip_decoder: Silence warnings from -Wsign-conversion. + + tests/test_lzip_decoder.c | 13 +++++++------ + 1 file changed, 7 insertions(+), 6 deletions(-) + +commit 62efd48a825e8f439e84c85e165d8774ddc68fd2 +Author: Jia Tan +Date: 2023-01-11 23:58:16 +0800 + + Add NEWS for 5.4.1. + + NEWS | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 70 insertions(+) + +commit d1561c47ec8cd3844a785d3741dc932f9b9c5790 +Author: Jia Tan +Date: 2023-01-11 22:46:48 +0800 + + xz: Fix warning -Wformat-nonliteral on clang in message.c. + + clang and gcc differ in how they handle -Wformat-nonliteral. gcc will + allow a non-literal format string as long as the function takes its + format arguments as a va_list. + + src/xz/message.c | 9 +++++++++ + 1 file changed, 9 insertions(+) + +commit 8c0f115cc489331c48df77beca92fe378039d919 +Author: Jia Tan +Date: 2023-01-11 20:58:31 +0800 + + Tests: Fix test_filter_flags copy/paste error. + + tests/test_filter_flags.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 25035813d1d596fde692addc33e7f715f1fe55eb +Author: Jia Tan +Date: 2023-01-11 20:42:29 +0800 + + Tests: Fix type-limits warning in test_filter_flags. + + This only occurs in test_filter_flags when the BCJ filters are not + configured and built. In this case, ARRAY_SIZE() returns 0 and causes a + type-limits warning with the loop variable since an unsigned number will + always be >= 0. + + tests/test_filter_flags.c | 13 ++++++++++--- + 1 file changed, 10 insertions(+), 3 deletions(-) + +commit 0b8fa310cf56fec55663f62340e49e8e1441594f +Author: Lasse Collin +Date: 2023-01-10 22:14:03 +0200 + + liblzma: CLMUL CRC64: Work around a bug in MSVC, second attempt. + + This affects only 32-bit x86 builds. x86-64 is OK as is. + + I still cannot easily test this myself. The reporter has tested + this and it passes the tests included in the CMake build and + performance is good: raw CRC64 is 2-3 times faster than the + C version of the slice-by-four method. (Note that liblzma doesn't + include a MSVC-compatible version of the 32-bit x86 assembly code + for the slice-by-four method.) + + Thanks to Iouri Kharon for figuring out a fix, testing, and + benchmarking. + + src/liblzma/check/crc64_fast.c | 18 ++++++++++++++++++ + 1 file changed, 18 insertions(+) + +commit 765354b50c2886fc0d294d6be3b207f7ae2ada70 +Author: Jia Tan +Date: 2023-01-11 01:18:50 +0800 + + Tests: Fix unused function warning in test_block_header. + + One of the global arrays of filters was only used in a test that + required both encoders and decoders to be configured in the build. + + tests/test_block_header.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 7c23c05befdcc73231c0d6632a7d943dbeaea1aa +Author: Jia Tan +Date: 2023-01-11 01:08:03 +0800 + + Tests: Fix unused function warning in test_index_hash. + + test_index_hash does not use fill_index_hash() unless both encoders + and decoders are configured in the build. + + tests/test_index_hash.c | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +commit 57464bb4ebd6c00dc8b19803f05ea55ddd0826f6 +Author: Jia Tan +Date: 2023-01-11 00:54:45 +0800 + + CI/CD: Add 32-bit build and test steps to Ubuntu autotools runner. + + If all goes well, Mac autotools and Linux and Mac CMake will be added + later for 32-bit builds. + + .github/workflows/ci.yml | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit 923eb689a4b863b6cca8df6360d4962aae994edf +Author: Jia Tan +Date: 2023-01-11 00:51:01 +0800 + + CI/CD: Enables warnings as errors in autotool build. + + This will help us catch warnings and potential bugs in builds that are + not often tested by us. + + build-aux/ci_build.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit feae5528a30c006b6e2f96a95116e20b983703fc +Author: Jia Tan +Date: 2023-01-11 00:48:35 +0800 + + CI/CD: Add -f argument to set CFLAGS in ci_build.sh. + + For now, the suggested option is for -m32 only, but this can be updated + later if other flags are deemed useful. + + build-aux/ci_build.sh | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit cfabb62a4874c146e7d6f30445637602545bc054 +Author: Lasse Collin +Date: 2023-01-10 12:47:16 +0200 + + Revert "liblzma: CLMUL CRC64: Workaround a bug in MSVC (VS2015-2022)." + + This reverts commit 36edc65ab4cf10a131f239acbd423b4510ba52d5. + + It was reported that it wasn't a good enough fix and MSVC + still produced (different kind of) bad code when building + for 32-bit x86 if optimizations are enabled. + + Thanks to Iouri Kharon. + + src/liblzma/check/crc64_fast.c | 6 ------ + 1 file changed, 6 deletions(-) + +commit 0b64215170dd3562f207ef26f794755bcd600526 +Author: Lasse Collin +Date: 2023-01-10 11:56:11 +0200 + + sysdefs.h: Don't include strings.h anymore. + + On some platforms src/xz/suffix.c may need for + strcasecmp() but suffix.c includes the header when it needs it. + + Unless there is an old system that otherwise supports enough C99 + to build XZ Utils but doesn't have C89/C90-compatible , + there should be no need to include in sysdefs.h. + + src/common/sysdefs.h | 6 ------ + 1 file changed, 6 deletions(-) + +commit ec2fc39fe4f4e6e242b3a669585049763968cdeb +Author: Lasse Collin +Date: 2023-01-10 11:23:41 +0200 + + xz: Include in suffix.c if needed for strcasecmp(). + + SUSv2 and POSIX.1‐2017 declare only a few functions in . + Of these, strcasecmp() is used on some platforms in suffix.c. + Nothing else in the project needs (at least if + building on a modern system). + + sysdefs.h currently includes if HAVE_STRINGS_H is + defined and suffix.c relied on this. + + Note that dos/config.h doesn't #define HAVE_STRINGS_H even though + DJGPP does have strings.h. It isn't needed with DJGPP as strcasecmp() + is also in in DJGPP. + + src/xz/suffix.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit 7049c4a76c805ad27d6cf4ee119a2ef2a7add59f +Author: Lasse Collin +Date: 2023-01-10 10:05:13 +0200 + + sysdefs.h: Fix a comment. + + src/common/sysdefs.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 194a5fab69277d9e804a6113b5f676b8666b3a61 +Author: Lasse Collin +Date: 2023-01-10 10:04:06 +0200 + + sysdefs.h: Don't include memory.h anymore even if it were available. + + It quite probably was never needed, that is, any system where memory.h + was required likely couldn't compile XZ Utils for other reasons anyway. + + XZ Utils 5.2.6 and later source packages were generated using + Autoconf 2.71 which no longer defines HAVE_MEMORY_H. So the code + being removed is no longer used anyway. + + src/common/sysdefs.h | 8 ++------ + 1 file changed, 2 insertions(+), 6 deletions(-) + +commit 5e34774c31d1b7509b5cb77a3be9973adec59ea0 +Author: Lasse Collin +Date: 2023-01-10 08:29:32 +0200 + + CMake: Fix appending to CMAKE_RC_FLAGS. + + It's a string, not a list. It only worked when the variable was empty. + + Thanks to Iouri Kharon. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 6e652ceb18c615c578c869db300fa0756788b4e0 +Author: Lasse Collin +Date: 2023-01-10 00:33:14 +0200 + + Windows: Update INSTALL-MSVC.txt to recommend CMake over project files. + + windows/INSTALL-MSVC.txt | 19 ++++++++++++------- + 1 file changed, 12 insertions(+), 7 deletions(-) + +commit 6b117d3b1fe91eb26d533ab16a2e552f84148d47 +Author: Lasse Collin +Date: 2023-01-09 23:41:25 +0200 + + CMake: Fix windres issues again. + + At least on some systems, GNU windres needs --use-temp-file + in addition to the \x20 hack to avoid spaces in the command line + argument. Hovever, that \x20 syntax is broken with llvm-windres + version 15.0.0 (results in "XZx20Utils") but luckily it works + with a regular space. Thus it is best to limit the workarounds + to GNU toolchain on Windows. + + CMakeLists.txt | 35 +++++++++++++++++++++++------------ + 1 file changed, 23 insertions(+), 12 deletions(-) + +commit 0c210ca7f489e971e94e1ddc72b0b0806e3c7935 +Author: Lasse Collin +Date: 2023-01-06 22:53:38 +0200 + + Tests: test_filter_flags: Clean up minor issues. + + Here are the list of the most significant issues addressed: + - Avoid using internal common.h header. It's not good to copy the + constants like this but common.h cannot be included for use outside + of liblzma. This is the quickest thing to do that could be fixed later. + + - Omit the INIT_FILTER macro. Initialization should be done with just + regular designated initializers. + + - Use start_offset = 257 for BCJ tests. It demonstrates that Filter + Flags encoder and decoder don't validate the options thoroughly. + 257 is valid only for the x86 filter. This is a bit silly but + not a significant problem in practice because the encoder and + decoder initialization functions will catch bad alignment still. + Perhaps this should be fixed but it's not urgent and doesn't need + to be in 5.4.x. + + - Various tweaks to comments such as filter id -> Filter ID + + tests/test_filter_flags.c | 153 +++++++++++++++++++++++----------------------- + 1 file changed, 78 insertions(+), 75 deletions(-) + +commit 5c9fdd3bf53a9655f5eb2807d662b3af0d5e1865 +Author: Jia Tan +Date: 2022-12-29 23:33:33 +0800 + + Tests: Refactors existing filter flags tests. + + Converts the existing filter flags tests into tuktests. + + tests/test_filter_flags.c | 655 ++++++++++++++++++++++++++++++++-------------- + 1 file changed, 457 insertions(+), 198 deletions(-) + +commit 36edc65ab4cf10a131f239acbd423b4510ba52d5 +Author: Lasse Collin +Date: 2023-01-09 12:22:05 +0200 + + liblzma: CLMUL CRC64: Workaround a bug in MSVC (VS2015-2022). + + I haven't tested with MSVC myself and there doesn't seem to be + information about the problem online, so I'm relying on the bug report. + + Thanks to Iouri Kharon for the bug report and the patch. + + src/liblzma/check/crc64_fast.c | 6 ++++++ + 1 file changed, 6 insertions(+) + +commit 790a12a95a78ff82d8c6d4efe3b789851ca9470d +Author: Lasse Collin +Date: 2023-01-09 11:27:24 +0200 + + CMake: Fix a copypaste error in xzdec Windows resource file handling. + + It was my mistake. Thanks to Iouri Kharon for the bug report. + + CMakeLists.txt | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 0e1545fea39c0514c7b7032a0a3592a9a33d2848 +Author: Lasse Collin +Date: 2023-01-08 00:32:29 +0200 + + Tests: tuktest.h: Support tuktest_malloc(0). + + It's not needed in XZ Utils at least for now. It's good to support + it still because if such use is needed later, it wouldn't be + caught on GNU/Linux since malloc(0) from glibc returns non-NULL. + + tests/tuktest.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 69d5d78c6904668eb09a131da86276beec3281f8 +Author: Lasse Collin +Date: 2023-01-08 00:24:23 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit dd38655f80c113c9db73b9ed370dc900e1c4dc41 +Author: Lasse Collin +Date: 2023-01-07 21:57:11 +0200 + + CMake: Update cmake_minimum_required from 3.13...3.16 to 3.13...3.25. + + The changes listed on cmake-policies(7) for versions 3.17 to 3.25 + shouldn't affect this project. + + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit a890a637bee9193d5b690aefa9a59eba5b8532ae +Author: Lasse Collin +Date: 2023-01-07 19:50:35 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 6e38e595dd56ac1800478cef1f6f754d0eba0d2e +Author: Lasse Collin +Date: 2023-01-07 19:50:03 +0200 + + CMake/Windows: Add resource files to xz.exe and xzdec.exe. + + The command line tools cannot be built with MSVC for now but + they can be built with MinGW-w64. + + Thanks to Iouri Kharon for the bug report and the original patch. + + CMakeLists.txt | 16 ++++++++++++++++ + 1 file changed, 16 insertions(+) + +commit 443dfebced041adc88f10d824188eeef5b5821a9 +Author: Lasse Collin +Date: 2023-01-07 19:48:52 +0200 + + CMake/Windows: Add a workaround for windres from GNU binutils. + + Thanks to Iouri Kharon for the bug report and the original patch. + + CMakeLists.txt | 21 ++++++++++++++++++++- + 1 file changed, 20 insertions(+), 1 deletion(-) + +commit ceb805011747d04a915f3f39e4bed9eed151c634 +Author: Lasse Collin +Date: 2023-01-07 19:31:15 +0200 + + Build: Require that _mm_set_epi64x() is usable to enable CLMUL support. + + VS2013 doesn't have _mm_set_epi64x() so this way CLMUL gets + disabled with VS2013. + + Thanks to Iouri Kharon for the bug report. + + CMakeLists.txt | 3 ++- + configure.ac | 8 ++++++-- + 2 files changed, 8 insertions(+), 3 deletions(-) + +commit 8d372bd94066b1a5b0570b2550f83c2868486adf +Author: Jia Tan +Date: 2023-01-07 21:05:15 +0800 + + CI/CD: Split CMake Linux and MacOS build phase to build and test. + + The phase split was only done for Autotools before, so should also + apply to CMake. + + .github/workflows/ci.yml | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit 747c7f2b34bd498f6702c6875500a26b06201772 +Author: Jia Tan +Date: 2023-01-07 11:16:55 +0800 + + CI/CD: Reduce job runners to 4 instead of using matrix strategy. + + The old version used too many runners that resulted in unnecessary + dependency downloads. Now, the runners are reused for the different + configurations for each OS and build system. + + .github/workflows/ci.yml | 95 ++++++++++++++++++++++++++++++++++++++++++------ + 1 file changed, 83 insertions(+), 12 deletions(-) + +commit 4de35fd6b58d46fc887c78faf163f6a37b790c45 +Author: Jia Tan +Date: 2023-01-07 10:07:20 +0800 + + CI/CD: Add new -p (PHASE) argument to ci_build.sh + + The new PHASE argument can be build, test, or all. all is the default. + This way, the CI/CD script can differentiate between the build and test + phases to make it easier to track down errors when they happen. + + build-aux/ci_build.sh | 140 +++++++++++++++++++++++++++----------------------- + 1 file changed, 76 insertions(+), 64 deletions(-) + +commit 6fd39664de47801e670a16617863196bfbde4755 +Merge: 78e0561d fc0c7884 +Author: Jia Tan +Date: 2023-01-07 00:10:50 +0800 + + Merge pull request #7 from tukaani-project/tuktest_index_hash + + Tuktest index hash + +commit fc0c788469159f634f09ff23c8cef6925c91da57 +Author: Lasse Collin +Date: 2023-01-06 17:58:48 +0200 + + Tests: test_index_hash: Add an assert_uint_eq(). + + tests/test_index_hash.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit d550304f5343b3a082da265107cd820e0d81dc71 +Author: Lasse Collin +Date: 2023-01-06 17:55:06 +0200 + + Tests: test_index_hash: Fix a memory leak. + + tests/test_index_hash.c | 2 ++ + 1 file changed, 2 insertions(+) + +commit 02608f74ea1f2d2d56585711ff241c34b4ad0937 +Author: Lasse Collin +Date: 2023-01-06 17:53:03 +0200 + + Tests: test_index_hash: Don't treat pointers as booleans. + + tests/test_index_hash.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 056766c8601a3808bea1761f6cc833197a35a3e0 +Author: Lasse Collin +Date: 2023-01-06 17:51:41 +0200 + + Tests: test_index_hash: Fix a typo in a comment. + + tests/test_index_hash.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 873e684028ba9738f071c5236db7d452ed797b4c +Author: Lasse Collin +Date: 2023-01-06 17:44:29 +0200 + + Tests: test_index_hash: Avoid the variable name "index". + + It can trigger warnings from -Wshadow on some systems. + + tests/test_index_hash.c | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit d1f24c35874eeba8432d75aa77b06c50375ed937 +Author: Lasse Collin +Date: 2023-01-06 17:35:50 +0200 + + Tests: test_index_hash: Use the word "Record" instead of "entry". + + tests/test_index_hash.c | 102 ++++++++++++++++++++++++------------------------ + 1 file changed, 51 insertions(+), 51 deletions(-) + +commit b93f7c5cbb02b42024ac866fc0af541de3d816e2 +Author: Lasse Collin +Date: 2023-01-06 17:35:05 +0200 + + Tests: test_index_hash: Tweak comments and style. + + The words defined in the .xz file format specification + begin with capital letter to emphasize that they have + a specific meaning. + + tests/test_index_hash.c | 62 ++++++++++++++++++++++++++----------------------- + 1 file changed, 33 insertions(+), 29 deletions(-) + +commit c48b24fc06d98569adb72f13c2e8e5ff30bb8036 +Author: Lasse Collin +Date: 2023-01-06 17:17:37 +0200 + + Tests: test_index_hash: Use INDEX_INDICATOR constant instead of 0. + + tests/test_index_hash.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 78e0561dfebaa9d5e34558de537efcda890e0629 +Author: Jia Tan +Date: 2023-01-06 20:43:31 +0800 + + Style: Change #if !defined() to #ifndef in mythread.h. + + src/common/mythread.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit e834e1e934ed0af673598d8c0c34afb2af56bee0 +Author: Jia Tan +Date: 2023-01-06 20:35:55 +0800 + + Build: Add missing stream_decoder_mt.c to .vcxproj files. + + The line in the .vcxproj files for building with was missing in 5.4.0. + Thank to Hajin Jang for reporting the issue. + + windows/vs2013/liblzma.vcxproj | 1 + + windows/vs2013/liblzma_dll.vcxproj | 1 + + windows/vs2017/liblzma.vcxproj | 1 + + windows/vs2017/liblzma_dll.vcxproj | 1 + + windows/vs2019/liblzma.vcxproj | 1 + + windows/vs2019/liblzma_dll.vcxproj | 1 + + 6 files changed, 6 insertions(+) + +commit 84f9687cbae972c2c342e10bf69f8ec8f70ae111 +Author: Jia Tan +Date: 2023-01-05 20:57:25 +0800 + + liblzma: Remove common.h include from common/index.h. + + common/index.h is needed by liblzma internally and tests. common.h will + include and define many things that are not needed by the tests. Also, + this prevents include order problems because common.h will redefine + LZMA_API resulting in a warning. + + src/liblzma/common/index.c | 1 + + src/liblzma/common/index.h | 9 +++++++-- + src/liblzma/common/index_decoder.h | 1 + + src/liblzma/common/stream_buffer_encoder.c | 1 + + 4 files changed, 10 insertions(+), 2 deletions(-) + +commit 7657ce1c3c4abff7560336a7b687d98e0e2bd14f +Author: Lasse Collin +Date: 2023-01-04 22:40:54 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit aafd67fba045ab99683971263a5a26fb2a6e8ce2 +Author: Lasse Collin +Date: 2023-01-04 18:40:28 +0200 + + Tests: Adjust style in test_compress.sh. + + tests/test_compress.sh | 12 +++++++----- + 1 file changed, 7 insertions(+), 5 deletions(-) + +commit 52380678f42364daa4510f92f6d3b18ec98c3638 +Author: Jia Tan +Date: 2023-01-04 23:58:58 +0800 + + Tests: Replace non portable shell parameter expansion + + The shell parameter expansion using # and ## is not supported in + Solaris 10 Bourne shell (/bin/sh). Even though this is POSIX, it is not fully + portable, so we should avoid it. + + tests/create_compress_files.c | 2 +- + tests/test_compress.sh | 20 +++++++++++++------- + tests/test_compress_prepared_bcj_sparc | 2 +- + tests/test_compress_prepared_bcj_x86 | 2 +- + 4 files changed, 16 insertions(+), 10 deletions(-) + +commit d0eb345bb7d148a62883ee299adec2b74a0f6f3b +Author: Jia Tan +Date: 2023-01-03 21:02:38 +0800 + + Translations: Add Korean translation of man pages. + + Thanks to Seong-ho Cho + + po4a/ko.po | 5552 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + po4a/po4a.conf | 2 +- + 2 files changed, 5553 insertions(+), 1 deletion(-) + +commit c4145978d95ebf1690c778d354e15f7c2823d7a8 +Author: Jia Tan +Date: 2023-01-03 20:47:27 +0800 + + Translations: Update the Esperanto translation. + + po/eo.po | 620 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 332 insertions(+), 288 deletions(-) + +commit 4103a2e78ac60b00c888485cd967a5fe5d1b917c +Author: Lasse Collin +Date: 2023-01-02 17:20:47 +0200 + + Bump version and soname for 5.5.0alpha. + + 5.5.0alpha won't be released, it's just to mark that + the branch is not for stable 5.4.x. + + Once again there is no API/ABI stability for new features + in devel versions. The major soname won't be bumped even + if API/ABI of new features breaks between devel releases. + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 4 ++-- + 2 files changed, 3 insertions(+), 3 deletions(-) + +commit 73c9e6d6b970ccc3d5ad61dcaa21cba050e5df0a +Author: Lasse Collin +Date: 2023-01-02 17:05:07 +0200 + + Build: Fix config.h comments. + + configure.ac | 2 +- + m4/tuklib_progname.m4 | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit bb740e3b117f1a3c65152d01e5755523a908ecb1 +Author: Jia Tan +Date: 2023-01-02 22:33:48 +0800 + + Build: Only define HAVE_PROGRAM_INVOCATION_NAME if it is set to 1. + + HAVE_DECL_PROGRAM_INVOCATION_NAME is renamed to + HAVE_PROGRAM_INVOCATION_NAME. Previously, + HAVE_DECL_PROGRAM_INVOCATION_NAME was always set when + building with autotools. CMake would only set this when it was 1, and the + dos/config.h did not define it. The new macro definition is consistent + across build systems. + + cmake/tuklib_progname.cmake | 5 ++--- + m4/tuklib_progname.m4 | 5 ++++- + src/common/tuklib_progname.c | 2 +- + src/common/tuklib_progname.h | 2 +- + 4 files changed, 8 insertions(+), 6 deletions(-) + +commit 064cd385a716abc78d93a3612411a82d69ceb221 +Author: Jia Tan +Date: 2022-12-29 00:30:52 +0800 + + Adds test_index_hash to .gitignore. + + .gitignore | 1 + + 1 file changed, 1 insertion(+) + +commit 3959162baec074511d83ba0fec1284c3ed724799 +Author: Jia Tan +Date: 2022-12-29 00:25:18 +0800 + + Tests: Creates test_index_hash.c + + Tests all API functions exported from index_hash.h. Does not have a + dedicated test for lzma_index_hash_end. + + CMakeLists.txt | 2 + + tests/Makefile.am | 3 + + tests/test_index_hash.c | 379 ++++++++++++++++++++++++++++++++++++++++++++++++ + 3 files changed, 384 insertions(+) + +commit f16e12d5e755d371247202fcccbcccd1ec16b2cf +Author: Jia Tan +Date: 2022-08-17 20:20:16 +0800 + + liblzma: Add NULL check to lzma_index_hash_append. + + This is for consistency with lzma_index_append. + + src/liblzma/common/index_hash.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 203b008eb220208981902e0db541c02d1c1c9f5e +Author: Jia Tan +Date: 2022-08-17 17:59:51 +0800 + + liblzma: Replaced hardcoded 0x0 index indicator byte with macro + + src/liblzma/common/index.h | 3 +++ + src/liblzma/common/index_decoder.c | 2 +- + src/liblzma/common/index_encoder.c | 2 +- + src/liblzma/common/index_hash.c | 2 +- + src/liblzma/common/stream_decoder.c | 3 ++- + src/liblzma/common/stream_decoder_mt.c | 2 +- + 6 files changed, 9 insertions(+), 5 deletions(-) + +commit dfecda875211f737d0db92dc1d3c58a3a2afb0c0 +Author: Lasse Collin +Date: 2022-12-30 20:10:08 +0200 + + Tests: test_check: Test corner cases of CLMUL CRC64. + + tests/test_check.c | 27 +++++++++++++++++++++++++++ + 1 file changed, 27 insertions(+) + +commit ce96bb20435212fe797d6d84738fb9fd4ea13cc7 +Author: Lasse Collin +Date: 2022-12-30 19:36:49 +0200 + + Tests: Clarify a comment in test_lzip_decoder.c. + + tests/test_lzip_decoder.c | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit 2fcba17fc4d7eda8fc60567169cf2a0e6fcfb2f8 +Author: Jia Tan +Date: 2022-12-29 01:55:19 +0800 + + xz: Includes and conditionally in mytime.c. + + Previously, mytime.c depended on mythread.h for to be included. + + src/xz/mytime.c | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit f82294c8318a7a0990583d51ac5c7de682ad36ef +Author: Jia Tan +Date: 2022-12-29 01:15:27 +0800 + + liblzma: Includes sys/time.h conditionally in mythread + + Previously, was always included, even if mythread only used + clock_gettime. is still needed even if clock_gettime is not used + though because struct timespec is needed for mythread_condtime. + + src/common/mythread.h | 8 +++++++- + 1 file changed, 7 insertions(+), 1 deletion(-) + +commit 74dae7d30091e906d6a92a57952dea4354473f9b +Author: Jia Tan +Date: 2022-12-29 01:10:53 +0800 + + Build: No longer require HAVE_DECL_CLOCK_MONOTONIC to always be set. + + Previously, if threading was enabled HAVE_DECL_CLOCK_MONOTONIC would always + be set to 0 or 1. However, this macro was needed in xz so if xz was not + built with threading and HAVE_DECL_CLOCK_MONOTONIC was not defined but + HAVE_CLOCK_GETTIME was, it caused a warning during build. Now, + HAVE_DECL_CLOCK_MONOTONIC has been renamed to HAVE_CLOCK_MONOTONIC and + will only be set if it is 1. + + CMakeLists.txt | 8 +++----- + configure.ac | 5 ++++- + src/common/mythread.h | 4 ++-- + src/xz/mytime.c | 5 ++--- + 4 files changed, 11 insertions(+), 11 deletions(-) + +commit 7339e39dc060df6eda74a2c5b69961befc3d5d24 +Author: Jia Tan +Date: 2022-12-28 01:14:07 +0800 + + Translations: Add Ukrainian translations of man pages. + + Thanks to Yuri Chornoivan + + po4a/po4a.conf | 2 +- + po4a/uk.po | 3676 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 3677 insertions(+), 1 deletion(-) + +commit 9f05c27a58ce8cd7803079aa295e41c24665ce6e +Author: Jia Tan +Date: 2022-12-23 00:34:48 +0800 + + CI/CD: Create initial version of CI/CD workflow. + + The CI/CD workflow will only execute on Ubuntu and MacOS latest version. + The workflow will attempt to build with autotools and CMake and execute + the tests. The workflow will run for all pull requests and pushes done + to the master branch. + + .github/workflows/ci.yml | 72 ++++++++++++++++++++++++ + build-aux/ci_build.sh | 141 +++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 213 insertions(+) + +commit 1275ebfba74230dbd028049141423c79c8b83b8f +Author: Jia Tan +Date: 2022-12-22 23:14:53 +0800 + + liblzma: Update documentation for lzma_filter_encoder. + + src/liblzma/common/filter_encoder.c | 7 +++++-- + 1 file changed, 5 insertions(+), 2 deletions(-) + +commit 7c9ff5f1667a16733163b75dfd4b509662c387f4 +Author: Jia Tan +Date: 2022-12-21 21:12:03 +0800 + + Tests: Adds lzip decoder tests + + .gitignore | 1 + + tests/Makefile.am | 2 + + tests/test_lzip_decoder.c | 471 ++++++++++++++++++++++++++++++++++++++++++++++ + 3 files changed, 474 insertions(+) + +commit 799ead162de63b8400733603d3abcd2e1977bdca +Author: Jia Cheong Tan +Date: 2022-12-20 22:05:21 +0800 + + Doxygen: Update .gitignore for generating docs for in source build. + + In source builds are not recommended, but we should still ignore + the generated artifacts. + + .gitignore | 2 ++ + 1 file changed, 2 insertions(+) + +commit 5f7ce42a16b1e86ca8408b5c670c25e2a12acc4e +Author: Jia Tan +Date: 2022-12-20 20:46:44 +0800 + + liblzma: Fix lzma_microlzma_encoder() return value. + + Using return_if_error on lzma_lzma_lclppb_encode was improper because + return_if_error is expecting an lzma_ret value, but + lzma_lzma_lclppb_encode returns a boolean. This could result in + lzma_microlzma_encoder, which would be misleading for applications. + + src/liblzma/common/microlzma_encoder.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +commit 8ace358d65059152d9a1f43f4770170d29d35754 +Author: Jia Tan +Date: 2022-12-16 20:58:55 +0800 + + CMake: Update .gitignore for CMake artifacts from in source build. + + In source builds are not recommended, but we can make it easier + by ignoring the generated artifacts from CMake. + + .gitignore | 23 +++++++++++++++++++++++ + 1 file changed, 23 insertions(+) + +commit 8fd225a2c149f30aeac377e68eb5abf6b28300ad +Author: Lasse Collin +Date: 2022-12-16 18:30:02 +0200 + + liblzma: Update authors list in arm64.c. + + src/liblzma/simple/arm64.c | 1 + + 1 file changed, 1 insertion(+) + +commit b69da6d4bb6bb11fc0cf066920791990d2b22a06 +Author: Lasse Collin +Date: 2022-12-13 20:37:17 +0200 + + Bump version to 5.4.0 and soname to 5.4.0. + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 6 +++--- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/NEWS b/NEWS new file mode 100644 index 0000000000..978ef54b91 --- /dev/null +++ b/NEWS @@ -0,0 +1,2920 @@ + +XZ Utils Release Notes +====================== + +5.8.1 (2025-04-03) + + IMPORTANT: This includes a security fix for CVE-2025-31115 which + affects XZ Utils from 5.3.3alpha to 5.8.0. No new 5.4.x or 5.6.x + releases will be made, but the fix is in the v5.4 and v5.6 branches + in the xz Git repository. A standalone patch for all affected + versions is available as well. + + * Multithreaded .xz decoder (lzma_stream_decoder_mt()): + + - Fix a bug that could at least result in a crash with + invalid input. (CVE-2025-31115) + + - Fix a performance bug: Only one thread was used if the whole + input file was provided at once to lzma_code(), the output + buffer was big enough, timeout was disabled, and LZMA_FINISH + was used. There are no bug reports about this, thus it's + possible that no real-world application was affected. + + * Avoid even with C11/C17 compilers. This fixes the + build with Oracle Developer Studio 12.6 on Solaris 10 when the + compiler is in C11 mode (the header doesn't exist). + + * Autotools: Restore compatibility with GNU make versions older + than 4.0 by creating the package using GNU gettext 0.23.1 + infrastructure instead of 0.24. + + * Update Croatian translation. + + +5.8.0 (2025-03-25) + + This bumps the minor version of liblzma because new features were + added. The API and ABI are still backward compatible with liblzma + 5.6.x, 5.4.x, 5.2.x, and 5.0.x. + + * liblzma on 32/64-bit x86: When possible, use SSE2 intrinsics + instead of memcpy() in the LZMA/LZMA2 decoder. In typical cases, + this may reduce decompression time by 0-5 %. However, when built + against musl libc, over 15 % time reduction was observed with + highly compressed files. + + * CMake: Make the feature test macros match the Autotools-based + build on NetBSD, Darwin, and mingw-w64. + + * Update the Croatian, Italian, Portuguese, and Romanian + translations. + + * Update the German, Italian, Korean, Romanian, Serbian, and + Ukrainian man page translations. + + Summary of changes in the 5.7.x development releases: + + * Mark the following LZMA Utils script aliases as deprecated: + lzcmp, lzdiff, lzless, lzmore, lzgrep, lzegrep, and lzfgrep. + + * liblzma: + + - Improve LZMA/LZMA2 encoder speed on 64-bit PowerPC (both + endiannesses) and those 64-bit RISC-V processors that + support fast unaligned access. + + - Add low-level APIs for RISC-V, ARM64, and x86 BCJ filters + to lzma/bcj.h. These are primarily for erofs-utils. + + - x86/x86-64/E2K CLMUL CRC code was rewritten. + + - Use the CRC32 instructions on LoongArch. + + * xz: + + - Synchronize the output file and its directory using fsync() + before deleting the input file. No syncing is done when xz + isn't going to delete the input file. + + - Add --no-sync to disable the sync-before-delete behavior. + + - Make --single-stream imply --keep. + + * xz, xzdec, lzmainfo: When printing messages, replace + non-printable characters with question marks. + + * xz and xzdec on Linux: Support Landlock ABI versions 5 and 6. + + * CMake: Revise the configuration variables and some of their + options, and document them in the file INSTALL. CMake support + is no longer experimental. (It was already not experimental + when building for native Windows.) + + * Add build-aux/license-check.sh. + + +5.7.2beta (2025-03-08) + + * On the man pages, mark the following LZMA Utils script aliases as + deprecated: lzcmp, lzdiff, lzless, lzmore, lzgrep, lzegrep, and + lzfgrep. The commands that start with xz* instead of lz* have + identical behavior. + + The LZMA Utils aliases lzma, unlzma, and lzcat aren't deprecated + because some of these aliases are still in common use. lzmadec + and lzmainfo aren't deprecated either. + + * xz: In the ENVIRONMENT section of the man page, warn about + problems that some uses of XZ_DEFAULTS and XZ_OPT may create. + + * Windows (native builds, not Cygwin): In xz, xzdec, and lzmadec, + avoid an error message on broken pipe. + + * Autotools: Fix out-of-tree builds when using the bundled + getopt_long. + + * Translations: + + - Updated: Chinese (traditional), Croatian, Finnish, Georgian, + German, Korean, Polish, Romanian, Serbian, Spanish, Swedish, + Turkish, and Ukrainian + + - Added: Dutch + + * Man page translations: + + - Updated: German, Korean, Romanian, and Ukrainian + + - Added: Italian and Serbian + + +5.7.1alpha (2025-01-23) + + * All fixes from 5.6.4. + + * liblzma: + + - Improve LZMA/LZMA2 encoder speed on 64-bit PowerPC (both + endiannesses) and those 64-bit RISC-V processors that + support fast unaligned access. + + - x86/x86-64/E2K CLMUL CRC code was rewritten. It's faster and + doesn't cause false positives from sanitizers. Attributes + like __attribute__((__no_sanitize_address__)) are no longer + present. + + - On 32-bit x86, CLMUL CRC and the old (but still good) + assembly versions now co-exist with runtime detection. + Both Autotools and CMake build systems handle this + automatically now. + + - Use the CRC32 instructions on LoongArch to make CRC32 + calculation faster. + + - Add low-level APIs for RISC-V, ARM64, and x86 BCJ filters + to lzma/bcj.h. These are primarily for erofs-utils. + + - Minor tweaks to ARM64 CRC32 code and BCJ filters were made. + + * xz: + + - Synchronize the output file and its directory before deleting + the input file using fsync(). This reduces the probability of + data loss after a system crash. However, it can be a major + performance hit if processing many small files. + + NOTE: No syncing is done when xz isn't going to delete + the input file. + + - Add a new option --no-sync to disable the sync-before-delete + behavior. It's useful when compressing many small files and + one doesn't worry about data loss in case of a system crash. + + - Make --single-stream imply --keep. + + - Use automatic word wrapping for the text in --help and + similar situations to hopefully make the strings easier for + majority of translators (no need to count spaces anymore). + + * xz, xzdec, lzmainfo: When printing messages, replace + non-printable characters with question marks. This way + malicious filenames cannot be used to send escape sequences + to a terminal. This change is also applied to filenames shown + in "xz --robot --list". + + * xz and xzdec on Linux: Add support for Landlock ABI versions 5 + and 6. + + * CMake updates: + + - Increase the minimum required CMake version to 3.20. + + - Revise the configuration variables and some of their options. + Document them in the file INSTALL. + + - Attempt to produce liblzma.pc so that the paths are based on + ${prefix}, which makes it simpler to override the paths + if the liblzma files have been moved. + + - To enable translations, gettext-tools is now required. The + CMake build no longer supports installing pre-compiled + message catalog binary files (po/*.gmo). + + - Apple: Use Mach-O shared library versioning that is + compatible with GNU Libtool. This should make it easier to + switch between the build systems on Apple OSes that enforce + the correct compatibility_version (macOS >= 12 doesn't?). + This change is backward compatible: binaries linked against + old CMake-built liblzma will run with liblzma that uses + Libtool style versioning. + + - Windows (not Cygwin): Document CMAKE_DLL_NAME_WITH_SOVERSION + (CMake >= 3.27) in the file INSTALL. This option should + usually be left to its default value (OFF). It can be set + to ON if the liblzma DLL filename must be compatible with + the versioned filename produced by GNU Libtool. For example, + binaries distributed in MSYS2 use a versioned DLL filename. + + - CMake support is no longer experimental. (It was already + not experimental when building for native Windows.) + + * Windows: Building liblzma with Visual Studio 2013 is no longer + supported. Visual Studio 2015 or later (with CMake) can be used + to build liblzma and the command line tools. + + * Add preliminary Georgian translation. This already contains + translations of most of the strings that are now automatically + word wrapped. + + * Add build-aux/license-check.sh. Without arguments, it checks that + no license information has been forgotten. With the -v argument, + it shows the license info (or the lack of it) for each file. + + If the .git directory is available, only the files in the + repository are checked. Without the .git directory, a clean tree + from an extracted release tarball is expected. + + +5.6.4 (2025-01-23) + + * liblzma: Fix LZMA/LZMA2 encoder on big endian ARM64. + + * xz: + + - Fix --filters= and --filters1= ... --filters9= options + parsing. They require an argument, thus "xz --filters lzma2" + should work in addition to "xz --filters=lzma2". + + - On the man page, note in the --compress and --decompress + options that the default behavior is to delete the input + file unless writing to standard output. It was already + documented in the DESCRIPTION section but new users in + a hurry might miss it. + + * Windows (native builds, not Cygwin): Fix regressions introduced + in XZ Utils 5.6.3 which caused non-ASCII characters to display + incorrectly. Only builds with translation support were affected + (--enable-nls or ENABLE_NLS=ON). The following changes affect + builds that have translations enabled: + + - Require UCRT because MSVCRT doesn't support UTF-8 + locales and thus translations won't be readable on + Windows 10 version 1903 and later. (MSVCRT builds + are still possible with --disable-nls or ENABLE_NLS=OFF.) + + - Require gettext-runtime >= 0.23.1 because older versions + don't autodetect the use of the UTF-8 code page. This + resulted in garbled non-ASCII characters even with UCRT. + + - Partially fix alignment issues in xz --verbose --list + with translated messages. Chinese (simplified), + Chinese (traditional), and Korean column headings + are misaligned still because Windows and MinGW-w64 + don't provide wcwidth() and XZ Utils doesn't include + a replacement function either. + + * CMake: Explicitly disable unity builds. This prevents build + failures when another project uses XZ Utils via CMake's + FetchContent module, and that project enables unity builds. + + * Update Chinese (traditional) and Serbian translations. + + +5.6.3 (2024-10-01) + + IMPORTANT: This includes a Windows-specific security fix to + the command line tools (CVE-2024-47611). liblzma isn't affected + by this issue. + + * liblzma: + + - Fix x86-64 inline assembly compatibility with GNU Binutils + older than 2.27. + + - Fix the build with GCC 4.2 on OpenBSD/sparc64. + + * xzdec: Display an error instead of failing silently if the + unsupported option -M is specified. + + * lzmainfo: Fix integer overflows when rounding the dictionary and + uncompressed sizes to the nearest mebibyte. + + * Windows (except Cygwin and MSYS2): Add an application manifest to + xz, xzdec, lzmadec, and lzmainfo executables: + + - Declare them compatible with Vista/7/8/8.1/10/11. This way + the programs won't needlessly use Operating System Context + of Vista when running on later Windows versions. This setting + doesn't mean that the executables cannot run on even older + versions if otherwise built that way. + + - Declare them as UAC-compliant. MSVC added this by default + already but it wasn't done with MinGW-w64, at least not + with all toolchain variants. + + - Declare them long path aware. This makes long path names + work on Windows 10 and 11 if the feature has been enabled + in the Windows registry. + + - Use the UTF-8 code page on Windows 10 version 1903 and later. + + * Now command line tools can access files whose names + contain characters that don't exist in the current + legacy code page. + + * The options --files and --files0 now expect file lists + to be in UTF-8 instead of the legacy code page. + + * This fixes a security issue: If a command line contains + Unicode characters (for example, filenames) that don't + exist in the current legacy code page, the characters are + converted to similar-looking characters with best-fit + mapping. Some best-fit mappings result in ASCII + characters that change the meaning of the command line, + which can be exploited with malicious filenames to do + argument injection or directory traversal attacks. + UTF-8 avoids best-fit mappings and thus fixes the issue. + (CVE-2024-47611) + + Forcing the process code page to UTF-8 is possible only + on Windows 10 version 1903 and later. The command line + tools remain vulnerable if used on an old older + version of Windows. + + This issue was discovered by Orange Tsai and splitline + from DEVCORE Research Team. + + A related smaller issue remains: Windows filenames may + contain unpaired surrogates (invalid UTF-16). These are + converted to the replacement character U+FFFD in the + UTF-8 code page. Thus, filenames with different unpaired + surrogates appear identical and aren't distinguishable + from filenames that contain the actual replacement + character U+FFFD. + + * When building with MinGW-w64, it is recommended to use + UCRT version instead of the old MSVCRT. For example, + non-ASCII characters from filenames won't print + correctly in messages to console with MSVCRT with + the UTF-8 code page (a cosmetic issue). liblzma-only + builds are still fine with MSVCRT. + + - Cygwin and MSYS2 process command line options differently and + the above issues don't exist. There is no need to replace the + default application manifest on Cygwin and MSYS2. + + * Autotools-based build: + + - Fix feature checks with link-time optimization (-flto). + + - Solaris: Fix a compatibility issue in version.sh. It matters + if one wants to regenerate configure by running autoconf. + + * CMake: + + - Use paths relative to ${prefix} in liblzma.pc when possible. + This is done only with CMake >= 3.20. + + - MSVC: Install liblzma.pc as it can be useful with MSVC too. + + - Windows: Fix liblzma filename prefix, for example: + + * Cygwin: The DLL was incorrectly named liblzma-5.dll. + Now it is cyglzma-5.dll. + + * MSVC: Rename import library from liblzma.lib to lzma.lib + while keeping liblzma.dll name as is. This helps with + "pkgconf --msvc-syntax --libs liblzma" because it mungles + "-llzma" in liblzma.pc to "lzma.lib". + + * MinGW-w64: No changes. + + - Windows: Use the correct resource file for lzmadec.exe. + Previously the resource file for xzdec.exe was used for both. + Autotools-based build isn't affected. + + - Prefer a C11 compiler over a C99 compiler but accept both. + + - Link Threads::Threads against liblzma using PRIVATE so that + -pthread and such flags won't unnecessarily get included in + the usage requirements of shared liblzma. That is, + target_link_libraries(foo PRIVATE liblzma::liblzma) no + longer adds -pthread if using POSIX threads and linking + against shared liblzma. The threading flags are still added + if linking against static liblzma. + + * Updated translations: Catalan, Chinese (simplified), and + Brazilian Portuguese. + + +5.6.2 (2024-05-29) + + * Remove the backdoor (CVE-2024-3094). + + * Not changed: Memory sanitizer (MSAN) has a false positive + in the CRC CLMUL code which also makes OSS Fuzz unhappy. + Valgrind is smarter and doesn't complain. + + A revision to the CLMUL code is coming anyway and this issue + will be cleaned up as part of it. It won't be backported to + 5.6.x or 5.4.x because the old code isn't wrong. There is + no reason to risk introducing regressions in old branches + just to silence a false positive. + + * liblzma: + + - lzma_index_decoder() and lzma_index_buffer_decode(): Fix + a missing output pointer initialization (*i = NULL) if the + functions are called with invalid arguments. The API docs + say that such an initialization is always done. In practice + this matters very little because the problem can only occur + if the calling application has a bug and these functions + return LZMA_PROG_ERROR. + + - lzma_str_to_filters(): Fix a missing output pointer + initialization (*error_pos = 0). This is very similar + to the fix above. + + - Fix C standard conformance with function pointer types. + + - Remove GNU indirect function (IFUNC) support. This is *NOT* + done for security reasons even though the backdoor relied on + this code. The performance benefits of IFUNC are too tiny in + this project to make the extra complexity worth it. + + - FreeBSD on ARM64: Add error checking to CRC32 instruction + support detection. + + - Fix building with NVIDIA HPC SDK. + + * xz: + + - Fix a C standard conformance issue in --block-list parsing + (arithmetic on a null pointer). + + - Fix a warning from GNU groff when processing the man page: + "warning: cannot select font 'CW'" + + * xzdec: Add support for Linux Landlock ABI version 4. xz already + had the v3-to-v4 change but it had been forgotten from xzdec. + + * Autotools-based build system (configure): + + - Symbol versioning variant can now be overridden with + --enable-symbol-versions. Documentation in INSTALL was + updated to match. + + - Add new configure option --enable-doxygen to enable + generation and installation of the liblzma API documentation + using Doxygen. Documentation in INSTALL and PACKAGERS was + updated to match. + + CMake: + + - Fix detection of Linux Landlock support. The detection code + in CMakeLists.txt had been sabotaged. + + - Disable symbol versioning on non-glibc Linux to match what + the Autotools build does. For example, symbol versioning + isn't enabled with musl. + + - Symbol versioning variant can now be overridden by setting + SYMBOL_VERSIONING to "OFF", "generic", or "linux". + + - Add support for all tests in typical build configurations. + Now the only difference to the tests coverage to Autotools + is that CMake-based build will skip more tests if features + are disabled. Such builds are only for special cases like + embedded systems. + + - Separate the CMake code for the tests into tests/tests.cmake. + It is used conditionally, thus it is possible to + + rm -rf tests + + and the CMake-based build will still work normally except + that no tests are then available. + + - Add a option ENABLE_DOXYGEN to enable generation and + installation of the liblzma API documentation using Doxygen. + + * Documentation: + + - Omit the Doxygen-generated liblzma API documentation from the + package. Instead, the generation and installation of the API + docs can be enabled with a configure or CMake option if + Doxygen is available. + + - Remove the XZ logo which was used in the API documentation. + The logo has been retired and isn't used by the project + anymore. However, it's OK to use it in contexts that refer + to the backdoor incident. + + - Remove the PDF versions of the man pages from the source + package. These existed primarily for users of operating + systems which don't come with tools to render man page + source files. The plain text versions are still included + in doc/man/txt. PDF files can still be generated to doc/man, + if the required tools are available, using "make pdf" after + running "configure". + + - Update home page URLs back to their old locations on + tukaani.org. + + - Update maintainer info. + + * Tests: + + - In tests/files/README, explain how to recreate the ARM64 + test files. + + - Remove two tests that used tiny x86 and SPARC object files + as the input files. The matching .c file was included but + the object files aren't easy to reproduce. The test cases + weren't great anyway; they were from the early days (2009) + of the project when the test suite had very few tests. + + - Improve a few tests. + + +5.6.1 (2024-03-09) + + IMPORTANT: This fixed bugs in the backdoor (CVE-2024-3094) (someone + had forgot to run Valgrind). + + * liblzma: Fixed two bugs relating to GNU indirect function (IFUNC) + with GCC. The more serious bug caused a program linked with + liblzma to crash on start up if the flag -fprofile-generate was + used to build liblzma. The second bug caused liblzma to falsely + report an invalid write to Valgrind when loading liblzma. + + * xz: Changed the messages for thread reduction due to memory + constraints to only appear under the highest verbosity level. + + * Build: + + - Fixed a build issue when the header file + was present on the system but the Landlock system calls were + not defined in . + + - The CMake build now warns and disables NLS if both gettext + tools and pre-created .gmo files are missing. Previously, + this caused the CMake build to fail. + + * Minor improvements to man pages. + + * Minor improvements to tests. + + +5.6.0 (2024-02-24) + + IMPORTANT: This added a backdoor (CVE-2024-3094). It's enabled only + in the release tarballs. + + This bumps the minor version of liblzma because new features were + added. The API and ABI are still backward compatible with liblzma + 5.4.x and 5.2.x and 5.0.x. + + NOTE: As described in the NEWS for 5.5.2beta, the core components + are now under the BSD Zero Clause License (0BSD). + + Since 5.5.2beta: + + * liblzma: + + - Disabled the branchless C variant in the LZMA decoder based + on the benchmark results from the community. + + - Disabled x86-64 inline assembly on x32 to fix the build. + + * Sandboxing support in xz: + + - Landlock is now used even when xz needs to create files. + In this case the sandbox has to be more permissive than + when no files need to be created. A similar thing was + already in use with pledge(2) since 5.3.4alpha. + + - Landlock and pledge(2) are now stricter when reading from + more than one input file and only writing to standard output. + + - Added support for Landlock ABI version 4. + + * CMake: + + - Default to -O2 instead of -O3 with CMAKE_BUILD_TYPE=Release. + -O3 is not useful for speed and makes the code larger. + + - Now builds lzmainfo and lzmadec. + + - xzdiff, xzgrep, xzless, xzmore, and their symlinks are now + installed. The scripts are also tested during "make test". + + - Added translation support for xz, lzmainfo, and the + man pages. + + - Applied the symbol versioning workaround for MicroBlaze that + is used in the Autotools build. + + - The general XZ Utils and liblzma API documentation is now + installed. + + - The CMake component names were changed a little and several + were added. liblzma_Runtime and liblzma_Development are + unchanged. + + - Minimum required CMake version is now 3.14. However, + translation support is disabled with CMake versions + older than 3.20. + + - The CMake-based build is now close to feature parity with the + Autotools-based build. Most importantly a few tests aren't + run yet. Testing the CMake-based build on different operating + systems would be welcome now. See the comment at the top of + CMakeLists.txt. + + * Fixed a bug in the Autotools feature test for ARM64 CRC32 + instruction support for old versions of Clang. This did not + affect the CMake build. + + * Windows: + + - The build instructions in INSTALL and windows/INSTALL*.txt + were revised completely. + + - windows/build-with-cmake.bat along with the instructions + in windows/INSTALL-MinGW-w64_with_CMake.txt should make + it very easy to build liblzma.dll and xz.exe on Windows + using CMake and MinGW-w64 with either GCC or Clang/LLVM. + + - windows/build.bash was updated. It now works on MSYS2 and + on GNU/Linux (cross-compiling) to create a .zip and .7z + package for 32-bit and 64-bit x86 using GCC + MinGW-w64. + + * The TODO file is no longer installed as part of the + documentation. The file is out of date and does not reflect + the actual tasks that will be completed in the future. + + * Translations: + + - Translated lzmainfo man pages are now installed. These + had been forgotten in earlier versions. + + - Updated Croatian, Esperanto, German, Hungarian, Korean, + Polish, Romanian, Spanish, Swedish, Vietnamese, and Ukrainian + translations. + + - Updated German, Korean, Romanian, and Ukrainian man page + translations. + + * Added a few tests. + + Summary of new features added in the 5.5.x development releases: + + * liblzma: + + - LZMA decoder: Speed optimizations to the C code and + added GCC & Clang compatible inline assembly for x86-64. + + - Added lzma_mt_block_size() to recommend a Block size for + multithreaded encoding. + + - Added CLMUL-based CRC32 on x86-64 and E2K with runtime + processor detection. Similar to CRC64, on 32-bit x86 it + isn't available unless --disable-assembler is used. + + - Optimized the CRC32 calculation on ARM64 platforms using the + CRC32 instructions. Runtime detection for the instruction is + used on GNU/Linux, FreeBSD, Windows, and macOS. If the + compiler flags indicate unconditional CRC32 instruction + support (+crc) then the generic version is not built. + + - Added definitions of mask values like + LZMA_INDEX_CHECK_MASK_CRC32 to . + + * xz: + + - Multithreaded mode is now the default. This improves + compression speed and creates .xz files that can be + decompressed in multithreaded mode. The downsides are + increased memory usage and slightly worse compression ratio. + + - Added a new command line option --filters to set the filter + chain using the liblzma filter string syntax. + + - Added new command line options --filters1 ... --filters9 to + set additional filter chains using the liblzma filter string + syntax. The --block-list option now allows specifying filter + chains that were set using these new options. + + - Ported the command line tools to Windows MSVC. + Visual Studio 2015 or later is required. + + * Added lz4 support to xzdiff/xzcmp and xzgrep. + + +5.5.2beta (2024-02-14) + + * Licensing change: The core components are now under the + BSD Zero Clause License (0BSD). In XZ Utils 5.4.6 and older + and 5.5.1alpha these components are in the public domain and + obviously remain so; the change affects the new releases only. + + 0BSD is an extremely permissive license which doesn't require + retaining or reproducing copyright or license notices when + distributing the code, thus in practice there is extremely + little difference to public domain. + + * liblzma + + - Significant speed optimizations to the LZMA decoder were + made. There are now three variants that can be chosen at + build time: + + * Basic C version: This is a few percent faster than + 5.4.x due to some new optimizations. + + * Branchless C: This is currently the default on platforms + for which there is no assembly code. This should be a few + percent faster than the basic C version. + + * x86-64 inline assembly. This works with GCC and Clang. + + The default choice can currently be overridden by setting + LZMA_RANGE_DECODER_CONFIG in CPPFLAGS: 0 means the basic + version and 3 means that branchless C version. + + - Optimized the CRC32 calculation on ARM64 platforms using the + CRC32 instructions. The instructions are optional in ARMv8.0 + and are required in ARMv8.1 and later. Runtime detection for + the instruction is used on GNU/Linux, FreeBSD, Windows, and + macOS. If the compiler flags indicate unconditional CRC32 + instruction support (+crc) then the generic version is not + built. + + * Added lz4 support to xzdiff/xzcmp and xzgrep. + + * Man pages of xzdiff/xzcmp, xzgrep, and xzmore were rewritten + to simplify licensing of the man page translations. + + * Translations: + + - Updated Chinese (simplified), German, Korean, Polish, + Romanian, Spanish, Swedish, and Ukrainian translations. + + - Updated German, Korean, Romanian, and Ukrainian man page + translations. + + * Small improvements to the tests. + + * Added doc/examples/11_file_info.c. It was added to the Git + repository in 2017 but forgotten to be added into distribution + tarballs. + + * Removed doc/examples_old. These were from 2012. + + * Removed the macos/build.sh script. It had not been updated + since 2013. + + +5.5.1alpha (2024-01-26) + + * Added a new filter for RISC-V binaries. The filter can be used + for 32-bit and 64-bit binaries with either little or big + endianness. In liblzma, the Filter ID is LZMA_FILTER_RISCV (0x0B) + and the xz option is --riscv. liblzma filter string syntax + recognizes this filter as "riscv". + + * liblzma: + + - Added lzma_mt_block_size() to recommend a Block size for + multithreaded encoding + + - Added CLMUL-based CRC32 on x86-64 and E2K with runtime + processor detection. Similar to CRC64, on 32-bit x86 it + isn't available unless --disable-assembler is used. + + - Implemented GNU indirect function (IFUNC) as a runtime + function dispatching method for CRC32 and CRC64 fast + implementations on x86. Only GNU/Linux (glibc) and FreeBSD + builds will use IFUNC, unless --enable-ifunc is specified to + configure. + + - Added definitions of mask values like + LZMA_INDEX_CHECK_MASK_CRC32 to . + + - The XZ logo is now included in the Doxygen generated + documentation. It is licensed under Creative Commons + Attribution-ShareAlike 4.0. + + * xz: + + - Multithreaded mode is now the default. This improves + compression speed and creates .xz files that can be + decompressed multithreaded at the cost of increased memory + usage and slightly worse compression ratio. + + - Added new command line option --filters to set the filter + chain using liblzma filter string syntax. + + - Added new command line options --filters1 ... --filters9 to + set additional filter chains using liblzma filter string + syntax. The --block-list option now allows specifying filter + chains that were set using these new options. + + - Added support for Linux Landlock as a sandboxing method. + + - xzdec now supports pledge(2), Capsicum, and Linux Landlock as + sandboxing methods. + + - Progress indicator time stats remain accurate after pausing + xz with SIGTSTP. + + - Ported xz and xzdec to Windows MSVC. Visual Studio 2015 or + later is required. + + * CMake Build: + + - Supports pledge(2), Capsicum, and Linux Landlock sandboxing + methods. + + - Replacement functions for getopt_long() are used on platforms + that do not have it. + + * Enabled unaligned access by default on PowerPC64LE and on RISC-V + targets that define __riscv_misaligned_fast. + + * Tests: + + - Added two new fuzz targets to OSS-Fuzz. + + - Implemented Continuous Integration (CI) testing using + GitHub Actions. + + * Changed quoting style from `...' to '...' in all messages, + scripts, and documentation. + + * Added basic Codespell support to help catch typo errors. + + +5.4.7 (2024-05-29) + + * Not changed: Memory sanitizer (MSAN) has a false positive + in the CRC CLMUL code which also makes OSS Fuzz unhappy. + Valgrind is smarter and doesn't complain. + + A revision to the CLMUL code is coming anyway and this issue + will be cleaned up as part of it. It won't be backported to + 5.6.x or 5.4.x because the old code isn't wrong. There is + no reason to risk introducing regressions in old branches + just to silence a false positive. + + * liblzma: + + - lzma_index_decoder() and lzma_index_buffer_decode(): Fix + a missing output pointer initialization (*i = NULL) if the + functions are called with invalid arguments. The API docs + say that such an initialization is always done. In practice + this matters very little because the problem can only occur + if the calling application has a bug and these functions + return LZMA_PROG_ERROR. + + - lzma_str_to_filters(): Fix a missing output pointer + initialization (*error_pos = 0). This is very similar + to the fix above. + + - Fix C standard conformance with function pointer types. + This newly showed up with Clang 17 with -fsanitize=undefined. + There are no bug reports about this. + + - Fix building with NVIDIA HPC SDK. + + * xz: + + - Fix a C standard conformance issue in --block-list parsing + (arithmetic on a null pointer). + + - Fix a warning from GNU groff when processing the man page: + "warning: cannot select font 'CW'" + + - Fix outdated threading related information on the man page. + + * xzless: + + - With "less" version 451 and later, use "||-" instead of "|-" + in the environment variable LESSOPEN. This way compressed + files that contain no uncompressed data are shown correctly + as empty. + + - With "less" version 632 and later, use --show-preproc-errors + to make "less" show a warning on decompression errors. + + * Autotools-based build system (configure): + + - Symbol versioning variant can now be overridden with + --enable-symbol-versions. Documentation in INSTALL was + updated to match. + + CMake: + + - Linux on MicroBlaze is handled specially now. This matches + the changes made to the Autotools-based build in XZ Utils + 5.4.2 and 5.2.11. + + - Disable symbol versioning on non-glibc Linux to match what + the Autotools build does. For example, symbol versioning + isn't enabled with musl. + + - Symbol versioning variant can now be overridden by setting + SYMBOL_VERSIONING to "OFF", "generic", or "linux". + + * Documentation: + + - Clarify the description of --disable-assembler in INSTALL. + The option only affects 32-bit x86 assembly usage. + + - Add doc/examples/11_file_info.c. It was added to the + Git repository in 2017 but forgotten to be added into + distribution tarballs. + + - Don't install the TODO file as part of the documentation. + The file is out of date. + + - Update home page URLs back to their old locations on + tukaani.org. + + - Update maintainer info. + + +5.4.6 (2024-01-26) + + * Fixed a bug involving internal function pointers in liblzma not + being initialized to NULL. The bug can only be triggered if + lzma_filters_update() is called on a LZMA1 encoder, so it does + not affect xz or any application known to us that uses liblzma. + + * xz: + + - Fixed a regression introduced in 5.4.2 that caused encoding + in the raw format to unnecessarily fail if --suffix was not + used. For instance, the following command no longer reports + that --suffix must be used: + + echo foo | xz --format=raw --lzma2 | wc -c + + - Fixed an issue on MinGW-w64 builds that prevented reading + from or writing to non-terminal character devices like NUL. + + * Added a new test. + + +5.4.5 (2023-11-01) + + * liblzma: + + - Use __attribute__((__no_sanitize_address__)) to avoid address + sanitization with CRC64 CLMUL. It uses 16-byte-aligned reads + which can extend past the bounds of the input buffer and + inherently trigger address sanitization errors. This isn't + a bug. + + - Fixed an assertion failure that could be triggered by a large + unpadded_size argument. It was verified that there was no + other bug than the assertion failure. + + - Fixed a bug that prevented building with Windows Vista + threading when __attribute__((__constructor__)) is not + supported. + + * xz now properly handles special files such as "con" or "nul" on + Windows. Before this fix, the following wrote "foo" to the + console and deleted the input file "con_xz": + + echo foo | xz > con_xz + xz --suffix=_xz --decompress con_xz + + * Build systems: + + - Allow builds with Windows win95 threading and small mode when + __attribute__((__constructor__)) is supported. + + - Added a new line to liblzma.pc for MSYS2 (Windows): + + Cflags.private: -DLZMA_API_STATIC + + When compiling code that will link against static liblzma, + the LZMA_API_STATIC macro needs to be defined on Windows. + + - CMake specific changes: + + * Fixed a bug that allowed CLOCK_MONOTONIC to be used even + if the check for it failed. + + * Fixed a bug where configuring CMake multiple times + resulted in HAVE_CLOCK_GETTIME and HAVE_CLOCK_MONOTONIC + not being set. + + * Fixed the build with MinGW-w64-based Clang/LLVM 17. + llvm-windres now has more accurate GNU windres emulation + so the GNU windres workaround from 5.4.1 is needed with + llvm-windres version 17 too. + + * The import library on Windows is now properly named + "liblzma.dll.a" instead of "libliblzma.dll.a" + + * Fixed a bug causing the Ninja Generator to fail on + UNIX-like systems. This bug was introduced in 5.4.0. + + * Added a new option to disable CLMUL CRC64. + + * A module-definition (.def) file is now created when + building liblzma.dll with MinGW-w64. + + * The pkg-config liblzma.pc file is now installed on all + builds except when using MSVC on Windows. + + * Added large file support by default for platforms that + need it to handle files larger than 2 GiB. This includes + MinGW-w64, even 64-bit builds. + + * Small fixes and improvements to the tests. + + * Updated translations: Chinese (simplified) and Esperanto. + + +5.4.4 (2023-08-02) + + * liblzma and xzdec can now build against WASI SDK when threading + support is disabled. xz and tests don't build yet. + + * CMake: + + - Fixed a bug preventing other projects from including liblzma + multiple times using find_package(). + + - Don't create broken symlinks in Cygwin and MSYS2 unless + supported by the environment. This prevented building for the + default MSYS2 environment. The problem was introduced in + xz 5.4.0. + + * Documentation: + + - Small improvements to man pages. + + - Small improvements and typo fixes for liblzma API + documentation. + + * Tests: + + - Added a new section to INSTALL to describe basic test usage + and address recent questions about building the tests when + cross compiling. + + - Small fixes and improvements to the tests. + + * Translations: + + - Fixed a mistake that caused one of the error messages to not + be translated. This only affected versions 5.4.2 and 5.4.3. + + - Updated the Chinese (simplified), Croatian, Esperanto, German, + Korean, Polish, Romanian, Spanish, Swedish, Ukrainian, and + Vietnamese translations. + + - Updated the German, Korean, Romanian, and Ukrainian man page + translations. + + +5.4.3 (2023-05-04) + + * All fixes from 5.2.12 + + * Features in the CMake build can now be disabled as CMake cache + variables, similar to the Autotools build. + + * Minor update to the Croatian translation. + + +5.4.2 (2023-03-18) + + * All fixes from 5.2.11 that were not included in 5.4.1. + + * If xz is built with support for the Capsicum sandbox but running + in an environment that doesn't support Capsicum, xz now runs + normally without sandboxing instead of exiting with an error. + + * liblzma: + + - Documentation was updated to improve the style, consistency, + and completeness of the liblzma API headers. + + - The Doxygen-generated HTML documentation for the liblzma API + header files is now included in the source release and is + installed as part of "make install". All JavaScript is + removed to simplify license compliance and to reduce the + install size. + + - Fixed a minor bug in lzma_str_from_filters() that produced + too many filters in the output string instead of reporting + an error if the input array had more than four filters. This + bug did not affect xz. + + * Build systems: + + - autogen.sh now invokes the doxygen tool via the new wrapper + script doxygen/update-doxygen, unless the command line option + --no-doxygen is used. + + - Added microlzma_encoder.c and microlzma_decoder.c to the + VS project files for Windows and to the CMake build. These + should have been included in 5.3.2alpha. + + * Tests: + + - Added a test to the CMake build that was forgotten in the + previous release. + + - Added and refactored a few tests. + + * Translations: + + - Updated the Brazilian Portuguese translation. + + - Added Brazilian Portuguese man page translation. + + +5.4.1 (2023-01-11) + + * liblzma: + + - Fixed the return value of lzma_microlzma_encoder() if the + LZMA options lc/lp/pb are invalid. Invalid lc/lp/pb options + made the function return LZMA_STREAM_END without encoding + anything instead of returning LZMA_OPTIONS_ERROR. + + - Windows / Visual Studio: Workaround a possible compiler bug + when targeting 32-bit x86 and compiling the CLMUL version of + the CRC64 code. The CLMUL code isn't enabled by the Windows + project files but it is in the CMake-based builds. + + * Build systems: + + - Windows-specific CMake changes: + + * Don't try to enable CLMUL CRC64 code if _mm_set_epi64x() + isn't available. This fixes CMake-based build with Visual + Studio 2013. + + * Created a workaround for a build failure with windres + from GNU binutils. It is used only when the C compiler + is GCC (not Clang). The workaround is incompatible + with llvm-windres, resulting in "XZx20Utils" instead + of "XZ Utils" in the resource file, but without the + workaround llvm-windres works correctly. See the + comment in CMakeLists.txt for details. + + * Included the resource files in the xz and xzdec build + rules. Building the command line tools is still + experimental but possible with MinGW-w64. + + - Visual Studio: Added stream_decoder_mt.c to the project + files. Now the threaded decompressor lzma_stream_decoder_mt() + gets built. CMake-based build wasn't affected. + + - Updated windows/INSTALL-MSVC.txt to mention that CMake-based + build is now the preferred method with Visual Studio. The + project files will probably be removed after 5.4.x releases. + + - Changes to #defines in config.h: + + * HAVE_DECL_CLOCK_MONOTONIC was replaced by + HAVE_CLOCK_MONOTONIC. The old macro was always defined + in configure-generated config.h to either 0 or 1. The + new macro is defined (to 1) only if the declaration of + CLOCK_MONOTONIC is available. This matches the way most + other config.h macros work and makes things simpler with + other build systems. + + * HAVE_DECL_PROGRAM_INVOCATION_NAME was replaced by + HAVE_PROGRAM_INVOCATION_NAME for the same reason. + + * Tests: + + - Fixed test script compatibility with ancient /bin/sh + versions. Now the five test_compress_* tests should + no longer fail on Solaris 10. + + - Added and refactored a few tests. + + * Translations: + + - Updated the Catalan and Esperanto translations. + + - Added Korean and Ukrainian man page translations. + + +5.4.0 (2022-12-13) + + This bumps the minor version of liblzma because new features were + added. The API and ABI are still backward compatible with liblzma + 5.2.x and 5.0.x. + + Since 5.3.5beta: + + * All fixes from 5.2.10. + + * The ARM64 filter is now stable. The xz option is now --arm64. + Decompression requires XZ Utils 5.4.0. In the future the ARM64 + filter will be supported by XZ for Java, XZ Embedded (including + the version in Linux), LZMA SDK, and 7-Zip. + + * Translations: + + - Updated Catalan, Croatian, German, Romanian, and Turkish + translations. + + - Updated German man page translations. + + - Added Romanian man page translations. + + Summary of new features added in the 5.3.x development releases: + + * liblzma: + + - Added threaded .xz decompressor lzma_stream_decoder_mt(). + It can use multiple threads with .xz files that have multiple + Blocks with size information in Block Headers. The threaded + encoder in xz has always created such files. + + Single-threaded encoder cannot store the size information in + Block Headers even if one used LZMA_FULL_FLUSH to create + multiple Blocks, so this threaded decoder cannot use multiple + threads with such files. + + If there are multiple Streams (concatenated .xz files), one + Stream will be decompressed completely before starting the + next Stream. + + - A new decoder flag LZMA_FAIL_FAST was added. It makes the + threaded decompressor report errors soon instead of first + flushing all pending data before the error location. + + - New Filter IDs: + * LZMA_FILTER_ARM64 is for ARM64 binaries. + * LZMA_FILTER_LZMA1EXT is for raw LZMA1 streams that don't + necessarily use the end marker. + + - Added lzma_str_to_filters(), lzma_str_from_filters(), and + lzma_str_list_filters() to convert a preset or a filter chain + string to a lzma_filter[] and vice versa. These should make + it easier to write applications that allow users to specify + custom compression options. + + - Added lzma_filters_free() which can be convenient for freeing + the filter options in a filter chain (an array of lzma_filter + structures). + + - lzma_file_info_decoder() to makes it a little easier to get + the Index field from .xz files. This helps in getting the + uncompressed file size but an easy-to-use random access + API is still missing which has existed in XZ for Java for + a long time. + + - Added lzma_microlzma_encoder() and lzma_microlzma_decoder(). + It is used by erofs-utils and may be used by others too. + + The MicroLZMA format is a raw LZMA stream (without end marker) + whose first byte (always 0x00) has been replaced with + bitwise-negation of the LZMA properties (lc/lp/pb). It was + created for use in EROFS but may be used in other contexts + as well where it is important to avoid wasting bytes for + stream headers or footers. The format is also supported by + XZ Embedded (the XZ Embedded version in Linux got MicroLZMA + support in Linux 5.16). + + The MicroLZMA encoder API in liblzma can compress into a + fixed-sized output buffer so that as much data is compressed + as can be fit into the buffer while still creating a valid + MicroLZMA stream. This is needed for EROFS. + + - Added lzma_lzip_decoder() to decompress the .lz (lzip) file + format version 0 and the original unextended version 1 files. + Also lzma_auto_decoder() supports .lz files. + + - lzma_filters_update() can now be used with the multi-threaded + encoder (lzma_stream_encoder_mt()) to change the filter chain + after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH. + + - In lzma_options_lzma, allow nice_len = 2 and 3 with the match + finders that require at least 3 or 4. Now it is internally + rounded up if needed. + + - CLMUL-based CRC64 on x86-64 and E2K with runtime processor + detection. On 32-bit x86 it currently isn't available unless + --disable-assembler is used which can make the non-CLMUL + CRC64 slower; this might be fixed in the future. + + - Building with --disable-threads --enable-small + is now thread-safe if the compiler supports + __attribute__((__constructor__)). + + * xz: + + - Using -T0 (--threads=0) will now use multi-threaded encoder + even on a single-core system. This is to ensure that output + from the same xz binary is identical on both single-core and + multi-core systems. + + - --threads=+1 or -T+1 is now a way to put xz into + multi-threaded mode while using only one worker thread. + The + is ignored if the number is not 1. + + - A default soft memory usage limit is now used for compression + when -T0 is used and no explicit limit has been specified. + This soft limit is used to restrict the number of threads + but if the limit is exceeded with even one thread then xz + will continue with one thread using the multi-threaded + encoder and this limit is ignored. If the number of threads + is specified manually then no default limit will be used; + this affects only -T0. + + This change helps on systems that have very many cores and + using all of them for xz makes no sense. Previously xz -T0 + could run out of memory on such systems because it attempted + to reserve memory for too many threads. + + This also helps with 32-bit builds which don't have a large + amount of address space that would be required for many + threads. The default soft limit for -T0 is at most 1400 MiB + on all 32-bit platforms. + + - Previously a low value in --memlimit-compress wouldn't cause + xz to switch from multi-threaded mode to single-threaded mode + if the limit cannot otherwise be met; xz failed instead. Now + xz can switch to single-threaded mode and then, if needed, + scale down the LZMA2 dictionary size too just like it already + did when it was started in single-threaded mode. + + - The option --no-adjust no longer prevents xz from scaling down + the number of threads as that doesn't affect the compressed + output (only performance). Now --no-adjust only prevents + adjustments that affect compressed output, that is, with + --no-adjust xz won't switch from multi-threaded mode to + single-threaded mode and won't scale down the LZMA2 + dictionary size. + + - Added a new option --memlimit-mt-decompress=LIMIT. This is + used to limit the number of decompressor threads (possibly + falling back to single-threaded mode) but it will never make + xz refuse to decompress a file. This has a system-specific + default value because without any limit xz could end up + allocating memory for the whole compressed input file, the + whole uncompressed output file, multiple thread-specific + decompressor instances and so on. Basically xz could + attempt to use an insane amount of memory even with fairly + common files. The system-specific default value is currently + the same as the one used for compression with -T0. + + The new option works together with the existing option + --memlimit-decompress=LIMIT. The old option sets a hard limit + that must not be exceeded (xz will refuse to decompress) + while the new option only restricts the number of threads. + If the limit set with --memlimit-mt-decompress is greater + than the limit set with --memlimit-compress, then the latter + value is used also for --memlimit-mt-decompress. + + - Added new information to the output of xz --info-memory and + new fields to the output of xz --robot --info-memory. + + - In --lzma2=nice=NUMBER allow 2 and 3 with all match finders + now that liblzma handles it. + + - Don't mention endianness for ARM and ARM-Thumb filters in + --long-help. The filters only work for little endian + instruction encoding but modern ARM processors using + big endian data access still use little endian + instruction encoding. So the help text was misleading. + In contrast, the PowerPC filter is only for big endian + 32/64-bit PowerPC code. Little endian PowerPC would need + a separate filter. + + - Added decompression support for the .lz (lzip) file format + version 0 and the original unextended version 1. It is + autodetected by default. See also the option --format on + the xz man page. + + - Sandboxing enabled by default: + * Capsicum (FreeBSD) + * pledge(2) (OpenBSD) + + * Scripts now support the .lz format using xz. + + * A few new tests were added. + + * The liblzma-specific tests are now supported in CMake-based + builds too ("make test"). + + +5.3.5beta (2022-12-01) + + * All fixes from 5.2.9. + + * liblzma: + + - Added new LZMA_FILTER_LZMA1EXT for raw encoder and decoder to + handle raw LZMA1 streams that don't have end of payload marker + (EOPM) alias end of stream (EOS) marker. It can be used in + filter chains, for example, with the x86 BCJ filter. + + - Added lzma_str_to_filters(), lzma_str_from_filters(), and + lzma_str_list_filters() to make it easier for applications + to get custom compression options from a user and convert + it to an array of lzma_filter structures. + + - Added lzma_filters_free(). + + - lzma_filters_update() can now be used with the multi-threaded + encoder (lzma_stream_encoder_mt()) to change the filter chain + after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH. + + - In lzma_options_lzma, allow nice_len = 2 and 3 with the match + finders that require at least 3 or 4. Now it is internally + rounded up if needed. + + - ARM64 filter was modified. It is still experimental. + + - Fixed LTO build with Clang if -fgnuc-version=10 or similar + was used to make Clang look like GCC >= 10. Now it uses + __has_attribute(__symver__) which should be reliable. + + * xz: + + - --threads=+1 or -T+1 is now a way to put xz into multi-threaded + mode while using only one worker thread. + + - In --lzma2=nice=NUMBER allow 2 and 3 with all match finders + now that liblzma handles it. + + * Updated translations: Chinese (simplified), Korean, and Turkish. + + +5.3.4alpha (2022-11-15) + + * All fixes from 5.2.7 and 5.2.8. + + * liblzma: + + - Minor improvements to the threaded decoder. + + - Added CRC64 implementation that uses SSSE3, SSE4.1, and CLMUL + instructions on 32/64-bit x86 and E2K. On 32-bit x86 it's + not enabled unless --disable-assembler is used but then + the non-CLMUL code might be slower. Processor support is + detected at runtime so this is built by default on x86-64 + and E2K. On these platforms, if compiler flags indicate + unconditional CLMUL support (-msse4.1 -mpclmul) then the + generic version is not built, making liblzma 8-9 KiB smaller + compared to having both versions included. + + With extremely compressible files this can make decompression + up to twice as fast but with typical files 5 % improvement + is a more realistic expectation. + + The CLMUL version is slower than the generic version with + tiny inputs (especially at 1-8 bytes per call, but up to + 16 bytes). In normal use in xz this doesn't matter at all. + + - Added an experimental ARM64 filter. This is *not* the final + version! Files created with this experimental version won't + be supported in the future versions! The filter design is + a compromise where improving one use case makes some other + cases worse. + + - Added decompression support for the .lz (lzip) file format + version 0 and the original unextended version 1. See the + API docs of lzma_lzip_decoder() for details. Also + lzma_auto_decoder() supports .lz files. + + - Building with --disable-threads --enable-small + is now thread-safe if the compiler supports + __attribute__((__constructor__)) + + * xz: + + - Added support for OpenBSD's pledge(2) as a sandboxing method. + + - Don't mention endianness for ARM and ARM-Thumb filters in + --long-help. The filters only work for little endian + instruction encoding but modern ARM processors using + big endian data access still use little endian + instruction encoding. So the help text was misleading. + In contrast, the PowerPC filter is only for big endian + 32/64-bit PowerPC code. Little endian PowerPC would need + a separate filter. + + - Added --experimental-arm64. This will be renamed once the + filter is finished. Files created with this experimental + filter will not be supported in the future! + + - Added new fields to the output of xz --robot --info-memory. + + - Added decompression support for the .lz (lzip) file format + version 0 and the original unextended version 1. It is + autodetected by default. See also the option --format on + the xz man page. + + * Scripts now support the .lz format using xz. + + * Build systems: + + - New #defines in config.h: HAVE_ENCODER_ARM64, + HAVE_DECODER_ARM64, HAVE_LZIP_DECODER, HAVE_CPUID_H, + HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR, HAVE_USABLE_CLMUL + + - New configure options: --disable-clmul-crc, + --disable-microlzma, --disable-lzip-decoder, and + 'pledge' is now an option in --enable-sandbox (but + it's autodetected by default anyway). + + - INSTALL was updated to document the new configure options. + + - PACKAGERS now lists also --disable-microlzma and + --disable-lzip-decoder as configure options that must + not be used in builds for non-embedded use. + + * Tests: + + - Fix some of the tests so that they skip instead of fail if + certain features have been disabled with configure options. + It's still not perfect. + + - Other improvements to tests. + + * Updated translations: Croatian, Finnish, Hungarian, Polish, + Romanian, Spanish, Swedish, and Ukrainian. + + +5.3.3alpha (2022-08-22) + + * All fixes from 5.2.6. + + * liblzma: + + - Fixed 32-bit build. + + - Added threaded .xz decompressor lzma_stream_decoder_mt(). + It can use multiple threads with .xz files that have multiple + Blocks with size information in Block Headers. The threaded + encoder in xz has always created such files. + + Single-threaded encoder cannot store the size information in + Block Headers even if one used LZMA_FULL_FLUSH to create + multiple Blocks, so this threaded decoder cannot use multiple + threads with such files. + + If there are multiple Streams (concatenated .xz files), one + Stream will be decompressed completely before starting the + next Stream. + + - A new decoder flag LZMA_FAIL_FAST was added. It makes the + threaded decompressor report errors soon instead of first + flushing all pending data before the error location. + + * xz: + + - Using -T0 (--threads=0) will now use multi-threaded encoder + even on a single-core system. This is to ensure that output + from the same xz binary is identical on both single-core and + multi-core systems. + + - A default soft memory usage limit is now used for compression + when -T0 is used and no explicit limit has been specified. + This soft limit is used to restrict the number of threads + but if the limit is exceeded with even one thread then xz + will continue with one thread using the multi-threaded + encoder and this limit is ignored. If the number of threads + is specified manually then no default limit will be used; + this affects only -T0. + + This change helps on systems that have very many cores and + using all of them for xz makes no sense. Previously xz -T0 + could run out of memory on such systems because it attempted + to reserve memory for too many threads. + + This also helps with 32-bit builds which don't have a large + amount of address space that would be required for many + threads. The default limit is 1400 MiB on all 32-bit + platforms with -T0. + + Now xz -T0 should just work. It might use too few threads + in some cases but at least it shouldn't easily run out of + memory. It's possible that this will be tweaked before 5.4.0. + + - Changes to --memlimit-compress and --no-adjust: + + In single-threaded mode, --memlimit-compress can make xz + scale down the LZMA2 dictionary size to meet the memory usage + limit. This obviously affects the compressed output. However, + if xz was in threaded mode, --memlimit-compress could make xz + reduce the number of threads but it wouldn't make xz switch + from multi-threaded mode to single-threaded mode or scale + down the LZMA2 dictionary size. This seemed illogical. + + Now --memlimit-compress can make xz switch to single-threaded + mode if one thread in multi-threaded mode uses too much + memory. If memory usage is still too high, then the LZMA2 + dictionary size can be scaled down too. + + The option --no-adjust was also changed so that it no longer + prevents xz from scaling down the number of threads as that + doesn't affect compressed output (only performance). After + this commit --no-adjust only prevents adjustments that affect + compressed output, that is, with --no-adjust xz won't switch + from multithreaded mode to single-threaded mode and won't + scale down the LZMA2 dictionary size. + + - Added a new option --memlimit-mt-decompress=LIMIT. This is + used to limit the number of decompressor threads (possibly + falling back to single-threaded mode) but it will never make + xz refuse to decompress a file. This has a system-specific + default value because without any limit xz could end up + allocating memory for the whole compressed input file, the + whole uncompressed output file, multiple thread-specific + decompressor instances and so on. Basically xz could + attempt to use an insane amount of memory even with fairly + common files. + + The new option works together with the existing option + --memlimit-decompress=LIMIT. The old option sets a hard limit + that must not be exceeded (xz will refuse to decompress) + while the new option only restricts the number of threads. + If the limit set with --memlimit-mt-decompress is greater + than the limit set with --memlimit-compress, then the latter + value is used also for --memlimit-mt-decompress. + + * Tests: + + - Added a few more tests. + + - Added tests/code_coverage.sh to create a code coverage report + of the tests. + + * Build systems: + + - Automake's parallel test harness is now used to make tests + finish faster. + + - Added the CMake files to the distribution tarball. These were + supposed to be in 5.2.5 already. + + - Added liblzma tests to the CMake build. + + - Windows: Fix building of liblzma.dll with the included + Visual Studio project files. + + +5.3.2alpha (2021-10-28) + + This release was made on short notice so that recent erofs-utils can + be built with LZMA support without needing a snapshot from xz.git. + Thus many pending things were not included, not even updated + translations (which would need to be updated for the new --list + strings anyway). + + * All fixes from 5.2.5. + + * xz: + + - When copying metadata from the source file to the destination + file, don't try to set the group (GID) if it is already set + correctly. This avoids a failure on OpenBSD (and possibly on + a few other OSes) where files may get created so that their + group doesn't belong to the user, and fchown(2) can fail even + if it needs to do nothing. + + - The --keep option now accepts symlinks, hardlinks, and + setuid, setgid, and sticky files. Previously this required + using --force. + + - Split the long strings used in --list and --info-memory modes + to make them much easier for translators. + + - If built with sandbox support and enabling the sandbox fails, + xz will now immediately exit with exit status of 1. Previously + it would only display a warning if -vv was used. + + - Cap --memlimit-compress to 2000 MiB on MIPS32 because on + MIPS32 userspace processes are limited to 2 GiB of address + space. + + * liblzma: + + - Added lzma_microlzma_encoder() and lzma_microlzma_decoder(). + The API is in lzma/container.h. + + The MicroLZMA format is a raw LZMA stream (without end marker) + whose first byte (always 0x00) has been replaced with + bitwise-negation of the LZMA properties (lc/lp/pb). It was + created for use in EROFS but may be used in other contexts + as well where it is important to avoid wasting bytes for + stream headers or footers. The format is also supported by + XZ Embedded. + + The MicroLZMA encoder API in liblzma can compress into a + fixed-sized output buffer so that as much data is compressed + as can be fit into the buffer while still creating a valid + MicroLZMA stream. This is needed for EROFS. + + - Added fuzzing support. + + - Support Intel Control-flow Enforcement Technology (CET) in + 32-bit x86 assembly files. + + - Visual Studio: Use non-standard _MSVC_LANG to detect C++ + standard version in the lzma.h API header. It's used to + detect when "noexcept" can be used. + + * Scripts: + + - Fix exit status of xzdiff/xzcmp. Exit status could be 2 when + the correct value is 1. + + - Fix exit status of xzgrep. + + - Detect corrupt .bz2 files in xzgrep. + + - Add zstd support to xzgrep and xzdiff/xzcmp. + + - Fix less(1) version detection in xzless. It failed if the + version number from "less -V" contained a dot. + + * Fix typos and technical issues in man pages. + + * Build systems: + + - Windows: Fix building of resource files when config.h isn't + used. CMake + Visual Studio can now build liblzma.dll. + + - Various fixes to the CMake support. It might still need a few + more fixes even for liblzma-only builds. + + +5.3.1alpha (2018-04-29) + + * All fixes from 5.2.4. + + * Add lzma_file_info_decoder() into liblzma and use it in xz to + implement the --list feature. + + * Capsicum sandbox support is enabled by default where available + (FreeBSD >= 10). + + +5.2.13 (2024-05-29) + + * liblzma: + + - lzma_index_append(): Fix an assertion failure that could be + triggered by a large unpadded_size argument. It was verified + that there was no other bug than the assertion failure. + + - lzma_index_decoder() and lzma_index_buffer_decode(): Fix + a missing output pointer initialization (*i = NULL) if the + functions are called with invalid arguments. The API docs + say that such an initialization is always done. In practice + this matters very little because the problem can only occur + if the calling application has a bug and these functions + return LZMA_PROG_ERROR. + + - Fix C standard conformance with function pointer types. + This newly showed up with Clang 17 with -fsanitize=undefined. + There are no bug reports about this. + + - Fix building with NVIDIA HPC SDK. + + - Fix building with Windows Vista threads and --enable-small. + (CMake build doesn't support ENABLE_SMALL in XZ Utils 5.2.x.) + + * xz: + + - Fix a C standard conformance issue in --block-list parsing + (arithmetic on a null pointer). + + - Fix a warning from GNU groff when processing the man page: + "warning: cannot select font 'CW'" + + - Windows: Handle special files such as "con" or "nul". Earlier + the following wrote "foo" to the console and deleted the input + file "con_xz": + + echo foo | xz > con_xz + xz --suffix=_xz --decompress con_xz + + - Windows: Fix an issue that prevented reading from or writing + to non-terminal character devices like NUL. + + * xzless: + + - With "less" version 451 and later, use "||-" instead of "|-" + in the environment variable LESSOPEN. This way compressed + files that contain no uncompressed data are shown correctly + as empty. + + - With "less" version 632 and later, use --show-preproc-errors + to make "less" show a warning on decompression errors. + + * Build systems: + + - Add a new line to liblzma.pc for MSYS2 (Windows): + + Cflags.private: -DLZMA_API_STATIC + + When compiling code that will link against static liblzma, + the LZMA_API_STATIC macro needs to be defined on Windows. + + - Autotools (configure): + + * Symbol versioning variant can now be overridden with + --enable-symbol-versions. Documentation in INSTALL was + updated to match. + + - CMake: + + * Fix a bug that prevented other projects from including + liblzma multiple times using find_package(). + + * Fix a bug where configuring CMake multiple times resulted + in HAVE_CLOCK_GETTIME and HAVE_CLOCK_MONOTONIC not being + defined. + + * Fix the build with MinGW-w64-based Clang/LLVM 17. + llvm-windres now has more accurate GNU windres emulation + so the GNU windres workaround from 5.4.1 is needed with + llvm-windres version 17 too. + + * The import library on Windows is now properly named + "liblzma.dll.a" instead of "libliblzma.dll.a" + + * Add large file support by default for platforms that + need it to handle files larger than 2 GiB. This includes + MinGW-w64, even 64-bit builds. + + * Linux on MicroBlaze is handled specially now. This + matches the changes made to the Autotools-based build + in XZ Utils 5.4.2 and 5.2.11. + + * Disable symbol versioning on non-glibc Linux to match + what the Autotools build does. For example, symbol + versioning isn't enabled with musl. + + * Symbol versioning variant can now be overridden by + setting SYMBOL_VERSIONING to "OFF", "generic", or + "linux". + + * Documentation: + + - Clarify the description of --disable-assembler in INSTALL. + The option only affects 32-bit x86 assembly usage. + + - Don't install the TODO file as part of the documentation. + The file is out of date. + + - Update home page URLs back to their old locations on + tukaani.org. + + - Update maintainer info. + + +5.2.12 (2023-05-04) + + * Fixed a build system bug that prevented building liblzma as a + shared library when configured with --disable-threads. This bug + affected releases 5.2.6 to 5.2.11 and 5.4.0 to 5.4.2. + + * Include for Windows intrinsic functions where they are + needed. This fixed a bug that prevented building liblzma using + clang-cl on Windows. + + * Minor update to the Croatian translation. The small change + applies to a string in both 5.2 and 5.4 branches. + + +5.2.11 (2023-03-18) + + * Removed all possible cases of null pointer + 0. It is undefined + behavior in C99 and C17. This was detected by a sanitizer and had + not caused any known issues. + + * Build systems: + + - Added a workaround for building with GCC on MicroBlaze Linux. + GCC 12 on MicroBlaze doesn't support the __symver__ attribute + even though __has_attribute(__symver__) returns true. The + build is now done without the extra RHEL/CentOS 7 symbols + that were added in XZ Utils 5.2.7. The workaround only + applies to the Autotools build (not CMake). + + - CMake: Ensure that the C compiler language is set to C99 or + a newer standard. + + - CMake changes from XZ Utils 5.4.1: + + * Added a workaround for a build failure with + windres from GNU binutils. + + * Included the Windows resource files in the xz + and xzdec build rules. + + +5.2.10 (2022-12-13) + + * xz: Don't modify argv[] when parsing the --memlimit* and + --block-list command line options. This fixes confusing + arguments in process listing (like "ps auxf"). + + * GNU/Linux only: Use __has_attribute(__symver__) to detect if + that attribute is supported. This fixes build on Mandriva where + Clang is patched to define __GNUC__ to 11 by default (instead + of 4 as used by Clang upstream). + + +5.2.9 (2022-11-30) + + * liblzma: + + - Fixed an infinite loop in LZMA encoder initialization + if dict_size >= 2 GiB. (The encoder only supports up + to 1536 MiB.) + + - Fixed two cases of invalid free() that can happen if + a tiny allocation fails in encoder re-initialization + or in lzma_filters_update(). These bugs had some + similarities with the bug fixed in 5.2.7. + + - Fixed lzma_block_encoder() not allowing the use of + LZMA_SYNC_FLUSH with lzma_code() even though it was + documented to be supported. The sync-flush code in + the Block encoder was already used internally via + lzma_stream_encoder(), so this was just a missing flag + in the lzma_block_encoder() API function. + + - GNU/Linux only: Don't put symbol versions into static + liblzma as it breaks things in some cases (and even if + it didn't break anything, symbol versions in static + libraries are useless anyway). The downside of the fix + is that if the configure options --with-pic or --without-pic + are used then it's not possible to build both shared and + static liblzma at the same time on GNU/Linux anymore; + with those options --disable-static or --disable-shared + must be used too. + + * New email address for bug reports is which + forwards messages to Lasse Collin and Jia Tan. + + +5.2.8 (2022-11-13) + + * xz: + + - If xz cannot remove an input file when it should, this + is now treated as a warning (exit status 2) instead of + an error (exit status 1). This matches GNU gzip and it + is more logical as at that point the output file has + already been successfully closed. + + - Fix handling of .xz files with an unsupported check type. + Previously such printed a warning message but then xz + behaved as if an error had occurred (didn't decompress, + exit status 1). Now a warning is printed, decompression + is done anyway, and exit status is 2. This used to work + slightly before 5.0.0. In practice this bug matters only + if xz has been built with some check types disabled. As + instructed in PACKAGERS, such builds should be done in + special situations only. + + - Fix "xz -dc --single-stream tests/files/good-0-empty.xz" + which failed with "Internal error (bug)". That is, + --single-stream was broken if the first .xz stream in + the input file didn't contain any uncompressed data. + + - Fix displaying file sizes in the progress indicator when + working in passthru mode and there are multiple input files. + Just like "gzip -cdf", "xz -cdf" works like "cat" when the + input file isn't a supported compressed file format. In + this case the file size counters weren't reset between + files so with multiple input files the progress indicator + displayed an incorrect (too large) value. + + * liblzma: + + - API docs in lzma/container.h: + * Update the list of decoder flags in the decoder + function docs. + * Explain LZMA_CONCATENATED behavior with .lzma files + in lzma_auto_decoder() docs. + + - OpenBSD: Use HW_NCPUONLINE to detect the number of + available hardware threads in lzma_physmem(). + + - Fix use of wrong macro to detect x86 SSE2 support. + __SSE2_MATH__ was used with GCC/Clang but the correct + one is __SSE2__. The first one means that SSE2 is used + for floating point math which is irrelevant here. + The affected SSE2 code isn't used on x86-64 so this affects + only 32-bit x86 builds that use -msse2 without -mfpmath=sse + (there is no runtime detection for SSE2). It improves LZMA + compression speed (not decompression). + + - Fix the build with Intel C compiler 2021 (ICC, not ICX) + on Linux. It defines __GNUC__ to 10 but doesn't support + the __symver__ attribute introduced in GCC 10. + + * Scripts: Ignore warnings from xz by using --quiet --no-warn. + This is needed if the input .xz files use an unsupported + check type. + + * Translations: + + - Updated Croatian and Turkish translations. + + - One new translations wasn't included because it needed + technical fixes. It will be in upcoming 5.4.0. No new + translations will be added to the 5.2.x branch anymore. + + - Renamed the French man page translation file from + fr_FR.po to fr.po and thus also its install directory + (like /usr/share/man/fr_FR -> .../fr). + + - Man page translations for upcoming 5.4.0 are now handled + in the Translation Project. + + * Update doc/faq.txt a little so it's less out-of-date. + + +5.2.7 (2022-09-30) + + * liblzma: + + - Made lzma_filters_copy() to never modify the destination + array if an error occurs. lzma_stream_encoder() and + lzma_stream_encoder_mt() already assumed this. Before this + change, if a tiny memory allocation in lzma_filters_copy() + failed it would lead to a crash (invalid free() or invalid + memory reads) in the cleanup paths of these two encoder + initialization functions. + + - Added missing integer overflow check to lzma_index_append(). + This affects xz --list and other applications that decode + the Index field from .xz files using lzma_index_decoder(). + Normal decompression of .xz files doesn't call this code + and thus most applications using liblzma aren't affected + by this bug. + + - Single-threaded .xz decoder (lzma_stream_decoder()): If + lzma_code() returns LZMA_MEMLIMIT_ERROR it is now possible + to use lzma_memlimit_set() to increase the limit and continue + decoding. This was supposed to work from the beginning + but there was a bug. With other decoders (.lzma or + threaded .xz decoder) this already worked correctly. + + - Fixed accumulation of integrity check type statistics in + lzma_index_cat(). This bug made lzma_index_checks() return + only the type of the integrity check of the last Stream + when multiple lzma_indexes were concatenated. Most + applications don't use these APIs but in xz it made + xz --list not list all check types from concatenated .xz + files. In xz --list --verbose only the per-file "Check:" + lines were affected and in xz --robot --list only the "file" + line was affected. + + - Added ABI compatibility with executables that were linked + against liblzma in RHEL/CentOS 7 or other liblzma builds + that had copied the problematic patch from RHEL/CentOS 7 + (xz-5.2.2-compat-libs.patch). For the details, see the + comment at the top of src/liblzma/validate_map.sh. + + WARNING: This uses __symver__ attribute with GCC >= 10. + In other cases the traditional __asm__(".symver ...") + is used. Using link-time optimization (LTO, -flto) with + GCC versions older than 10 can silently result in + broken liblzma.so.5 (incorrect symbol versions)! If you + want to use -flto with GCC, you must use GCC >= 10. + LTO with Clang seems to work even with the traditional + __asm__(".symver ...") method. + + * xzgrep: Fixed compatibility with old shells that break if + comments inside command substitutions have apostrophes ('). + This problem was introduced in 5.2.6. + + * Build systems: + + - New #define in config.h: HAVE_SYMBOL_VERSIONS_LINUX + + - Windows: Fixed liblzma.dll build with Visual Studio project + files. It broke in 5.2.6 due to a change that was made to + improve CMake support. + + - Windows: Building liblzma with UNICODE defined should now + work. + + - CMake files are now actually included in the release tarball. + They should have been in 5.2.5 already. + + - Minor CMake fixes and improvements. + + * Added a new translation: Turkish + + +5.2.6 (2022-08-12) + + * xz: + + - The --keep option now accepts symlinks, hardlinks, and + setuid, setgid, and sticky files. Previously this required + using --force. + + - When copying metadata from the source file to the destination + file, don't try to set the group (GID) if it is already set + correctly. This avoids a failure on OpenBSD (and possibly on + a few other OSes) where files may get created so that their + group doesn't belong to the user, and fchown(2) can fail even + if it needs to do nothing. + + - Cap --memlimit-compress to 2000 MiB instead of 4020 MiB on + MIPS32 because on MIPS32 userspace processes are limited + to 2 GiB of address space. + + * liblzma: + + - Fixed a missing error-check in the threaded encoder. If a + small memory allocation fails, a .xz file with an invalid + Index field would be created. Decompressing such a file would + produce the correct output but result in an error at the end. + Thus this is a "mild" data corruption bug. Note that while + a failed memory allocation can trigger the bug, it cannot + cause invalid memory access. + + - The decoder for .lzma files now supports files that have + uncompressed size stored in the header and still use the + end of payload marker (end of stream marker) at the end + of the LZMA stream. Such files are rare but, according to + the documentation in LZMA SDK, they are valid. + doc/lzma-file-format.txt was updated too. + + - Improved 32-bit x86 assembly files: + * Support Intel Control-flow Enforcement Technology (CET) + * Use non-executable stack on FreeBSD. + + - Visual Studio: Use non-standard _MSVC_LANG to detect C++ + standard version in the lzma.h API header. It's used to + detect when "noexcept" can be used. + + * xzgrep: + + - Fixed arbitrary command injection via a malicious filename + (CVE-2022-1271, ZDI-CAN-16587). A standalone patch for + this was released to the public on 2022-04-07. A slight + robustness improvement has been made since then and, if + using GNU or *BSD grep, a new faster method is now used + that doesn't use the old sed-based construct at all. This + also fixes bad output with GNU grep >= 3.5 (2020-09-27) + when xzgrepping binary files. + + This vulnerability was discovered by: + cleemy desu wayo working with Trend Micro Zero Day Initiative + + - Fixed detection of corrupt .bz2 files. + + - Improved error handling to fix exit status in some situations + and to fix handling of signals: in some situations a signal + didn't make xzgrep exit when it clearly should have. It's + possible that the signal handling still isn't quite perfect + but hopefully it's good enough. + + - Documented exit statuses on the man page. + + - xzegrep and xzfgrep now use "grep -E" and "grep -F" instead + of the deprecated egrep and fgrep commands. + + - Fixed parsing of the options -E, -F, -G, -P, and -X. The + problem occurred when multiple options were specified in + a single argument, for example, + + echo foo | xzgrep -Fe foo + + treated foo as a filename because -Fe wasn't correctly + split into -F -e. + + - Added zstd support. + + * xzdiff/xzcmp: + + - Fixed wrong exit status. Exit status could be 2 when the + correct value is 1. + + - Documented on the man page that exit status of 2 is used + for decompression errors. + + - Added zstd support. + + * xzless: + + - Fix less(1) version detection. It failed if the version number + from "less -V" contained a dot. + + * Translations: + + - Added new translations: Catalan, Croatian, Esperanto, + Korean, Portuguese, Romanian, Serbian, Spanish, Swedish, + and Ukrainian + + - Updated the Brazilian Portuguese translation. + + - Added French man page translation. This and the existing + German translation aren't complete anymore because the + English man pages got a few updates and the translators + weren't reached so that they could update their work. + + * Build systems: + + - Windows: Fix building of resource files when config.h isn't + used. CMake + Visual Studio can now build liblzma.dll. + + - Various fixes to the CMake support. Building static or shared + liblzma should work fine in most cases. In contrast, building + the command line tools with CMake is still clearly incomplete + and experimental and should be used for testing only. + + +5.2.5 (2020-03-17) + + * liblzma: + + - Fixed several C99/C11 conformance bugs. Now the code is clean + under gcc/clang -fsanitize=undefined. Some of these changes + might have a negative effect on performance with old GCC + versions or compilers other than GCC and Clang. The configure + option --enable-unsafe-type-punning can be used to (mostly) + restore the old behavior but it shouldn't normally be used. + + - Improved API documentation of lzma_properties_decode(). + + - Added a very minor encoder speed optimization. + + * xz: + + - Fixed a crash in "xz -dcfv not_an_xz_file". All four options + were required to trigger it. The crash occurred in the + progress indicator code when xz was in passthru mode where + xz works like "cat". + + - Fixed an integer overflow with 32-bit off_t. It could happen + when decompressing a file that has a long run of zero bytes + which xz would try to write as a sparse file. Since the build + system enables large file support by default, off_t is + normally 64-bit even on 32-bit systems. + + - Fixes for --flush-timeout: + * Fix semi-busy-waiting. + * Avoid unneeded flushes when no new input has arrived + since the previous flush was completed. + + - Added a special case for 32-bit xz: If --memlimit-compress is + used to specify a limit that exceeds 4020 MiB, the limit will + be set to 4020 MiB. The values "0" and "max" aren't affected + by this and neither is decompression. This hack can be + helpful when a 32-bit xz has access to 4 GiB address space + but the specified memlimit exceeds 4 GiB. This can happen + e.g. with some scripts. + + - Capsicum sandbox is now enabled by default where available + (FreeBSD >= 10). The sandbox debug messages (xz -vv) were + removed since they seemed to be more annoying than useful. + + - DOS build now requires DJGPP 2.05 instead of 2.04beta. + A workaround for a locale problem with DJGPP 2.05 was added. + + * xzgrep and other scripts: + + - Added a configure option --enable-path-for-scripts=PREFIX. + It is disabled by default except on Solaris where the default + is /usr/xpg4/bin. See INSTALL for details. + + - Added a workaround for a POSIX shell detection problem on + Solaris. + + * Build systems: + + - Added preliminary build instructions for z/OS. See INSTALL + section 1.2.9. + + - Experimental CMake support was added. It should work to build + static liblzma on a few operating systems. It may or may not + work to build shared liblzma. On some platforms it can build + xz and xzdec too but those are only for testing. See the + comment in the beginning of CMakeLists.txt for details. + + - Visual Studio project files were updated. + WindowsTargetPlatformVersion was removed from VS2017 files + and set to "10.0" in the added VS2019 files. In the future + the VS project files will be removed when CMake support is + good enough. + + - New #defines in config.h: HAVE___BUILTIN_ASSUME_ALIGNED, + HAVE___BUILTIN_BSWAPXX, and TUKLIB_USE_UNSAFE_TYPE_PUNNING. + + - autogen.sh has a new optional dependency on po4a and a new + option --no-po4a to skip that step. This matters only if one + wants to remake the build files. po4a is used to update the + translated man pages but as long as the man pages haven't + been modified, there's nothing to update and one can use + --no-po4a to avoid the dependency on po4a. + + * Translations: + + - XZ Utils translations are now handled by the Translation + Project: https://translationproject.org/domain/xz.html + + - All man pages are now included in German too. + + - New xz translations: Brazilian Portuguese, Finnish, + Hungarian, Chinese (simplified), Chinese (traditional), + and Danish (partial translation) + + - Updated xz translations: French, German, Italian, and Polish + + - Unfortunately a few new xz translations weren't included due + to technical problems like too long lines in --help output or + misaligned column headings in tables. In the future, many of + these strings will be split and e.g. the table column + alignment will be handled in software. This should make the + strings easier to translate. + + +5.2.4 (2018-04-29) + + * liblzma: + + - Allow 0 as memory usage limit instead of returning + LZMA_PROG_ERROR. Now 0 is treated as if 1 byte was specified, + which effectively is the same as 0. + + - Use "noexcept" keyword instead of "throw()" in the public + headers when a C++11 (or newer standard) compiler is used. + + - Added a portability fix for recent Intel C Compilers. + + - Microsoft Visual Studio build files have been moved under + windows/vs2013 and windows/vs2017. + + * xz: + + - Fix "xz --list --robot missing_or_bad_file.xz" which would + try to print an uninitialized string and thus produce garbage + output. Since the exit status is non-zero, most uses of such + a command won't try to interpret the garbage output. + + - "xz --list foo.xz" could print "Internal error (bug)" in a + corner case where a specific memory usage limit had been set. + + +5.2.3 (2016-12-30) + + * xz: + + - Always close a file before trying to delete it to avoid + problems on some operating system and file system combinations. + + - Fixed copying of file timestamps on Windows. + + - Added experimental (disabled by default) sandbox support using + Capsicum (FreeBSD >= 10). See --enable-sandbox in INSTALL. + + * C99/C11 conformance fixes to liblzma. The issues affected at least + some builds using link-time optimizations. + + * Fixed bugs in the rarely-used function lzma_index_dup(). + + * Use of external SHA-256 code is now disabled by default. + It can still be enabled by passing --enable-external-sha256 + to configure. The reasons to disable it by default (see INSTALL + for more details): + + - Some OS-specific SHA-256 implementations conflict with + OpenSSL and cause problems in programs that link against both + liblzma and libcrypto. At least FreeBSD 10 and MINIX 3.3.0 + are affected. + + - The internal SHA-256 is faster than the SHA-256 code in + some operating systems. + + * Changed CPU core count detection to use sched_getaffinity() on + GNU/Linux and GNU/kFreeBSD. + + * Fixes to the build-system and xz to make xz buildable even when + encoders, decoders, or threading have been disabled from libilzma + using configure options. These fixes added two new #defines to + config.h: HAVE_ENCODERS and HAVE_DECODERS. + + +5.2.2 (2015-09-29) + + * Fixed bugs in QNX-specific code. + + * Omitted the use of pipe2() even if it is available to avoid + portability issues with some old Linux and glibc combinations. + + * Updated German translation. + + * Added project files to build static and shared liblzma (not the + whole XZ Utils) with Visual Studio 2013 update 2 or later. + + * Documented that threaded decompression hasn't been implemented + yet. A 5.2.0 NEWS entry describing multi-threading support had + incorrectly said "decompression" when it should have said + "compression". + + +5.2.1 (2015-02-26) + + * Fixed a compression-ratio regression in fast mode of LZMA1 and + LZMA2. The bug is present in 5.1.4beta and 5.2.0 releases. + + * Fixed a portability problem in xz that affected at least OpenBSD. + + * Fixed xzdiff to be compatible with FreeBSD's mktemp which differs + from most other mktemp implementations. + + * Changed CPU core count detection to use cpuset_getaffinity() on + FreeBSD. + + +5.2.0 (2014-12-21) + + Since 5.1.4beta: + + * All fixes from 5.0.8 + + * liblzma: Fixed lzma_stream_encoder_mt_memusage() when a preset + was used. + + * xzdiff: If mktemp isn't installed, mkdir will be used as + a fallback to create a temporary directory. Installing mktemp + is still recommended. + + * Updated French, German, Italian, Polish, and Vietnamese + translations. + + Summary of fixes and new features added in the 5.1.x development + releases: + + * liblzma: + + - Added support for multi-threaded compression. See the + lzma_mt structure, lzma_stream_encoder_mt(), and + lzma_stream_encoder_mt_memusage() in , + lzma_get_progress() in , and lzma_cputhreads() + in for details. + + - Made the uses of lzma_allocator const correct. + + - Added lzma_block_uncomp_encode() to create uncompressed + .xz Blocks using LZMA2 uncompressed chunks. + + - Added support for LZMA_IGNORE_CHECK. + + - A few speed optimizations were made. + + - Added support for symbol versioning. It is enabled by default + on GNU/Linux, other GNU-based systems, and FreeBSD. + + - liblzma (not the whole XZ Utils) should now be buildable + with MSVC 2013 update 2 or later using windows/config.h. + + * xz: + + - Fixed a race condition in the signal handling. It was + possible that e.g. the first SIGINT didn't make xz exit + if reading or writing blocked and one had bad luck. The fix + is non-trivial, so as of writing it is unknown if it will be + backported to the v5.0 branch. + + - Multi-threaded compression can be enabled with the + --threads (-T) option. + [Fixed: This originally said "decompression".] + + - New command line options in xz: --single-stream, + --block-size=SIZE, --block-list=SIZES, + --flush-timeout=TIMEOUT, and --ignore-check. + + - xz -lvv now shows the minimum xz version that is required to + decompress the file. Currently it is 5.0.0 for all supported + .xz files except files with empty LZMA2 streams require 5.0.2. + + * xzdiff and xzgrep now support .lzo files if lzop is installed. + The .tzo suffix is also recognized as a shorthand for .tar.lzo. + + +5.1.4beta (2014-09-14) + + * All fixes from 5.0.6 + + * liblzma: Fixed the use of presets in threaded encoder + initialization. + + * xz --block-list and --block-size can now be used together + in single-threaded mode. Previously the combination only + worked in multi-threaded mode. + + * Added support for LZMA_IGNORE_CHECK to liblzma and made it + available in xz as --ignore-check. + + * liblzma speed optimizations: + + - Initialization of a new LZMA1 or LZMA2 encoder has been + optimized. (The speed of reinitializing an already-allocated + encoder isn't affected.) This helps when compressing many + small buffers with lzma_stream_buffer_encode() and other + similar situations where an already-allocated encoder state + isn't reused. This speed-up is visible in xz too if one + compresses many small files one at a time instead running xz + once and giving all files as command-line arguments. + + - Buffer comparisons are now much faster when unaligned access + is allowed (configured with --enable-unaligned-access). This + speeds up encoding significantly. There is arch-specific code + for 32-bit and 64-bit x86 (32-bit needs SSE2 for the best + results and there's no run-time CPU detection for now). + For other archs there is only generic code which probably + isn't as optimal as arch-specific solutions could be. + + - A few speed optimizations were made to the SHA-256 code. + (Note that the builtin SHA-256 code isn't used on all + operating systems.) + + * liblzma can now be built with MSVC 2013 update 2 or later + using windows/config.h. + + * Vietnamese translation was added. + + +5.1.3alpha (2013-10-26) + + * All fixes from 5.0.5 + + * liblzma: + + - Fixed a deadlock in the threaded encoder. + + - Made the uses of lzma_allocator const correct. + + - Added lzma_block_uncomp_encode() to create uncompressed + .xz Blocks using LZMA2 uncompressed chunks. + + - Added support for native threads on Windows and the ability + to detect the number of CPU cores. + + * xz: + + - Fixed a race condition in the signal handling. It was + possible that e.g. the first SIGINT didn't make xz exit + if reading or writing blocked and one had bad luck. The fix + is non-trivial, so as of writing it is unknown if it will be + backported to the v5.0 branch. + + - Made the progress indicator work correctly in threaded mode. + + - Threaded encoder now works together with --block-list=SIZES. + + - Added preliminary support for --flush-timeout=TIMEOUT. + It can be useful for (somewhat) real-time streaming. For + now the decompression side has to be done with something + else than the xz tool due to how xz does buffering, but this + should be fixed. + + +5.1.2alpha (2012-07-04) + + * All fixes from 5.0.3 and 5.0.4 + + * liblzma: + + - Fixed a deadlock and an invalid free() in the threaded encoder. + + - Added support for symbol versioning. It is enabled by default + on GNU/Linux, other GNU-based systems, and FreeBSD. + + - Use SHA-256 implementation from the operating system if one is + available in libc, libmd, or libutil. liblzma won't use e.g. + OpenSSL or libgcrypt to avoid introducing new dependencies. + + - Fixed liblzma.pc for static linking. + + - Fixed a few portability bugs. + + * xz --decompress --single-stream now fixes the input position after + successful decompression. Now the following works: + + echo foo | xz > foo.xz + echo bar | xz >> foo.xz + ( xz -dc --single-stream ; xz -dc --single-stream ) < foo.xz + + Note that it doesn't work if the input is not seekable + or if there is Stream Padding between the concatenated + .xz Streams. + + * xz -lvv now shows the minimum xz version that is required to + decompress the file. Currently it is 5.0.0 for all supported .xz + files except files with empty LZMA2 streams require 5.0.2. + + * Added an *incomplete* implementation of --block-list=SIZES to xz. + It only works correctly in single-threaded mode and when + --block-size isn't used at the same time. --block-list allows + specifying the sizes of Blocks which can be useful e.g. when + creating files for random-access reading. + + +5.1.1alpha (2011-04-12) + + * All fixes from 5.0.2 + + * liblzma fixes that will also be included in 5.0.3: + + - A memory leak was fixed. + + - lzma_stream_buffer_encode() no longer creates an empty .xz + Block if encoding an empty buffer. Such an empty Block with + LZMA2 data would trigger a bug in 5.0.1 and older (see the + first bullet point in 5.0.2 notes). When releasing 5.0.2, + I thought that no encoder creates this kind of files but + I was wrong. + + - Validate function arguments better in a few functions. Most + importantly, specifying an unsupported integrity check to + lzma_stream_buffer_encode() no longer creates a corrupt .xz + file. Probably no application tries to do that, so this + shouldn't be a big problem in practice. + + - Document that lzma_block_buffer_encode(), + lzma_easy_buffer_encode(), lzma_stream_encoder(), and + lzma_stream_buffer_encode() may return LZMA_UNSUPPORTED_CHECK. + + - The return values of the _memusage() functions are now + documented better. + + * Support for multithreaded compression was added using the simplest + method, which splits the input data into blocks and compresses + them independently. Other methods will be added in the future. + The current method has room for improvement, e.g. it is possible + to reduce the memory usage. + + * Added the options --single-stream and --block-size=SIZE to xz. + + * xzdiff and xzgrep now support .lzo files if lzop is installed. + The .tzo suffix is also recognized as a shorthand for .tar.lzo. + + * Support for short 8.3 filenames under DOS was added to xz. It is + experimental and may change before it gets into a stable release. + + +5.0.8 (2014-12-21) + + * Fixed an old bug in xzgrep that affected OpenBSD and probably + a few other operating systems too. + + * Updated French and German translations. + + * Added support for detecting the amount of RAM on AmigaOS/AROS. + + * Minor build system updates. + + +5.0.7 (2014-09-20) + + * Fix regressions introduced in 5.0.6: + + - Fix building with non-GNU make. + + - Fix invalid Libs.private value in liblzma.pc which broke + static linking against liblzma if the linker flags were + taken from pkg-config. + + +5.0.6 (2014-09-14) + + * xzgrep now exits with status 0 if at least one file matched. + + * A few minor portability and build system fixes + + +5.0.5 (2013-06-30) + + * lzmadec and liblzma's lzma_alone_decoder(): Support decompressing + .lzma files that have less common settings in the headers + (dictionary size other than 2^n or 2^n + 2^(n-1), or uncompressed + size greater than 256 GiB). The limitations existed to avoid false + positives when detecting .lzma files. The lc + lp <= 4 limitation + still remains since liblzma's LZMA decoder has that limitation. + + NOTE: xz's .lzma support or liblzma's lzma_auto_decoder() are NOT + affected by this change. They still consider uncommon .lzma headers + as not being in the .lzma format. Changing this would give way too + many false positives. + + * xz: + + - Interaction of preset and custom filter chain options was + made less illogical. This affects only certain less typical + uses cases so few people are expected to notice this change. + + Now when a custom filter chain option (e.g. --lzma2) is + specified, all preset options (-0 ... -9, -e) earlier are on + the command line are completely forgotten. Similarly, when + a preset option is specified, all custom filter chain options + earlier on the command line are completely forgotten. + + Example 1: "xz -9 --lzma2=preset=5 -e" is equivalent to "xz -e" + which is equivalent to "xz -6e". Earlier -e didn't put xz back + into preset mode and thus the example command was equivalent + to "xz --lzma2=preset=5". + + Example 2: "xz -9e --lzma2=preset=5 -7" is equivalent to + "xz -7". Earlier a custom filter chain option didn't make + xz forget the -e option so the example was equivalent to + "xz -7e". + + - Fixes and improvements to error handling. + + - Various fixes to the man page. + + * xzless: Fixed to work with "less" versions 448 and later. + + * xzgrep: Made -h an alias for --no-filename. + + * Include the previously missing debug/translation.bash which can + be useful for translators. + + * Include a build script for Mac OS X. This has been in the Git + repository since 2010 but due to a mistake in Makefile.am the + script hasn't been included in a release tarball before. + + +5.0.4 (2012-06-22) + + * liblzma: + + - Fix lzma_index_init(). It could crash if memory allocation + failed. + + - Fix the possibility of an incorrect LZMA_BUF_ERROR when a BCJ + filter is used and the application only provides exactly as + much output space as is the uncompressed size of the file. + + - Fix a bug in doc/examples_old/xz_pipe_decompress.c. It didn't + check if the last call to lzma_code() really returned + LZMA_STREAM_END, which made the program think that truncated + files are valid. + + - New example programs in doc/examples (old programs are now in + doc/examples_old). These have more comments and more detailed + error handling. + + * Fix "xz -lvv foo.xz". It could crash on some corrupted files. + + * Fix output of "xz --robot -lv" and "xz --robot -lvv" which + incorrectly printed the filename also in the "foo (x/x)" format. + + * Fix exit status of "xzdiff foo.xz bar.xz". + + * Fix exit status of "xzgrep foo binary_file". + + * Fix portability to EBCDIC systems. + + * Fix a configure issue on AIX with the XL C compiler. See INSTALL + for details. + + * Update French, German, Italian, and Polish translations. + + +5.0.3 (2011-05-21) + + * liblzma fixes: + + - A memory leak was fixed. + + - lzma_stream_buffer_encode() no longer creates an empty .xz + Block if encoding an empty buffer. Such an empty Block with + LZMA2 data would trigger a bug in 5.0.1 and older (see the + first bullet point in 5.0.2 notes). When releasing 5.0.2, + I thought that no encoder creates this kind of files but + I was wrong. + + - Validate function arguments better in a few functions. Most + importantly, specifying an unsupported integrity check to + lzma_stream_buffer_encode() no longer creates a corrupt .xz + file. Probably no application tries to do that, so this + shouldn't be a big problem in practice. + + - Document that lzma_block_buffer_encode(), + lzma_easy_buffer_encode(), lzma_stream_encoder(), and + lzma_stream_buffer_encode() may return LZMA_UNSUPPORTED_CHECK. + + - The return values of the _memusage() functions are now + documented better. + + * Fix command name detection in xzgrep. xzegrep and xzfgrep now + correctly use egrep and fgrep instead of grep. + + * French translation was added. + + +5.0.2 (2011-04-01) + + * LZMA2 decompressor now correctly accepts LZMA2 streams with no + uncompressed data. Previously it considered them corrupt. The + bug can affect applications that use raw LZMA2 streams. It is + very unlikely to affect .xz files because no compressor creates + .xz files with empty LZMA2 streams. (Empty .xz files are a + different thing than empty LZMA2 streams.) + + * "xz --suffix=.foo filename.foo" now refuses to compress the + file due to it already having the suffix .foo. It was already + documented on the man page, but the code lacked the test. + + * "xzgrep -l foo bar.xz" works now. + + * Polish translation was added. + + +5.0.1 (2011-01-29) + + * xz --force now (de)compresses files that have setuid, setgid, + or sticky bit set and files that have multiple hard links. + The man page had it documented this way already, but the code + had a bug. + + * gzip and bzip2 support in xzdiff was fixed. + + * Portability fixes + + * Minor fix to Czech translation + + +5.0.0 (2010-10-23) + + Only the most important changes compared to 4.999.9beta are listed + here. One change is especially important: + + * The memory usage limit is now disabled by default. Some scripts + written before this change may have used --memory=max on xz command + line or in XZ_OPT. THESE USES OF --memory=max SHOULD BE REMOVED + NOW, because they interfere with user's ability to set the memory + usage limit himself. If user-specified limit causes problems to + your script, blame the user. + + Other significant changes: + + * Added support for XZ_DEFAULTS environment variable. This variable + allows users to set default options for xz, e.g. default memory + usage limit or default compression level. Scripts that use xz + must never set or unset XZ_DEFAULTS. Scripts should use XZ_OPT + instead if they need a way to pass options to xz via an + environment variable. + + * The compression settings associated with the preset levels + -0 ... -9 have been changed. --extreme was changed a little too. + It is now less likely to make compression worse, but with some + files the new --extreme may compress slightly worse than the old + --extreme. + + * If a preset level (-0 ... -9) is specified after a custom filter + chain options have been used (e.g. --lzma2), the custom filter + chain will be forgotten. Earlier the preset options were + completely ignored after custom filter chain options had been + seen. + + * xz will create sparse files when decompressing if the uncompressed + data contains long sequences of binary zeros. This is done even + when writing to standard output that is connected to a regular + file and certain additional conditions are met to make it safe. + + * Support for "xz --list" was added. Combine with --verbose or + --verbose --verbose (-vv) for detailed output. + + * I had hoped that liblzma API would have been stable after + 4.999.9beta, but there have been a couple of changes in the + advanced features, which don't affect most applications: + + - Index handling code was revised. If you were using the old + API, you will get a compiler error (so it's easy to notice). + + - A subtle but important change was made to the Block handling + API. lzma_block.version has to be initialized even for + lzma_block_header_decode(). Code that doesn't do it will work + for now, but might break in the future, which makes this API + change easy to miss. + + * The major soname has been bumped to 5.0.0. liblzma API and ABI + are now stable, so the need to recompile programs linking against + liblzma shouldn't arise soon. + diff --git a/PACKAGERS b/PACKAGERS new file mode 100644 index 0000000000..b12c4851a4 --- /dev/null +++ b/PACKAGERS @@ -0,0 +1,245 @@ + +Information to packagers of XZ Utils +==================================== + + 0. Preface + 1. Package naming + 2. Package description + 3. License + 4. configure options + 5. Additional documentation + 6. Extra files + 7. Installing XZ Utils and LZMA Utils in parallel + 8. Example + + +0. Preface +---------- + + This document is meant for people who create and maintain XZ Utils + packages for operating system distributions. The focus is on GNU/Linux + systems, but most things apply to other systems too. + + While the standard "configure && make DESTDIR=$PKG install" should + give a pretty good package, there are some details which packagers + may want to tweak. + + Packagers should also read the INSTALL file. + + +1. Package naming +----------------- + + The preferred name for the XZ Utils package is "xz", because that's + the name of the upstream tarball. Naturally you may have good reasons + to use some other name; I won't get angry about it. ;-) It's just nice + to be able to point people to the correct package name without asking + what distro they have. + + If your distro policy is to split things into small pieces, here is + one suggestion: + + xz xz, xzdec, scripts (xzdiff, xzgrep, etc.), docs + xz-lzma lzma, unlzma, lzcat, lzgrep etc. symlinks and + lzmadec binary for compatibility with LZMA Utils + liblzma liblzma.so.* + liblzma-devel liblzma.so, liblzma.a, API headers + liblzma-doc Example programs and, if enabled at build time, + Doxygen-generated liblzma API docs (HTML) + + +2. Package description +---------------------- + + Here is a suggestion which you may use as the package description. + If you can use only one-line description, pick only the first line. + Naturally, feel free to use some other description if you find it + better, and maybe send it to me too. + + Library and command line tools for XZ and LZMA compressed files + + XZ Utils provide a general purpose data compression library + and command line tools. The native file format is the .xz + format, but also the legacy .lzma format is supported. The .xz + format supports multiple compression algorithms, of which LZMA2 + is currently the primary algorithm. With typical files, XZ Utils + create about 30 % smaller files than gzip. + + If you are splitting XZ Utils into multiple packages, here are some + suggestions for package descriptions: + + xz: + + Command line tools for XZ and LZMA compressed files + + This package includes the xz compression tool and other command + line tools from XZ Utils. xz has command line syntax similar to + that of gzip. The native file format is the .xz format, but also + the legacy .lzma format is supported. The .xz format supports + multiple compression algorithms, of which LZMA2 is currently the + primary algorithm. With typical files, XZ Utils create about 30 % + smaller files than gzip. + + Note that this package doesn't include the files needed for + LZMA Utils 4.32.x compatibility. Install also the xz-lzma + package to make XZ Utils emulate LZMA Utils 4.32.x. + + xz-lzma: + + LZMA Utils emulation with XZ Utils + + This package includes executables and symlinks to make + XZ Utils emulate lzma, unlzma, lzcat, and other command + line tools found from the legacy LZMA Utils 4.32.x package. + + liblzma: + + Library for XZ and LZMA compressed files + + liblzma is a general purpose data compression library with + an API similar to that of zlib. liblzma supports multiple + algorithms, of which LZMA2 is currently the primary algorithm. + The native file format is .xz, but also the legacy .lzma + format and raw streams (no headers at all) are supported. + + This package includes the shared library. + + liblzma-devel: + + Library for XZ and LZMA compressed files + + This package includes the API headers, static library, and + other development files related to liblzma. + + liblzma-doc: + + liblzma API documentation in HTML and example usage + + This package includes the Doxygen-generated liblzma API + HTML docs and example programs showing how to use liblzma. + + +3. License +---------- + + If the package manager supports a license field, you probably should + put GPLv2+ there (GNU GPL v2 or later). The interesting parts of + XZ Utils are under the BSD Zero Clause License (0BSD), but some less + important files ending up into the binary package are under GPLv2+. + So it is simplest to just say GPLv2+ if you cannot specify + "BSD0 and GPLv2+". + + If you split XZ Utils into multiple packages as described earlier + in this file, liblzma and liblzma-dev packages will contain only + 0BSD-licensed code from XZ Utils (compiler or linker may add some + third-party code which may have other licenses). + + +4. configure options +-------------------- + + Unless you are building a package for a distribution that is meant + only for embedded systems, don't use the following configure options: + + --enable-debug + --enable-encoders (*) + --enable-decoders + --enable-match-finders + --enable-checks + --enable-small (*) + --disable-threads (*) + --disable-microlzma (*) + --disable-lzip-decoder (*) + + (*) These are OK when building xzdec and lzmadec as described + in INSTALL. + + xzdec and lzmadec don't provide any functionality that isn't already + available in the xz tool. Shipping xzdec and lzmadec without size + optimization and statically-linked liblzma isn't very useful. Doing + that would give users the xzdec man page, which may make it easier + for people to find out that such tools exists, but the executables + wouldn't have any advantage over the full-featured xz. + + +5. Additional documentation +--------------------------- + + "make install" copies some additional documentation to $docdir + (--docdir in configure). There is a copy of the GNU GPL v2, which + can be replaced with a symlink if your distro ships with shared + copies of the common license texts. + + The Doxygen-generated liblzma API documentation (HTML) is built and + installed if the configure option --enable-doxygen is used (it's + disabled by default). This requires that Doxygen is available. The + API documentation is installed by "make install" to $docdir/api. + + NOTE: The files generated by Doxygen include content from + Doxygen itself. Check the license info before distributing + the Doxygen-generated files. + + +6. Extra files +-------------- + + The "extra" directory contains some small extra tools or other files. + The exact set of extra files can vary between XZ Utils releases. The + extra files have only limited use or they are too dangerous to be + put directly to $bindir (7z2lzma.sh is a good example, since it can + silently create corrupt output if certain conditions are not met). + + If you feel like it, you may copy the extra directory under the doc + directory (e.g. /usr/share/doc/xz/extra). Maybe some people will find + them useful. However, most people needing these tools probably are + able to find them from the source package too. + + The "debug" directory contains some tools that are useful only when + hacking on XZ Utils. Don't package these tools. + + +7. Installing XZ Utils and LZMA Utils in parallel +------------------------------------------------- + + XZ Utils and LZMA Utils 4.32.x can be installed in parallel by + omitting the compatibility symlinks (lzma, unlzma, lzcat, lzgrep etc.) + from the XZ Utils package. It's probably a good idea to still package + the symlinks into a separate package so that users may choose if they + want to use XZ Utils or LZMA Utils for handling .lzma files. + + +8. Example +---------- + + Here is an example for i686 GNU/Linux that + - links xz and lzmainfo against shared liblzma; + - links size-optimized xzdec and lzmadec against static liblzma + while avoiding libpthread dependency; + - includes only shared liblzma in the final package; and + - copies also the "extra" directory to the package. + + PKG=/tmp/xz-pkg + tar xf xz-x.y.z.tar.gz + cd xz-x.y.z + ./configure \ + --prefix=/usr \ + --disable-static \ + --disable-xzdec \ + --disable-lzmadec \ + CFLAGS='-march=i686 -mtune=generic -O2' + make + make DESTDIR=$PKG install-strip + make clean + ./configure \ + --prefix=/usr \ + --disable-shared \ + --disable-nls \ + --disable-encoders \ + --enable-small \ + --disable-threads \ + CFLAGS='-march=i686 -mtune=generic -Os' + make -C src/liblzma + make -C src/xzdec + make -C src/xzdec DESTDIR=$PKG install-strip + cp -a extra $PKG/usr/share/doc/xz + diff --git a/README b/README new file mode 100644 index 0000000000..41671676a5 --- /dev/null +++ b/README @@ -0,0 +1,281 @@ + +XZ Utils +======== + + 0. Overview + 1. Documentation + 1.1. Overall documentation + 1.2. Documentation for command-line tools + 1.3. Documentation for liblzma + 2. Version numbering + 3. Reporting bugs + 4. Translations + 4.1. Testing translations + 5. Other implementations of the .xz format + 6. Contact information + + +0. Overview +----------- + + XZ Utils provide a general-purpose data-compression library plus + command-line tools. The native file format is the .xz format, but + also the legacy .lzma format is supported. The .xz format supports + multiple compression algorithms, which are called "filters" in the + context of XZ Utils. The primary filter is currently LZMA2. With + typical files, XZ Utils create about 30 % smaller files than gzip. + + To ease adapting support for the .xz format into existing applications + and scripts, the API of liblzma is somewhat similar to the API of the + popular zlib library. For the same reason, the command-line tool xz + has a command-line syntax similar to that of gzip. + + When aiming for the highest compression ratio, the LZMA2 encoder uses + a lot of CPU time and may use, depending on the settings, even + hundreds of megabytes of RAM. However, in fast modes, the LZMA2 encoder + competes with bzip2 in compression speed, RAM usage, and compression + ratio. + + LZMA2 is reasonably fast to decompress. It is a little slower than + gzip, but a lot faster than bzip2. Being fast to decompress means + that the .xz format is especially nice when the same file will be + decompressed very many times (usually on different computers), which + is the case e.g. when distributing software packages. In such + situations, it's not too bad if the compression takes some time, + since that needs to be done only once to benefit many people. + + With some file types, combining (or "chaining") LZMA2 with an + additional filter can improve the compression ratio. A filter chain may + contain up to four filters, although usually only one or two are used. + For example, putting a BCJ (Branch/Call/Jump) filter before LZMA2 + in the filter chain can improve compression ratio of executable files. + + Since the .xz format allows adding new filter IDs, it is possible that + some day there will be a filter that is, for example, much faster to + compress than LZMA2 (but probably with worse compression ratio). + Similarly, it is possible that some day there is a filter that will + compress better than LZMA2. + + XZ Utils supports multithreaded compression. XZ Utils doesn't support + multithreaded decompression yet. It has been planned though and taken + into account when designing the .xz file format. In the future, files + that were created in threaded mode can be decompressed in threaded + mode too. + + +1. Documentation +---------------- + +1.1. Overall documentation + + README This file + + INSTALL.generic Generic install instructions for those not + familiar with packages using GNU Autotools + INSTALL Installation instructions specific to XZ Utils + PACKAGERS Information to packagers of XZ Utils + + COPYING XZ Utils copyright and license information + COPYING.0BSD BSD Zero Clause License + COPYING.GPLv2 GNU General Public License version 2 + COPYING.GPLv3 GNU General Public License version 3 + COPYING.LGPLv2.1 GNU Lesser General Public License version 2.1 + + AUTHORS The main authors of XZ Utils + THANKS Incomplete list of people who have helped making + this software + NEWS User-visible changes between XZ Utils releases + ChangeLog Detailed list of changes (commit log) + TODO Known bugs and some sort of to-do list + + Note that only some of the above files are included in binary + packages. + + +1.2. Documentation for command-line tools + + The command-line tools are documented as man pages. In source code + releases (and possibly also in some binary packages), the man pages + are also provided in plain text (ASCII only) format in the directory + "doc/man" to make the man pages more accessible to those whose + operating system doesn't provide an easy way to view man pages. + + +1.3. Documentation for liblzma + + The liblzma API headers include short docs about each function + and data type as Doxygen tags. These docs should be quite OK as + a quick reference. + + There are a few example/tutorial programs that should help in + getting started with liblzma. In the source package the examples + are in "doc/examples" and in binary packages they may be under + "examples" in the same directory as this README. + + Since the liblzma API has similarities to the zlib API, some people + may find it useful to read the zlib docs and tutorial too: + + https://zlib.net/manual.html + https://zlib.net/zlib_how.html + + +2. Version numbering +-------------------- + + The version number format of XZ Utils is X.Y.ZS: + + - X is the major version. When this is incremented, the library + API and ABI break. + + - Y is the minor version. It is incremented when new features + are added without breaking the existing API or ABI. An even Y + indicates a stable release and an odd Y indicates unstable + (alpha or beta version). + + - Z is the revision. This has a different meaning for stable and + unstable releases: + + * Stable: Z is incremented when bugs get fixed without adding + any new features. This is intended to be convenient for + downstream distributors that want bug fixes but don't want + any new features to minimize the risk of introducing new bugs. + + * Unstable: Z is just a counter. API or ABI of features added + in earlier unstable releases having the same X.Y may break. + + - S indicates stability of the release. It is missing from the + stable releases, where Y is an even number. When Y is odd, S + is either "alpha" or "beta" to make it very clear that such + versions are not stable releases. The same X.Y.Z combination is + not used for more than one stability level, i.e. after X.Y.Zalpha, + the next version can be X.Y.(Z+1)beta but not X.Y.Zbeta. + + +3. Reporting bugs +----------------- + + Naturally it is easiest for me if you already know what causes the + unexpected behavior. Even better if you have a patch to propose. + However, quite often the reason for unexpected behavior is unknown, + so here are a few things to do before sending a bug report: + + 1. Try to create a small example how to reproduce the issue. + + 2. Compile XZ Utils with debugging code using configure switches + --enable-debug and, if possible, --disable-shared. If you are + using GCC, use CFLAGS='-O0 -ggdb3'. Don't strip the resulting + binaries. + + 3. Turn on core dumps. The exact command depends on your shell; + for example in GNU bash it is done with "ulimit -c unlimited", + and in tcsh with "limit coredumpsize unlimited". + + 4. Try to reproduce the suspected bug. If you get "assertion failed" + message, be sure to include the complete message in your bug + report. If the application leaves a coredump, get a backtrace + using gdb: + $ gdb /path/to/app-binary # Load the app to the debugger. + (gdb) core core # Open the coredump. + (gdb) bt # Print the backtrace. Copy & paste to bug report. + (gdb) quit # Quit gdb. + + Report your bug via email or IRC (see Contact information below). + Don't send core dump files or any executables. If you have a small + example file(s) (total size less than 256 KiB), please include + it/them as an attachment. If you have bigger test files, put them + online somewhere and include a URL to the file(s) in the bug report. + + Always include the exact version number of XZ Utils in the bug report. + If you are using a snapshot from the git repository, use "git describe" + to get the exact snapshot version. If you are using XZ Utils shipped + in an operating system distribution, mention the distribution name, + distribution version, and exact xz package version; if you cannot + repeat the bug with the code compiled from unpatched source code, + you probably need to report a bug to your distribution's bug tracking + system. + + +4. Translations +--------------- + + The xz command line tool and all man pages can be translated. + The translations are handled via the Translation Project. If you + wish to help translating xz, please join the Translation Project: + + https://translationproject.org/html/translators.html + + Updates to translations won't be accepted by methods that bypass + the Translation Project because there is a risk of duplicate work: + translation updates made in the xz repository aren't seen by the + translators in the Translation Project. If you have found bugs in + a translation, please report them to the Language-Team address + which can be found near the beginning of the PO file. + + If you find language problems in the original English strings, + feel free to suggest improvements. Ask if something is unclear. + + +4.1. Testing translations + + Testing can be done by installing xz into a temporary directory. + + If building from Git repository (not tarball), generate the + Autotools files: + + ./autogen.sh + + Create a subdirectory for the build files. The tmp-build directory + can be deleted after testing. + + mkdir tmp-build + cd tmp-build + ../configure --disable-shared --enable-debug --prefix=$PWD/inst + + Edit the .po file in the po directory. Then build and install to + the "tmp-build/inst" directory, and use translations.bash to see + how some of the messages look. Repeat these steps if needed: + + make -C po update-po + make -j"$(nproc)" install + bash ../debug/translation.bash | less + bash ../debug/translation.bash | less -S # For --list outputs + + To test other languages, set the LANGUAGE environment variable + before running translations.bash. The value should match the PO file + name without the .po suffix. Example: + + export LANGUAGE=fi + + +5. Other implementations of the .xz format +------------------------------------------ + + 7-Zip and the p7zip port of 7-Zip support the .xz format starting + from the version 9.00alpha. + + https://7-zip.org/ + https://p7zip.sourceforge.net/ + + XZ Embedded is a limited implementation written for use in the Linux + kernel, but it is also suitable for other embedded use. + + https://tukaani.org/xz/embedded.html + + XZ for Java is a complete implementation written in pure Java. + + https://tukaani.org/xz/java.html + + +6. Contact information +---------------------- + + XZ Utils in general: + - Home page: https://tukaani.org/xz/ + - Email to maintainer(s): xz@tukaani.org + - IRC: #tukaani on Libera Chat + - GitHub: https://github.com/tukaani-project/xz + + Lead maintainer: + - Email: Lasse Collin + - IRC: Larhzu on Libera Chat + diff --git a/README.md b/README.md index 4cae0298f9..1714844c97 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,12 @@ # cpython-source-deps Source for packages that the cpython build process depends on + +## XZ Utils +Find updates for these at https://tukaani.org/xz/ + +We only include the bare minimum required files for building our _lzma module, +since the xz repository mixes licenses. liblzma is 0-BSD. + +The windows\config.h header is created manually. To find out what the settings +ought to be, use the full xz sources to generate build projects using CMake, +and then inspect the project files. \ No newline at end of file diff --git a/THANKS b/THANKS new file mode 100644 index 0000000000..a6a7a67210 --- /dev/null +++ b/THANKS @@ -0,0 +1,239 @@ + +Thanks +====== + +Some people have helped more, some less, but nevertheless everyone's help +has been important. :-) In alphabetical order: + - Mark Adler + - Kian-Meng Ang + - H. Peter Anvin + - Jeff Bastian + - Nelson H. F. Beebe + - Karl Beldan + - Karl Berry + - Anders F. Björklund + - Emmanuel Blot + - Melanie Blower + - Alexander Bluhm + - Martin Blumenstingl + - Ben Boeckel + - Jakub Bogusz + - Adam Borowski + - Maarten Bosmans + - Roel Bouckaert + - Lukas Braune + - Benjamin Buch + - Trent W. Buck + - Kevin R. Bulgrien + - James Buren + - David Burklund + - Frank Busse + - Daniel Mealha Cabrita + - Milo Casagrande + - Cristiano Ceglia + - Marek Černocký + - Tomer Chachamu + - Vitaly Chikunov + - Antoine Cœur + - Elijah Almeida Coimbra + - Felix Collin + - Ryan Colyer + - Marcus Comstedt + - Vincent Cruz + - Gabi Davar + - Ron Desmond + - İhsan Doğan + - Chris Donawa + - Andrew Dudman + - Markus Duft + - İsmail Dönmez + - Dexter Castor Döpping + - Paul Eggert + - Robert Elz + - Gilles Espinasse + - Denis Excoffier + - Vincent Fazio + - Michael Felt + - Sean Fenian + - Michael Fox + - Andres Freund + - Mike Frysinger + - Collin Funk + - Daniel Richard G. + - Tomasz Gajc + - Bjarni Ingi Gislason + - John Paul Adrian Glaubitz + - Bill Glessner + - Matthew Good + - Michał Górny + - Jason Gorski + - Alexander M. Greenham + - Juan Manuel Guerrero + - Gabriela Gutierrez + - Diederik de Haas + - Jan Terje Hansen + - Tobias Lahrmann Hansen + - Joachim Henke + - Lizandro Heredia + - Christian Hesse + - Vincenzo Innocente + - Peter Ivanov + - Nicholas Jackson + - Sam James + - Hajin Jang + - Hans Jansen + - Jouk Jansen + - Jun I Jin + - Christoph Junghans + - Kiyoshi Kanazawa + - Joona Kannisto + - Per Øyvind Karlsen + - Firas Khalil Khana + - Iouri Kharon + - Kim Jinyeong + - Thomas Klausner + - Richard Koch + - Anton Kochkov + - Harri K. Koskinen + - Ville Koskinen + - Sergey Kosukhin + - Marcin Kowalczyk + - Jan Kratochvil + - Christian Kujau + - Stephan Kulow + - Ilya Kurdyukov + - Peter Lawler + - James M Leddy + - Kelvin Lee + - Vincent Lefevre + - Hin-Tak Leung + - Andraž 'ruskie' Levstik + - Cary Lewis + - Wim Lewis + - Xin Li + - Yifeng Li + - Eric Lindblad + - Lorenzo De Liso + - H.J. Lu + - Bela Lubkin + - Chenxi Mao + - Gregory Margo + - Julien Marrec + - Pierre-Yves Martin + - Ed Maste + - Martin Matuška + - Scott McAllister + - Chris McCrohan + - Derwin McGeary + - Ivan A. Melnikov + - Jim Meyering + - Arkadiusz Miskiewicz + - Nathan Moinvaziri + - Étienne Mollier + - Conley Moorhous + - Dirk Müller + - Rainer Müller + - Andrew Murray + - Rafał Mużyło + - Adrien Nader + - Evan Nemerson + - Alexander Neumann + - Hongbo Ni + - Jonathan Nieder + - Asgeir Storesund Nilsen + - Andre Noll + - Ruarí Ødegaard + - Peter O'Gorman + - Dimitri Papadopoulos Orfanos + - Daniel Packard + - Filip Palian + - Peter Pallinger + - Kai Pastor + - Keith Patton + - Rui Paulo + - Igor Pavlov + - Diego Elio Pettenò + - Elbert Pol + - Guiorgy Potskhishvili + - Mikko Pouru + - Frank Prochnow + - Rich Prohaska + - Trần Ngọc Quân + - Pavel Raiskup + - Matthieu Rakotojaona + - Ole André Vadla Ravnås + - Eric S. Raymond + - Robert Readman + - Bernhard Reutner-Fischer + - Markus Rickert + - Cristian Rodríguez + - Jeroen Roovers + - Christian von Roques + - Boud Roukema + - Torsten Rupp + - Stephen Sachs + - Jukka Salmi + - Agostino Sarubbo + - Vijay Sarvepalli + - Alexandre Sauvé + - Benno Schulenberg + - Andreas Schwab + - Eli Schwartz + - Peter Seiderer + - Bhargava Shastry + - Dan Shechter + - Stuart Shelton + - Sebastian Andrzej Siewior + - Andrej Skenderija + - Ville Skyttä + - Brad Smith + - Bruce Stark + - Pippijn van Steenhoven + - Tobias Stoeckmann + - Martin Storsjö + - Jonathan Stott + - Dan Stromberg + - Douglas Thor + - Vincent Torri + - Alexey Tourbin + - Paul Townsend + - Mohammed Adnène Trojette + - Orange Tsai + - Taiki Tsunekawa + - Mathieu Vachon + - Maksym Vatsyk + - Loganaden Velvindron + - Patrick J. Volkerding + - Martin Väth + - Adam Walling + - Jeffrey Walton + - Christian Weisgerber + - Dan Weiss + - Bert Wesarg + - Mark Wielaard + - Fredrik Wikstrom + - Jim Wilcoxson + - Ralf Wildenhues + - Charles Wilson + - Lars Wirzenius + - Vincent Wixsom + - Pilorz Wojciech + - Chien Wong + - Xi Ruoyao + - Ryan Young + - Andreas Zieringer + - 榆柳松 (ZhengSen Wang) + +Companies: + - Google + - Sandfly Security + +Other credits: + - cleemy desu wayo working with Trend Micro Zero Day Initiative + - Orange Tsai and splitline from DEVCORE Research Team + +Also thanks to all the people who have participated in the Tukaani project. + +I have probably forgot to add some names to the above list. Sorry about +that and thanks for your help. + diff --git a/src/common/common_w32res.rc b/src/common/common_w32res.rc new file mode 100644 index 0000000000..8114ee311e --- /dev/null +++ b/src/common/common_w32res.rc @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: 0BSD */ + +/* + * Author: Lasse Collin + */ + +#include +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif +#define LZMA_H_INTERNAL +#define LZMA_H_INTERNAL_RC +#include "lzma/version.h" + +#ifndef MY_BUILD +# define MY_BUILD 0 +#endif +#define MY_VERSION LZMA_VERSION_MAJOR,LZMA_VERSION_MINOR,LZMA_VERSION_PATCH,MY_BUILD + +#define MY_FILENAME MY_NAME MY_SUFFIX +#define MY_COMPANY "The Tukaani Project " +#define MY_PRODUCT PACKAGE_NAME " <" PACKAGE_URL ">" + +LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US + +VS_VERSION_INFO VERSIONINFO +FILEVERSION MY_VERSION +PRODUCTVERSION MY_VERSION +FILEFLAGSMASK VS_FFI_FILEFLAGSMASK +FILEFLAGS 0 +FILEOS VOS_NT_WINDOWS32 +FILETYPE MY_TYPE +FILESUBTYPE 0x0L +BEGIN + BLOCK "StringFileInfo" + BEGIN + BLOCK "040904b0" + BEGIN + VALUE "CompanyName", MY_COMPANY + VALUE "FileDescription", MY_DESC + VALUE "FileVersion", LZMA_VERSION_STRING + VALUE "InternalName", MY_NAME + VALUE "OriginalFilename", MY_FILENAME + VALUE "ProductName", MY_PRODUCT + VALUE "ProductVersion", LZMA_VERSION_STRING + END + END + BLOCK "VarFileInfo" + BEGIN + VALUE "Translation", 0x409, 1200 + END +END + +/* Omit the manifest on Cygwin and MSYS2 (both define __CYGWIN__). */ +#if MY_TYPE == VFT_APP && !defined(__CYGWIN__) +CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST "w32_application.manifest" +#endif diff --git a/src/common/my_landlock.h b/src/common/my_landlock.h new file mode 100644 index 0000000000..e135d08c85 --- /dev/null +++ b/src/common/my_landlock.h @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file my_landlock.h +/// \brief Linux Landlock sandbox helper functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef MY_LANDLOCK_H +#define MY_LANDLOCK_H + +#include "sysdefs.h" + +#include +#include +#include + + +/// \brief Initialize Landlock ruleset attributes to forbid everything +/// +/// The supported Landlock ABI is checked at runtime and only the supported +/// actions are forbidden in the attributes. Thus, if the attributes are +/// used with my_landlock_create_ruleset(), it shouldn't fail. +/// +/// \return On success, the Landlock ABI version is returned (a positive +/// integer). If Landlock isn't supported, -1 is returned. +static int +my_landlock_ruleset_attr_forbid_all(struct landlock_ruleset_attr *attr) +{ + memzero(attr, sizeof(*attr)); + + const int abi_version = syscall(SYS_landlock_create_ruleset, + (void *)NULL, 0, LANDLOCK_CREATE_RULESET_VERSION); + if (abi_version <= 0) + return -1; + + // ABI 1 except the few at the end + attr->handled_access_fs + = LANDLOCK_ACCESS_FS_EXECUTE + | LANDLOCK_ACCESS_FS_WRITE_FILE + | LANDLOCK_ACCESS_FS_READ_FILE + | LANDLOCK_ACCESS_FS_READ_DIR + | LANDLOCK_ACCESS_FS_REMOVE_DIR + | LANDLOCK_ACCESS_FS_REMOVE_FILE + | LANDLOCK_ACCESS_FS_MAKE_CHAR + | LANDLOCK_ACCESS_FS_MAKE_DIR + | LANDLOCK_ACCESS_FS_MAKE_REG + | LANDLOCK_ACCESS_FS_MAKE_SOCK + | LANDLOCK_ACCESS_FS_MAKE_FIFO + | LANDLOCK_ACCESS_FS_MAKE_BLOCK + | LANDLOCK_ACCESS_FS_MAKE_SYM +#ifdef LANDLOCK_ACCESS_FS_REFER + | LANDLOCK_ACCESS_FS_REFER // ABI 2 +#endif +#ifdef LANDLOCK_ACCESS_FS_TRUNCATE + | LANDLOCK_ACCESS_FS_TRUNCATE // ABI 3 +#endif +#ifdef LANDLOCK_ACCESS_FS_IOCTL_DEV + | LANDLOCK_ACCESS_FS_IOCTL_DEV // ABI 5 +#endif + ; + +#ifdef LANDLOCK_ACCESS_NET_BIND_TCP + // ABI 4 + attr->handled_access_net + = LANDLOCK_ACCESS_NET_BIND_TCP + | LANDLOCK_ACCESS_NET_CONNECT_TCP; +#endif + +#ifdef LANDLOCK_SCOPE_SIGNAL + // ABI 6 + attr->scoped + = LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET + | LANDLOCK_SCOPE_SIGNAL; +#endif + + // Disable flags that require a new ABI version. + switch (abi_version) { + case 1: +#ifdef LANDLOCK_ACCESS_FS_REFER + attr->handled_access_fs &= ~LANDLOCK_ACCESS_FS_REFER; +#endif + FALLTHROUGH; + + case 2: +#ifdef LANDLOCK_ACCESS_FS_TRUNCATE + attr->handled_access_fs &= ~LANDLOCK_ACCESS_FS_TRUNCATE; +#endif + FALLTHROUGH; + + case 3: +#ifdef LANDLOCK_ACCESS_NET_BIND_TCP + attr->handled_access_net = 0; +#endif + FALLTHROUGH; + + case 4: +#ifdef LANDLOCK_ACCESS_FS_IOCTL_DEV + attr->handled_access_fs &= ~LANDLOCK_ACCESS_FS_IOCTL_DEV; +#endif + FALLTHROUGH; + + case 5: +#ifdef LANDLOCK_SCOPE_SIGNAL + attr->scoped = 0; +#endif + FALLTHROUGH; + + default: + // We only know about the features of the ABIs 1-6. + break; + } + + return abi_version; +} + + +/// \brief Wrapper for the landlock_create_ruleset(2) syscall +/// +/// Syscall wrappers provide argument type checking. +/// +/// \note Remember to call `prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)` too! +static inline int +my_landlock_create_ruleset(const struct landlock_ruleset_attr *attr, + size_t size, uint32_t flags) +{ + return syscall(SYS_landlock_create_ruleset, attr, size, flags); +} + + +/// \brief Wrapper for the landlock_restrict_self(2) syscall +static inline int +my_landlock_restrict_self(int ruleset_fd, uint32_t flags) +{ + return syscall(SYS_landlock_restrict_self, ruleset_fd, flags); +} + +#endif diff --git a/src/common/mythread.h b/src/common/mythread.h new file mode 100644 index 0000000000..10ea2d42c2 --- /dev/null +++ b/src/common/mythread.h @@ -0,0 +1,548 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file mythread.h +/// \brief Some threading related helper macros and functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef MYTHREAD_H +#define MYTHREAD_H + +#include "sysdefs.h" + +// If any type of threading is enabled, #define MYTHREAD_ENABLED. +#if defined(MYTHREAD_POSIX) || defined(MYTHREAD_WIN95) \ + || defined(MYTHREAD_VISTA) +# define MYTHREAD_ENABLED 1 +#endif + + +#ifdef MYTHREAD_ENABLED + +//////////////////////////////////////// +// Shared between all threading types // +//////////////////////////////////////// + +// Locks a mutex for a duration of a block. +// +// Perform mythread_mutex_lock(&mutex) in the beginning of a block +// and mythread_mutex_unlock(&mutex) at the end of the block. "break" +// may be used to unlock the mutex and jump out of the block. +// mythread_sync blocks may be nested. +// +// Example: +// +// mythread_sync(mutex) { +// foo(); +// if (some_error) +// break; // Skips bar() +// bar(); +// } +// +// At least GCC optimizes the loops completely away so it doesn't slow +// things down at all compared to plain mythread_mutex_lock(&mutex) +// and mythread_mutex_unlock(&mutex) calls. +// +#define mythread_sync(mutex) mythread_sync_helper1(mutex, __LINE__) +#define mythread_sync_helper1(mutex, line) mythread_sync_helper2(mutex, line) +#define mythread_sync_helper2(mutex, line) \ + for (unsigned int mythread_i_ ## line = 0; \ + mythread_i_ ## line \ + ? (mythread_mutex_unlock(&(mutex)), 0) \ + : (mythread_mutex_lock(&(mutex)), 1); \ + mythread_i_ ## line = 1) \ + for (unsigned int mythread_j_ ## line = 0; \ + !mythread_j_ ## line; \ + mythread_j_ ## line = 1) +#endif + + +#if !defined(MYTHREAD_ENABLED) + +////////////////// +// No threading // +////////////////// + +// Calls the given function once. This isn't thread safe. +#define mythread_once(func) \ +do { \ + static bool once_ = false; \ + if (!once_) { \ + func(); \ + once_ = true; \ + } \ +} while (0) + + +#if !(defined(_WIN32) && !defined(__CYGWIN__)) && !defined(__wasm__) +// Use sigprocmask() to set the signal mask in single-threaded programs. +#include + +static inline void +mythread_sigmask(int how, const sigset_t *restrict set, + sigset_t *restrict oset) +{ + int ret = sigprocmask(how, set, oset); + assert(ret == 0); + (void)ret; +} +#endif + + +#elif defined(MYTHREAD_POSIX) + +//////////////////// +// Using pthreads // +//////////////////// + +#include +#include +#include +#include + +// If clock_gettime() isn't available, use gettimeofday() from +// as a fallback. gettimeofday() is in SUSv2 and thus is supported on all +// relevant POSIX systems. +#ifndef HAVE_CLOCK_GETTIME +# include +#endif + +// MinGW-w64 with winpthreads: +// +// NOTE: Typical builds with MinGW-w64 don't use this code (MYTHREAD_POSIX). +// Instead, native Windows threading APIs are used (MYTHREAD_VISTA or +// MYTHREAD_WIN95). +// +// MinGW-w64 has _sigset_t (an integer type) in . +// If _POSIX was #defined, the header would add the alias sigset_t too. +// Let's keep this working even without _POSIX. +// +// There are no functions that actually do something with sigset_t +// because signals barely exist on Windows. The sigfillset macro below +// is just to silence warnings. There is no sigfillset() in MinGW-w64. +#ifdef __MINGW32__ +# include +# define sigset_t _sigset_t +# define sigfillset(set_ptr) do { *(set_ptr) = 0; } while (0) +#endif + +#define MYTHREAD_RET_TYPE void * +#define MYTHREAD_RET_VALUE NULL + +typedef pthread_t mythread; +typedef pthread_mutex_t mythread_mutex; + +typedef struct { + pthread_cond_t cond; +#ifdef HAVE_CLOCK_GETTIME + // Clock ID (CLOCK_REALTIME or CLOCK_MONOTONIC) associated with + // the condition variable. + clockid_t clk_id; +#endif +} mythread_cond; + +typedef struct timespec mythread_condtime; + + +// Calls the given function once in a thread-safe way. +#define mythread_once(func) \ + do { \ + static pthread_once_t once_ = PTHREAD_ONCE_INIT; \ + pthread_once(&once_, &func); \ + } while (0) + + +// Use pthread_sigmask() to set the signal mask in multi-threaded programs. +// Do nothing on OpenVMS since it lacks pthread_sigmask(). +// Do nothing on MinGW-w64 too to silence warnings (its pthread_sigmask() +// is #defined to 0 so it's a no-op). +static inline void +mythread_sigmask(int how, const sigset_t *restrict set, + sigset_t *restrict oset) +{ +#if defined(__VMS) || defined(__MINGW32__) + (void)how; + (void)set; + (void)oset; +#else + int ret = pthread_sigmask(how, set, oset); + assert(ret == 0); + (void)ret; +#endif +} + + +// Creates a new thread with all signals blocked. Returns zero on success +// and non-zero on error. +static inline int +mythread_create(mythread *thread, void *(*func)(void *arg), void *arg) +{ + sigset_t old; + sigset_t all; + sigfillset(&all); + + mythread_sigmask(SIG_SETMASK, &all, &old); + const int ret = pthread_create(thread, NULL, func, arg); + mythread_sigmask(SIG_SETMASK, &old, NULL); + + return ret; +} + +// Joins a thread. Returns zero on success and non-zero on error. +static inline int +mythread_join(mythread thread) +{ + return pthread_join(thread, NULL); +} + + +// Initializes a mutex. Returns zero on success and non-zero on error. +static inline int +mythread_mutex_init(mythread_mutex *mutex) +{ + return pthread_mutex_init(mutex, NULL); +} + +static inline void +mythread_mutex_destroy(mythread_mutex *mutex) +{ + int ret = pthread_mutex_destroy(mutex); + assert(ret == 0); + (void)ret; +} + +static inline void +mythread_mutex_lock(mythread_mutex *mutex) +{ + int ret = pthread_mutex_lock(mutex); + assert(ret == 0); + (void)ret; +} + +static inline void +mythread_mutex_unlock(mythread_mutex *mutex) +{ + int ret = pthread_mutex_unlock(mutex); + assert(ret == 0); + (void)ret; +} + + +// Initializes a condition variable. +// +// Using CLOCK_MONOTONIC instead of the default CLOCK_REALTIME makes the +// timeout in pthread_cond_timedwait() work correctly also if system time +// is suddenly changed. Unfortunately CLOCK_MONOTONIC isn't available +// everywhere while the default CLOCK_REALTIME is, so the default is +// used if CLOCK_MONOTONIC isn't available. +// +// If clock_gettime() isn't available at all, gettimeofday() will be used. +static inline int +mythread_cond_init(mythread_cond *mycond) +{ +#ifdef HAVE_CLOCK_GETTIME +# if defined(HAVE_PTHREAD_CONDATTR_SETCLOCK) && \ + defined(HAVE_CLOCK_MONOTONIC) + struct timespec ts; + pthread_condattr_t condattr; + + // POSIX doesn't seem to *require* that pthread_condattr_setclock() + // will fail if given an unsupported clock ID. Test that + // CLOCK_MONOTONIC really is supported using clock_gettime(). + if (clock_gettime(CLOCK_MONOTONIC, &ts) == 0 + && pthread_condattr_init(&condattr) == 0) { + int ret = pthread_condattr_setclock( + &condattr, CLOCK_MONOTONIC); + if (ret == 0) + ret = pthread_cond_init(&mycond->cond, &condattr); + + pthread_condattr_destroy(&condattr); + + if (ret == 0) { + mycond->clk_id = CLOCK_MONOTONIC; + return 0; + } + } + + // If anything above fails, fall back to the default CLOCK_REALTIME. + // POSIX requires that all implementations of clock_gettime() must + // support at least CLOCK_REALTIME. +# endif + + mycond->clk_id = CLOCK_REALTIME; +#endif + + return pthread_cond_init(&mycond->cond, NULL); +} + +static inline void +mythread_cond_destroy(mythread_cond *cond) +{ + int ret = pthread_cond_destroy(&cond->cond); + assert(ret == 0); + (void)ret; +} + +static inline void +mythread_cond_signal(mythread_cond *cond) +{ + int ret = pthread_cond_signal(&cond->cond); + assert(ret == 0); + (void)ret; +} + +static inline void +mythread_cond_wait(mythread_cond *cond, mythread_mutex *mutex) +{ + int ret = pthread_cond_wait(&cond->cond, mutex); + assert(ret == 0); + (void)ret; +} + +// Waits on a condition or until a timeout expires. If the timeout expires, +// non-zero is returned, otherwise zero is returned. +static inline int +mythread_cond_timedwait(mythread_cond *cond, mythread_mutex *mutex, + const mythread_condtime *condtime) +{ + int ret = pthread_cond_timedwait(&cond->cond, mutex, condtime); + assert(ret == 0 || ret == ETIMEDOUT); + return ret; +} + +// Sets condtime to the absolute time that is timeout_ms milliseconds +// in the future. The type of the clock to use is taken from cond. +static inline void +mythread_condtime_set(mythread_condtime *condtime, const mythread_cond *cond, + uint32_t timeout_ms) +{ + condtime->tv_sec = (time_t)(timeout_ms / 1000); + condtime->tv_nsec = (long)((timeout_ms % 1000) * 1000000); + +#ifdef HAVE_CLOCK_GETTIME + struct timespec now; + int ret = clock_gettime(cond->clk_id, &now); + assert(ret == 0); + (void)ret; + + condtime->tv_sec += now.tv_sec; + condtime->tv_nsec += now.tv_nsec; +#else + (void)cond; + + struct timeval now; + gettimeofday(&now, NULL); + + condtime->tv_sec += now.tv_sec; + condtime->tv_nsec += now.tv_usec * 1000L; +#endif + + // tv_nsec must stay in the range [0, 999_999_999]. + if (condtime->tv_nsec >= 1000000000L) { + condtime->tv_nsec -= 1000000000L; + ++condtime->tv_sec; + } +} + + +#elif defined(MYTHREAD_WIN95) || defined(MYTHREAD_VISTA) + +///////////////////// +// Windows threads // +///////////////////// + +#define WIN32_LEAN_AND_MEAN +#ifdef MYTHREAD_VISTA +# undef _WIN32_WINNT +# define _WIN32_WINNT 0x0600 +#endif +#include +#include + +#define MYTHREAD_RET_TYPE unsigned int __stdcall +#define MYTHREAD_RET_VALUE 0 + +typedef HANDLE mythread; +typedef CRITICAL_SECTION mythread_mutex; + +#ifdef MYTHREAD_WIN95 +typedef HANDLE mythread_cond; +#else +typedef CONDITION_VARIABLE mythread_cond; +#endif + +typedef struct { + // Tick count (milliseconds) in the beginning of the timeout. + // NOTE: This is 32 bits so it wraps around after 49.7 days. + // Multi-day timeouts may not work as expected. + DWORD start; + + // Length of the timeout in milliseconds. The timeout expires + // when the current tick count minus "start" is equal or greater + // than "timeout". + DWORD timeout; +} mythread_condtime; + + +// mythread_once() is only available with Vista threads. +#ifdef MYTHREAD_VISTA +#define mythread_once(func) \ + do { \ + static INIT_ONCE once_ = INIT_ONCE_STATIC_INIT; \ + BOOL pending_; \ + if (!InitOnceBeginInitialize(&once_, 0, &pending_, NULL)) \ + abort(); \ + if (pending_) { \ + func(); \ + if (!InitOnceComplete(&once_, 0, NULL)) \ + abort(); \ + } \ + } while (0) +#endif + + +// mythread_sigmask() isn't available on Windows. Even a dummy version would +// make no sense because the other POSIX signal functions are missing anyway. + + +static inline int +mythread_create(mythread *thread, + unsigned int (__stdcall *func)(void *arg), void *arg) +{ + uintptr_t ret = _beginthreadex(NULL, 0, func, arg, 0, NULL); + if (ret == 0) + return -1; + + *thread = (HANDLE)ret; + return 0; +} + +static inline int +mythread_join(mythread thread) +{ + int ret = 0; + + if (WaitForSingleObject(thread, INFINITE) != WAIT_OBJECT_0) + ret = -1; + + if (!CloseHandle(thread)) + ret = -1; + + return ret; +} + + +static inline int +mythread_mutex_init(mythread_mutex *mutex) +{ + InitializeCriticalSection(mutex); + return 0; +} + +static inline void +mythread_mutex_destroy(mythread_mutex *mutex) +{ + DeleteCriticalSection(mutex); +} + +static inline void +mythread_mutex_lock(mythread_mutex *mutex) +{ + EnterCriticalSection(mutex); +} + +static inline void +mythread_mutex_unlock(mythread_mutex *mutex) +{ + LeaveCriticalSection(mutex); +} + + +static inline int +mythread_cond_init(mythread_cond *cond) +{ +#ifdef MYTHREAD_WIN95 + *cond = CreateEvent(NULL, FALSE, FALSE, NULL); + return *cond == NULL ? -1 : 0; +#else + InitializeConditionVariable(cond); + return 0; +#endif +} + +static inline void +mythread_cond_destroy(mythread_cond *cond) +{ +#ifdef MYTHREAD_WIN95 + CloseHandle(*cond); +#else + (void)cond; +#endif +} + +static inline void +mythread_cond_signal(mythread_cond *cond) +{ +#ifdef MYTHREAD_WIN95 + SetEvent(*cond); +#else + WakeConditionVariable(cond); +#endif +} + +static inline void +mythread_cond_wait(mythread_cond *cond, mythread_mutex *mutex) +{ +#ifdef MYTHREAD_WIN95 + LeaveCriticalSection(mutex); + WaitForSingleObject(*cond, INFINITE); + EnterCriticalSection(mutex); +#else + BOOL ret = SleepConditionVariableCS(cond, mutex, INFINITE); + assert(ret); + (void)ret; +#endif +} + +static inline int +mythread_cond_timedwait(mythread_cond *cond, mythread_mutex *mutex, + const mythread_condtime *condtime) +{ +#ifdef MYTHREAD_WIN95 + LeaveCriticalSection(mutex); +#endif + + DWORD elapsed = GetTickCount() - condtime->start; + DWORD timeout = elapsed >= condtime->timeout + ? 0 : condtime->timeout - elapsed; + +#ifdef MYTHREAD_WIN95 + DWORD ret = WaitForSingleObject(*cond, timeout); + assert(ret == WAIT_OBJECT_0 || ret == WAIT_TIMEOUT); + + EnterCriticalSection(mutex); + + return ret == WAIT_TIMEOUT; +#else + BOOL ret = SleepConditionVariableCS(cond, mutex, timeout); + assert(ret || GetLastError() == ERROR_TIMEOUT); + return !ret; +#endif +} + +static inline void +mythread_condtime_set(mythread_condtime *condtime, const mythread_cond *cond, + uint32_t timeout) +{ + (void)cond; + condtime->start = GetTickCount(); + condtime->timeout = timeout; +} + +#endif + +#endif diff --git a/src/common/sysdefs.h b/src/common/sysdefs.h new file mode 100644 index 0000000000..b10ffa7c3b --- /dev/null +++ b/src/common/sysdefs.h @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file sysdefs.h +/// \brief Common includes, definitions, system-specific things etc. +/// +/// This file is used also by the lzma command line tool, that's why this +/// file is separate from common.h. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_SYSDEFS_H +#define LZMA_SYSDEFS_H + +////////////// +// Includes // +////////////// + +#ifdef HAVE_CONFIG_H +# include +#endif + +// Choose if MinGW-w64's stdio replacement functions should be used. +// The default has varied slightly in the past so it's clearest to always +// set it explicitly. +// +// Modern MinGW-w64 enables the replacement functions even with UCRT +// when _GNU_SOURCE is defined. That's good because UCRT doesn't support +// the POSIX thousand separator flag in printf (like "%'u"). Otherwise +// XZ Utils works with the UCRT stdio functions. +// +// The replacement functions add over 20 KiB to each executable. For +// size-optimized builds (HAVE_SMALL), disable the replacements. +// Then thousand separators aren't shown in xz's messages but this is +// a minor downside compare to the slower speed of the HAVE_SMALL builds. +// +// The legacy MSVCRT is pre-C99 and it's best to always use the stdio +// replacements functions from MinGW-w64. +#if defined(__MINGW32__) && !defined(__USE_MINGW_ANSI_STDIO) +# define __USE_MINGW_ANSI_STDIO 1 +# include <_mingw.h> +# if defined(_UCRT) && defined(HAVE_SMALL) +# undef __USE_MINGW_ANSI_STDIO +# define __USE_MINGW_ANSI_STDIO 0 +# endif +#endif + +// size_t and NULL +#include + +#ifdef HAVE_INTTYPES_H +# include +#endif + +// C99 says that inttypes.h always includes stdint.h, but some systems +// don't do that, and require including stdint.h separately. +#ifdef HAVE_STDINT_H +# include +#endif + +// Some pre-C99 systems have SIZE_MAX in limits.h instead of stdint.h. The +// limits are also used to figure out some macros missing from pre-C99 systems. +#include + +// Be more compatible with systems that have non-conforming inttypes.h. +// We assume that int is 32-bit and that long is either 32-bit or 64-bit. +// Full Autoconf test could be more correct, but this should work well enough. +// Note that this duplicates some code from lzma.h, but this is better since +// we can work without inttypes.h thanks to Autoconf tests. +#ifndef UINT32_C +# if UINT_MAX != 4294967295U +# error UINT32_C is not defined and unsigned int is not 32-bit. +# endif +# define UINT32_C(n) n ## U +#endif +#ifndef UINT32_MAX +# define UINT32_MAX UINT32_C(4294967295) +#endif +#ifndef PRIu32 +# define PRIu32 "u" +#endif +#ifndef PRIx32 +# define PRIx32 "x" +#endif +#ifndef PRIX32 +# define PRIX32 "X" +#endif + +#if ULONG_MAX == 4294967295UL +# ifndef UINT64_C +# define UINT64_C(n) n ## ULL +# endif +# ifndef PRIu64 +# define PRIu64 "llu" +# endif +# ifndef PRIx64 +# define PRIx64 "llx" +# endif +# ifndef PRIX64 +# define PRIX64 "llX" +# endif +#else +# ifndef UINT64_C +# define UINT64_C(n) n ## UL +# endif +# ifndef PRIu64 +# define PRIu64 "lu" +# endif +# ifndef PRIx64 +# define PRIx64 "lx" +# endif +# ifndef PRIX64 +# define PRIX64 "lX" +# endif +#endif +#ifndef UINT64_MAX +# define UINT64_MAX UINT64_C(18446744073709551615) +#endif + +// Incorrect(?) SIZE_MAX: +// - Interix headers typedef size_t to unsigned long, +// but a few lines later define SIZE_MAX to INT32_MAX. +// - SCO OpenServer (x86) headers typedef size_t to unsigned int +// but define SIZE_MAX to INT32_MAX. +#if defined(__INTERIX) || defined(_SCO_DS) +# undef SIZE_MAX +#endif + +// The code currently assumes that size_t is either 32-bit or 64-bit. +#ifndef SIZE_MAX +# if SIZEOF_SIZE_T == 4 +# define SIZE_MAX UINT32_MAX +# elif SIZEOF_SIZE_T == 8 +# define SIZE_MAX UINT64_MAX +# else +# error size_t is not 32-bit or 64-bit +# endif +#endif +#if SIZE_MAX != UINT32_MAX && SIZE_MAX != UINT64_MAX +# error size_t is not 32-bit or 64-bit +#endif + +#include +#include + +// Pre-C99 systems lack stdbool.h. All the code in XZ Utils must be written +// so that it works with fake bool type, for example: +// +// bool foo = (flags & 0x100) != 0; +// bool bar = !!(flags & 0x100); +// +// This works with the real C99 bool but breaks with fake bool: +// +// bool baz = (flags & 0x100); +// +#ifdef HAVE_STDBOOL_H +# include +#else +# if ! HAVE__BOOL +typedef unsigned char _Bool; +# endif +# define bool _Bool +# define false 0 +# define true 1 +# define __bool_true_false_are_defined 1 +#endif + +// We may need alignas from C11/C17/C23. +#if __STDC_VERSION__ >= 202311 + // alignas is a keyword in C23. Do nothing. +#elif __STDC_VERSION__ >= 201112 + // Oracle Developer Studio 12.6 lacks . + // For simplicity, avoid the header with all C11/C17 compilers. +# define alignas _Alignas +#elif defined(__GNUC__) || defined(__clang__) +# define alignas(n) __attribute__((__aligned__(n))) +#else +# define alignas(n) +#endif + +#include + +// MSVC v19.00 (VS 2015 version 14.0) and later should work. +// +// MSVC v19.27 (VS 2019 version 16.7) added support for restrict. +// Older ones support only __restrict. +#ifdef _MSC_VER +# if _MSC_VER < 1927 && !defined(restrict) +# define restrict __restrict +# endif +#endif + +//////////// +// Macros // +//////////// + +#undef memzero +#define memzero(s, n) memset(s, 0, n) + +// NOTE: Avoid using MIN() and MAX(), because even conditionally defining +// those macros can cause some portability trouble, since on some systems +// the system headers insist defining their own versions. +#define my_min(x, y) ((x) < (y) ? (x) : (y)) +#define my_max(x, y) ((x) > (y) ? (x) : (y)) + +#ifndef ARRAY_SIZE +# define ARRAY_SIZE(array) (sizeof(array) / sizeof((array)[0])) +#endif + +#if defined(__GNUC__) \ + && ((__GNUC__ == 4 && __GNUC_MINOR__ >= 3) || __GNUC__ > 4) +# define lzma_attr_alloc_size(x) __attribute__((__alloc_size__(x))) +#else +# define lzma_attr_alloc_size(x) +#endif + +#if __STDC_VERSION__ >= 202311 +# define FALLTHROUGH [[__fallthrough__]] +#elif (defined(__GNUC__) && __GNUC__ >= 7) \ + || (defined(__clang_major__) && __clang_major__ >= 10) +# define FALLTHROUGH __attribute__((__fallthrough__)) +#else +# define FALLTHROUGH ((void)0) +#endif + +#endif diff --git a/src/common/tuklib_common.h b/src/common/tuklib_common.h new file mode 100644 index 0000000000..d73f07255e --- /dev/null +++ b/src/common/tuklib_common.h @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_common.h +/// \brief Common definitions for tuklib modules +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_COMMON_H +#define TUKLIB_COMMON_H + +// The config file may be replaced by a package-specific file. +// It should include at least stddef.h, stdbool.h, inttypes.h, and limits.h. +#include "tuklib_config.h" + +// TUKLIB_SYMBOL_PREFIX is prefixed to all symbols exported by +// the tuklib modules. If you use a tuklib module in a library, +// you should use TUKLIB_SYMBOL_PREFIX to make sure that there +// are no symbol conflicts in case someone links your library +// into application that also uses the same tuklib module. +#ifndef TUKLIB_SYMBOL_PREFIX +# define TUKLIB_SYMBOL_PREFIX +#endif + +#define TUKLIB_CAT_X(a, b) a ## b +#define TUKLIB_CAT(a, b) TUKLIB_CAT_X(a, b) + +#ifndef TUKLIB_SYMBOL +# define TUKLIB_SYMBOL(sym) TUKLIB_CAT(TUKLIB_SYMBOL_PREFIX, sym) +#endif + +#ifndef TUKLIB_DECLS_BEGIN +# ifdef __cplusplus +# define TUKLIB_DECLS_BEGIN extern "C" { +# else +# define TUKLIB_DECLS_BEGIN +# endif +#endif + +#ifndef TUKLIB_DECLS_END +# ifdef __cplusplus +# define TUKLIB_DECLS_END } +# else +# define TUKLIB_DECLS_END +# endif +#endif + +#if defined(__GNUC__) && defined(__GNUC_MINOR__) +# define TUKLIB_GNUC_REQ(major, minor) \ + ((__GNUC__ == (major) && __GNUC_MINOR__ >= (minor)) \ + || __GNUC__ > (major)) +#else +# define TUKLIB_GNUC_REQ(major, minor) 0 +#endif + +#if defined(__GNUC__) || defined(__clang__) +# define tuklib_attr_format_printf(fmt_index, args_index) \ + __attribute__((__format__(__printf__, fmt_index, args_index))) +#else +# define tuklib_attr_format_printf(fmt_index, args_index) +#endif + +// tuklib_attr_noreturn attribute is used to mark functions as non-returning. +// We cannot use "noreturn" as the macro name because then C23 code that +// uses [[noreturn]] would break as it would expand to [[ [[noreturn]] ]]. +// +// tuklib_attr_noreturn must be used at the beginning of function declaration +// to work in all cases. The [[noreturn]] syntax is the most limiting, it +// must be even before any GNU C's __attribute__ keywords: +// +// tuklib_attr_noreturn +// __attribute__((nonnull(1))) +// extern void foo(const char *s); +// +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 202311 +# define tuklib_attr_noreturn [[noreturn]] +#elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112 +# define tuklib_attr_noreturn _Noreturn +#elif TUKLIB_GNUC_REQ(2, 5) +# define tuklib_attr_noreturn __attribute__((__noreturn__)) +#elif defined(_MSC_VER) +# define tuklib_attr_noreturn __declspec(noreturn) +#else +# define tuklib_attr_noreturn +#endif + +#if (defined(_WIN32) && !defined(__CYGWIN__)) \ + || defined(__OS2__) || defined(__MSDOS__) +# define TUKLIB_DOSLIKE 1 +#endif + +#endif diff --git a/src/common/tuklib_config.h b/src/common/tuklib_config.h new file mode 100644 index 0000000000..b27251dc27 --- /dev/null +++ b/src/common/tuklib_config.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: 0BSD + +// If config.h isn't available, assume that the headers required by +// tuklib_common.h are available. This is required by crc32_tablegen.c. +#ifdef HAVE_CONFIG_H +# include "sysdefs.h" +#else +# include +# include +# include +# include +#endif diff --git a/src/common/tuklib_cpucores.c b/src/common/tuklib_cpucores.c new file mode 100644 index 0000000000..c4a781ac38 --- /dev/null +++ b/src/common/tuklib_cpucores.c @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_cpucores.c +/// \brief Get the number of CPU cores online +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_cpucores.h" + +#if defined(_WIN32) || defined(__CYGWIN__) +# ifndef _WIN32_WINNT +# define _WIN32_WINNT 0x0500 +# endif +# include + +// glibc >= 2.9 +#elif defined(TUKLIB_CPUCORES_SCHED_GETAFFINITY) +# include + +// FreeBSD +#elif defined(TUKLIB_CPUCORES_CPUSET) +# include +# include + +#elif defined(TUKLIB_CPUCORES_SYSCTL) +# ifdef HAVE_SYS_PARAM_H +# include +# endif +# include + +#elif defined(TUKLIB_CPUCORES_SYSCONF) +# include + +// HP-UX +#elif defined(TUKLIB_CPUCORES_PSTAT_GETDYNAMIC) +# include +# include +#endif + + +extern uint32_t +tuklib_cpucores(void) +{ + uint32_t ret = 0; + +#if defined(_WIN32) || defined(__CYGWIN__) + SYSTEM_INFO sysinfo; + GetSystemInfo(&sysinfo); + ret = sysinfo.dwNumberOfProcessors; + +#elif defined(TUKLIB_CPUCORES_SCHED_GETAFFINITY) + cpu_set_t cpu_mask; + if (sched_getaffinity(0, sizeof(cpu_mask), &cpu_mask) == 0) + ret = (uint32_t)CPU_COUNT(&cpu_mask); + +#elif defined(TUKLIB_CPUCORES_CPUSET) + cpuset_t set; + if (cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_PID, -1, + sizeof(set), &set) == 0) { +# ifdef CPU_COUNT + ret = (uint32_t)CPU_COUNT(&set); +# else + for (unsigned i = 0; i < CPU_SETSIZE; ++i) + if (CPU_ISSET(i, &set)) + ++ret; +# endif + } + +#elif defined(TUKLIB_CPUCORES_SYSCTL) + // On OpenBSD HW_NCPUONLINE tells the number of processor cores that + // are online so it is preferred over HW_NCPU which also counts cores + // that aren't currently available. The number of cores online is + // often less than HW_NCPU because OpenBSD disables simultaneous + // multi-threading (SMT) by default. +# ifdef HW_NCPUONLINE + int name[2] = { CTL_HW, HW_NCPUONLINE }; +# else + int name[2] = { CTL_HW, HW_NCPU }; +# endif + int cpus; + size_t cpus_size = sizeof(cpus); + if (sysctl(name, 2, &cpus, &cpus_size, NULL, 0) != -1 + && cpus_size == sizeof(cpus) && cpus > 0) + ret = (uint32_t)cpus; + +#elif defined(TUKLIB_CPUCORES_SYSCONF) +# ifdef _SC_NPROCESSORS_ONLN + // Most systems + const long cpus = sysconf(_SC_NPROCESSORS_ONLN); +# else + // IRIX + const long cpus = sysconf(_SC_NPROC_ONLN); +# endif + if (cpus > 0) + ret = (uint32_t)cpus; + +#elif defined(TUKLIB_CPUCORES_PSTAT_GETDYNAMIC) + struct pst_dynamic pst; + if (pstat_getdynamic(&pst, sizeof(pst), 1, 0) != -1) + ret = (uint32_t)pst.psd_proc_cnt; +#endif + + return ret; +} diff --git a/src/common/tuklib_cpucores.h b/src/common/tuklib_cpucores.h new file mode 100644 index 0000000000..edff9395e4 --- /dev/null +++ b/src/common/tuklib_cpucores.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_cpucores.h +/// \brief Get the number of CPU cores online +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_CPUCORES_H +#define TUKLIB_CPUCORES_H + +#include "tuklib_common.h" +TUKLIB_DECLS_BEGIN + +#define tuklib_cpucores TUKLIB_SYMBOL(tuklib_cpucores) +extern uint32_t tuklib_cpucores(void); + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_exit.c b/src/common/tuklib_exit.c new file mode 100644 index 0000000000..c84e0f679a --- /dev/null +++ b/src/common/tuklib_exit.c @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_exit.c +/// \brief Close stdout and stderr, and exit +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_common.h" + +#include +#include +#include + +#include "tuklib_gettext.h" +#include "tuklib_progname.h" +#include "tuklib_exit.h" + + +extern void +tuklib_exit(int status, int err_status, int show_error) +{ + if (status != err_status) { + // Close stdout. If something goes wrong, + // print an error message to stderr. + const int ferror_err = ferror(stdout); + const int fclose_err = fclose(stdout); + if (ferror_err || fclose_err) { + status = err_status; + + // If it was fclose() that failed, we have the reason + // in errno. If only ferror() indicated an error, + // we have no idea what the reason was. + if (show_error) + fprintf(stderr, "%s: %s: %s\n", progname, + _("Writing to standard " + "output failed"), + fclose_err ? strerror(errno) + : _("Unknown error")); + } + } + + if (status != err_status) { + // Close stderr. If something goes wrong, there's + // nothing where we could print an error message. + // Just set the exit status. + const int ferror_err = ferror(stderr); + const int fclose_err = fclose(stderr); + if (fclose_err || ferror_err) + status = err_status; + } + + exit(status); +} diff --git a/src/common/tuklib_exit.h b/src/common/tuklib_exit.h new file mode 100644 index 0000000000..d4e6b4a7e8 --- /dev/null +++ b/src/common/tuklib_exit.h @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_exit.h +/// \brief Close stdout and stderr, and exit +/// \note Requires tuklib_progname and tuklib_gettext modules +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_EXIT_H +#define TUKLIB_EXIT_H + +#include "tuklib_common.h" +TUKLIB_DECLS_BEGIN + +#define tuklib_exit TUKLIB_SYMBOL(tuklib_exit) +tuklib_attr_noreturn +extern void tuklib_exit(int status, int err_status, int show_error); + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_gettext.h b/src/common/tuklib_gettext.h new file mode 100644 index 0000000000..e5ad5e6f78 --- /dev/null +++ b/src/common/tuklib_gettext.h @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_gettext.h +/// \brief Wrapper for gettext and friends +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_GETTEXT_H +#define TUKLIB_GETTEXT_H + +#include "tuklib_common.h" +#include + +#ifndef TUKLIB_GETTEXT +# ifdef ENABLE_NLS +# define TUKLIB_GETTEXT 1 +# else +# define TUKLIB_GETTEXT 0 +# endif +#endif + +#if TUKLIB_GETTEXT +# include +# define tuklib_gettext_init(package, localedir) \ + do { \ + setlocale(LC_ALL, ""); \ + bindtextdomain(package, localedir); \ + textdomain(package); \ + } while (0) +# define _(msgid) gettext(msgid) +#else +# define tuklib_gettext_init(package, localedir) \ + setlocale(LC_ALL, "") +# define _(msgid) (msgid) +# define ngettext(msgid1, msgid2, n) ((n) == 1 ? (msgid1) : (msgid2)) +#endif +#define N_(msgid) msgid + +// Optional: Strings that are word wrapped using tuklib_mbstr_wrap may be +// marked with W_("foo) in the source code. xgettext can then add a comment +// to all such strings to inform translators. The following option needs to +// be added to XGETTEXT_OPTIONS in po/Makevars or in an equivalent place: +// +// '--keyword=W_:1,"This is word wrapped at spaces. The Unicode character U+00A0 works as a non-breaking space. Tab (\t) is interpret as a zero-width space (the tab itself is not displayed); U+200B is NOT supported. Manual word wrapping with \n is supported but requires care."' +// +// NOTE: The double-quotes in the --keyword argument above must be passed to +// xgettext as is, thus one needs the single-quotes in Makevars. +#define W_(msgid) _(msgid) + +#endif diff --git a/src/common/tuklib_integer.h b/src/common/tuklib_integer.h new file mode 100644 index 0000000000..4026249e54 --- /dev/null +++ b/src/common/tuklib_integer.h @@ -0,0 +1,954 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_integer.h +/// \brief Various integer and bit operations +/// +/// This file provides macros or functions to do some basic integer and bit +/// operations. +/// +/// Native endian inline functions (XX = 16, 32, or 64): +/// - Unaligned native endian reads: readXXne(ptr) +/// - Unaligned native endian writes: writeXXne(ptr, num) +/// - Aligned native endian reads: aligned_readXXne(ptr) +/// - Aligned native endian writes: aligned_writeXXne(ptr, num) +/// +/// Endianness-converting integer operations (these can be macros!) +/// (XX = 16, 32, or 64; Y = b or l): +/// - Byte swapping: byteswapXX(num) +/// - Byte order conversions to/from native (byteswaps if Y isn't +/// the native endianness): convXXYe(num) +/// - Unaligned reads: readXXYe(ptr) +/// - Unaligned writes: writeXXYe(ptr, num) +/// - Aligned reads: aligned_readXXYe(ptr) +/// - Aligned writes: aligned_writeXXYe(ptr, num) +/// +/// Since the above can macros, the arguments should have no side effects +/// because they may be evaluated more than once. +/// +/// Bit scan operations for non-zero 32-bit integers (inline functions): +/// - Bit scan reverse (find highest non-zero bit): bsr32(num) +/// - Count leading zeros: clz32(num) +/// - Count trailing zeros: ctz32(num) +/// - Bit scan forward (simply an alias for ctz32()): bsf32(num) +/// +/// The above bit scan operations return 0-31. If num is zero, +/// the result is undefined. +// +// Authors: Lasse Collin +// Joachim Henke +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_INTEGER_H +#define TUKLIB_INTEGER_H + +#include "tuklib_common.h" +#include + +// Newer Intel C compilers require immintrin.h for _bit_scan_reverse() +// and such functions. +#if defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 1500) +# include +// Only include when it is needed. GCC and Clang can both +// use __builtin's, so we only need Windows instrincs when using MSVC. +// GCC and Clang can set _MSC_VER on Windows, so we need to exclude these +// cases explicitly. +#elif defined(_MSC_VER) && !TUKLIB_GNUC_REQ(3, 4) && !defined(__clang__) +# include +#endif + + +/////////////////// +// Byte swapping // +/////////////////// + +#if defined(HAVE___BUILTIN_BSWAPXX) + // GCC >= 4.8 and Clang +# define byteswap16(num) __builtin_bswap16(num) +# define byteswap32(num) __builtin_bswap32(num) +# define byteswap64(num) __builtin_bswap64(num) + +#elif defined(HAVE_BYTESWAP_H) + // glibc, uClibc, dietlibc +# include +# ifdef HAVE_BSWAP_16 +# define byteswap16(num) bswap_16(num) +# endif +# ifdef HAVE_BSWAP_32 +# define byteswap32(num) bswap_32(num) +# endif +# ifdef HAVE_BSWAP_64 +# define byteswap64(num) bswap_64(num) +# endif + +#elif defined(HAVE_SYS_ENDIAN_H) + // *BSDs and Darwin +# include +# ifdef __OpenBSD__ +# define byteswap16(num) swap16(num) +# define byteswap32(num) swap32(num) +# define byteswap64(num) swap64(num) +# else +# define byteswap16(num) bswap16(num) +# define byteswap32(num) bswap32(num) +# define byteswap64(num) bswap64(num) +# endif + +#elif defined(HAVE_SYS_BYTEORDER_H) + // Solaris +# include +# ifdef BSWAP_16 +# define byteswap16(num) BSWAP_16(num) +# endif +# ifdef BSWAP_32 +# define byteswap32(num) BSWAP_32(num) +# endif +# ifdef BSWAP_64 +# define byteswap64(num) BSWAP_64(num) +# endif +# ifdef BE_16 +# define conv16be(num) BE_16(num) +# endif +# ifdef BE_32 +# define conv32be(num) BE_32(num) +# endif +# ifdef BE_64 +# define conv64be(num) BE_64(num) +# endif +# ifdef LE_16 +# define conv16le(num) LE_16(num) +# endif +# ifdef LE_32 +# define conv32le(num) LE_32(num) +# endif +# ifdef LE_64 +# define conv64le(num) LE_64(num) +# endif +#endif + +#ifndef byteswap16 +# define byteswap16(n) (uint16_t)( \ + (((n) & 0x00FFU) << 8) \ + | (((n) & 0xFF00U) >> 8) \ + ) +#endif + +#ifndef byteswap32 +# define byteswap32(n) (uint32_t)( \ + (((n) & UINT32_C(0x000000FF)) << 24) \ + | (((n) & UINT32_C(0x0000FF00)) << 8) \ + | (((n) & UINT32_C(0x00FF0000)) >> 8) \ + | (((n) & UINT32_C(0xFF000000)) >> 24) \ + ) +#endif + +#ifndef byteswap64 +# define byteswap64(n) (uint64_t)( \ + (((n) & UINT64_C(0x00000000000000FF)) << 56) \ + | (((n) & UINT64_C(0x000000000000FF00)) << 40) \ + | (((n) & UINT64_C(0x0000000000FF0000)) << 24) \ + | (((n) & UINT64_C(0x00000000FF000000)) << 8) \ + | (((n) & UINT64_C(0x000000FF00000000)) >> 8) \ + | (((n) & UINT64_C(0x0000FF0000000000)) >> 24) \ + | (((n) & UINT64_C(0x00FF000000000000)) >> 40) \ + | (((n) & UINT64_C(0xFF00000000000000)) >> 56) \ + ) +#endif + +// Define conversion macros using the basic byte swapping macros. +#ifdef WORDS_BIGENDIAN +# ifndef conv16be +# define conv16be(num) ((uint16_t)(num)) +# endif +# ifndef conv32be +# define conv32be(num) ((uint32_t)(num)) +# endif +# ifndef conv64be +# define conv64be(num) ((uint64_t)(num)) +# endif +# ifndef conv16le +# define conv16le(num) byteswap16(num) +# endif +# ifndef conv32le +# define conv32le(num) byteswap32(num) +# endif +# ifndef conv64le +# define conv64le(num) byteswap64(num) +# endif +#else +# ifndef conv16be +# define conv16be(num) byteswap16(num) +# endif +# ifndef conv32be +# define conv32be(num) byteswap32(num) +# endif +# ifndef conv64be +# define conv64be(num) byteswap64(num) +# endif +# ifndef conv16le +# define conv16le(num) ((uint16_t)(num)) +# endif +# ifndef conv32le +# define conv32le(num) ((uint32_t)(num)) +# endif +# ifndef conv64le +# define conv64le(num) ((uint64_t)(num)) +# endif +#endif + + +//////////////////////////////// +// Unaligned reads and writes // +//////////////////////////////// + +// No-strict-align archs like x86-64 +// --------------------------------- +// +// The traditional way of casting e.g. *(const uint16_t *)uint8_pointer +// is bad even if the uint8_pointer is properly aligned because this kind +// of casts break strict aliasing rules and result in undefined behavior. +// With unaligned pointers it's even worse: compilers may emit vector +// instructions that require aligned pointers even if non-vector +// instructions work with unaligned pointers. +// +// Using memcpy() is the standard compliant way to do unaligned access. +// Many modern compilers inline it so there is no function call overhead. +// For those compilers that don't handle the memcpy() method well, the +// old casting method (that violates strict aliasing) can be requested at +// build time. A third method, casting to a packed struct, would also be +// an option but isn't provided to keep things simpler (it's already a mess). +// Hopefully this is flexible enough in practice. +// +// Some compilers on x86-64 like Clang >= 10 and GCC >= 5.1 detect that +// +// buf[0] | (buf[1] << 8) +// +// reads a 16-bit value and can emit a single 16-bit load and produce +// identical code than with the memcpy() method. In other cases Clang and GCC +// produce either the same or better code with memcpy(). For example, Clang 9 +// on x86-64 can detect 32-bit load but not 16-bit load. +// +// MSVC uses unaligned access with the memcpy() method but emits byte-by-byte +// code for "buf[0] | (buf[1] << 8)". +// +// Conclusion: The memcpy() method is the best choice when unaligned access +// is supported. +// +// Strict-align archs like SPARC +// ----------------------------- +// +// GCC versions from around 4.x to to at least 13.2.0 produce worse code +// from the memcpy() method than from simple byte-by-byte shift-or code +// when reading a 32-bit integer: +// +// (1) It may be constructed on stack using four 8-bit loads, +// four 8-bit stores to stack, and finally one 32-bit load from stack. +// +// (2) Especially with -Os, an actual memcpy() call may be emitted. +// +// This is true on at least on ARM, ARM64, SPARC, SPARC64, MIPS64EL, and +// RISC-V. Of these, ARM, ARM64, and RISC-V support unaligned access in +// some processors but not all so this is relevant only in the case when +// GCC assumes that unaligned is not supported or -mstrict-align or +// -mno-unaligned-access is used. +// +// For Clang it makes little difference. ARM64 with -O2 -mstrict-align +// was one the very few with a minor difference: the memcpy() version +// was one instruction longer. +// +// Conclusion: At least in case of GCC and Clang, byte-by-byte code is +// the best choice for strict-align archs to do unaligned access. +// +// See also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111502 +// +// Thanks to it was easy to test different compilers. +// The following is for little endian targets: +/* +#include +#include + +uint32_t bytes16(const uint8_t *b) +{ + return (uint32_t)b[0] + | ((uint32_t)b[1] << 8); +} + +uint32_t copy16(const uint8_t *b) +{ + uint16_t v; + memcpy(&v, b, sizeof(v)); + return v; +} + +uint32_t bytes32(const uint8_t *b) +{ + return (uint32_t)b[0] + | ((uint32_t)b[1] << 8) + | ((uint32_t)b[2] << 16) + | ((uint32_t)b[3] << 24); +} + +uint32_t copy32(const uint8_t *b) +{ + uint32_t v; + memcpy(&v, b, sizeof(v)); + return v; +} + +void wbytes16(uint8_t *b, uint16_t v) +{ + b[0] = (uint8_t)v; + b[1] = (uint8_t)(v >> 8); +} + +void wcopy16(uint8_t *b, uint16_t v) +{ + memcpy(b, &v, sizeof(v)); +} + +void wbytes32(uint8_t *b, uint32_t v) +{ + b[0] = (uint8_t)v; + b[1] = (uint8_t)(v >> 8); + b[2] = (uint8_t)(v >> 16); + b[3] = (uint8_t)(v >> 24); +} + +void wcopy32(uint8_t *b, uint32_t v) +{ + memcpy(b, &v, sizeof(v)); +} +*/ + + +#ifdef TUKLIB_FAST_UNALIGNED_ACCESS + +static inline uint16_t +read16ne(const uint8_t *buf) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + return *(const uint16_t *)buf; +#else + uint16_t num; + memcpy(&num, buf, sizeof(num)); + return num; +#endif +} + + +static inline uint32_t +read32ne(const uint8_t *buf) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + return *(const uint32_t *)buf; +#else + uint32_t num; + memcpy(&num, buf, sizeof(num)); + return num; +#endif +} + + +static inline uint64_t +read64ne(const uint8_t *buf) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + return *(const uint64_t *)buf; +#else + uint64_t num; + memcpy(&num, buf, sizeof(num)); + return num; +#endif +} + + +static inline void +write16ne(uint8_t *buf, uint16_t num) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + *(uint16_t *)buf = num; +#else + memcpy(buf, &num, sizeof(num)); +#endif + return; +} + + +static inline void +write32ne(uint8_t *buf, uint32_t num) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + *(uint32_t *)buf = num; +#else + memcpy(buf, &num, sizeof(num)); +#endif + return; +} + + +static inline void +write64ne(uint8_t *buf, uint64_t num) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + *(uint64_t *)buf = num; +#else + memcpy(buf, &num, sizeof(num)); +#endif + return; +} + + +static inline uint16_t +read16be(const uint8_t *buf) +{ + uint16_t num = read16ne(buf); + return conv16be(num); +} + + +static inline uint16_t +read16le(const uint8_t *buf) +{ + uint16_t num = read16ne(buf); + return conv16le(num); +} + + +static inline uint32_t +read32be(const uint8_t *buf) +{ + uint32_t num = read32ne(buf); + return conv32be(num); +} + + +static inline uint32_t +read32le(const uint8_t *buf) +{ + uint32_t num = read32ne(buf); + return conv32le(num); +} + + +static inline uint64_t +read64be(const uint8_t *buf) +{ + uint64_t num = read64ne(buf); + return conv64be(num); +} + + +static inline uint64_t +read64le(const uint8_t *buf) +{ + uint64_t num = read64ne(buf); + return conv64le(num); +} + + +// NOTE: Possible byte swapping must be done in a macro to allow the compiler +// to optimize byte swapping of constants when using glibc's or *BSD's +// byte swapping macros. The actual write is done in an inline function +// to make type checking of the buf pointer possible. +#define write16be(buf, num) write16ne(buf, conv16be(num)) +#define write32be(buf, num) write32ne(buf, conv32be(num)) +#define write64be(buf, num) write64ne(buf, conv64be(num)) +#define write16le(buf, num) write16ne(buf, conv16le(num)) +#define write32le(buf, num) write32ne(buf, conv32le(num)) +#define write64le(buf, num) write64ne(buf, conv64le(num)) + +#else + +#ifdef WORDS_BIGENDIAN +# define read16ne read16be +# define read32ne read32be +# define read64ne read64be +# define write16ne write16be +# define write32ne write32be +# define write64ne write64be +#else +# define read16ne read16le +# define read32ne read32le +# define read64ne read64le +# define write16ne write16le +# define write32ne write32le +# define write64ne write64le +#endif + + +static inline uint16_t +read16be(const uint8_t *buf) +{ + uint16_t num = ((uint16_t)buf[0] << 8) | (uint16_t)buf[1]; + return num; +} + + +static inline uint16_t +read16le(const uint8_t *buf) +{ + uint16_t num = ((uint16_t)buf[0]) | ((uint16_t)buf[1] << 8); + return num; +} + + +static inline uint32_t +read32be(const uint8_t *buf) +{ + uint32_t num = (uint32_t)buf[0] << 24; + num |= (uint32_t)buf[1] << 16; + num |= (uint32_t)buf[2] << 8; + num |= (uint32_t)buf[3]; + return num; +} + + +static inline uint32_t +read32le(const uint8_t *buf) +{ + uint32_t num = (uint32_t)buf[0]; + num |= (uint32_t)buf[1] << 8; + num |= (uint32_t)buf[2] << 16; + num |= (uint32_t)buf[3] << 24; + return num; +} + + +static inline uint64_t +read64be(const uint8_t *buf) +{ + uint64_t num = (uint64_t)buf[0] << 56; + num |= (uint64_t)buf[1] << 48; + num |= (uint64_t)buf[2] << 40; + num |= (uint64_t)buf[3] << 32; + num |= (uint64_t)buf[4] << 24; + num |= (uint64_t)buf[5] << 16; + num |= (uint64_t)buf[6] << 8; + num |= (uint64_t)buf[7]; + return num; +} + + +static inline uint64_t +read64le(const uint8_t *buf) +{ + uint64_t num = (uint64_t)buf[0]; + num |= (uint64_t)buf[1] << 8; + num |= (uint64_t)buf[2] << 16; + num |= (uint64_t)buf[3] << 24; + num |= (uint64_t)buf[4] << 32; + num |= (uint64_t)buf[5] << 40; + num |= (uint64_t)buf[6] << 48; + num |= (uint64_t)buf[7] << 56; + return num; +} + + +static inline void +write16be(uint8_t *buf, uint16_t num) +{ + buf[0] = (uint8_t)(num >> 8); + buf[1] = (uint8_t)num; + return; +} + + +static inline void +write16le(uint8_t *buf, uint16_t num) +{ + buf[0] = (uint8_t)num; + buf[1] = (uint8_t)(num >> 8); + return; +} + + +static inline void +write32be(uint8_t *buf, uint32_t num) +{ + buf[0] = (uint8_t)(num >> 24); + buf[1] = (uint8_t)(num >> 16); + buf[2] = (uint8_t)(num >> 8); + buf[3] = (uint8_t)num; + return; +} + + +static inline void +write32le(uint8_t *buf, uint32_t num) +{ + buf[0] = (uint8_t)num; + buf[1] = (uint8_t)(num >> 8); + buf[2] = (uint8_t)(num >> 16); + buf[3] = (uint8_t)(num >> 24); + return; +} + + +static inline void +write64be(uint8_t *buf, uint64_t num) +{ + buf[0] = (uint8_t)(num >> 56); + buf[1] = (uint8_t)(num >> 48); + buf[2] = (uint8_t)(num >> 40); + buf[3] = (uint8_t)(num >> 32); + buf[4] = (uint8_t)(num >> 24); + buf[5] = (uint8_t)(num >> 16); + buf[6] = (uint8_t)(num >> 8); + buf[7] = (uint8_t)num; + return; +} + + +static inline void +write64le(uint8_t *buf, uint64_t num) +{ + buf[0] = (uint8_t)num; + buf[1] = (uint8_t)(num >> 8); + buf[2] = (uint8_t)(num >> 16); + buf[3] = (uint8_t)(num >> 24); + buf[4] = (uint8_t)(num >> 32); + buf[5] = (uint8_t)(num >> 40); + buf[6] = (uint8_t)(num >> 48); + buf[7] = (uint8_t)(num >> 56); + return; +} + +#endif + + +////////////////////////////// +// Aligned reads and writes // +////////////////////////////// + +// Separate functions for aligned reads and writes are provided since on +// strict-align archs aligned access is much faster than unaligned access. +// +// Just like in the unaligned case, memcpy() is needed to avoid +// strict aliasing violations. However, on archs that don't support +// unaligned access the compiler cannot know that the pointers given +// to memcpy() are aligned which results in slow code. As of C11 there is +// no standard way to tell the compiler that we know that the address is +// aligned but some compilers have language extensions to do that. With +// such language extensions the memcpy() method gives excellent results. +// +// What to do on a strict-align system when no known language extensions +// are available? Falling back to byte-by-byte access would be safe but ruin +// optimizations that have been made specifically with aligned access in mind. +// As a compromise, aligned reads will fall back to non-compliant type punning +// but aligned writes will be byte-by-byte, that is, fast reads are preferred +// over fast writes. This obviously isn't great but hopefully it's a working +// compromise for now. +// +// __builtin_assume_aligned is support by GCC >= 4.7 and clang >= 3.6. +#ifdef HAVE___BUILTIN_ASSUME_ALIGNED +# define tuklib_memcpy_aligned(dest, src, size) \ + memcpy(dest, __builtin_assume_aligned(src, size), size) +#else +# define tuklib_memcpy_aligned(dest, src, size) \ + memcpy(dest, src, size) +# ifndef TUKLIB_FAST_UNALIGNED_ACCESS +# define TUKLIB_USE_UNSAFE_ALIGNED_READS 1 +# endif +#endif + + +static inline uint16_t +aligned_read16ne(const uint8_t *buf) +{ +#if defined(TUKLIB_USE_UNSAFE_TYPE_PUNNING) \ + || defined(TUKLIB_USE_UNSAFE_ALIGNED_READS) + return *(const uint16_t *)buf; +#else + uint16_t num; + tuklib_memcpy_aligned(&num, buf, sizeof(num)); + return num; +#endif +} + + +static inline uint32_t +aligned_read32ne(const uint8_t *buf) +{ +#if defined(TUKLIB_USE_UNSAFE_TYPE_PUNNING) \ + || defined(TUKLIB_USE_UNSAFE_ALIGNED_READS) + return *(const uint32_t *)buf; +#else + uint32_t num; + tuklib_memcpy_aligned(&num, buf, sizeof(num)); + return num; +#endif +} + + +static inline uint64_t +aligned_read64ne(const uint8_t *buf) +{ +#if defined(TUKLIB_USE_UNSAFE_TYPE_PUNNING) \ + || defined(TUKLIB_USE_UNSAFE_ALIGNED_READS) + return *(const uint64_t *)buf; +#else + uint64_t num; + tuklib_memcpy_aligned(&num, buf, sizeof(num)); + return num; +#endif +} + + +static inline void +aligned_write16ne(uint8_t *buf, uint16_t num) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + *(uint16_t *)buf = num; +#else + tuklib_memcpy_aligned(buf, &num, sizeof(num)); +#endif + return; +} + + +static inline void +aligned_write32ne(uint8_t *buf, uint32_t num) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + *(uint32_t *)buf = num; +#else + tuklib_memcpy_aligned(buf, &num, sizeof(num)); +#endif + return; +} + + +static inline void +aligned_write64ne(uint8_t *buf, uint64_t num) +{ +#ifdef TUKLIB_USE_UNSAFE_TYPE_PUNNING + *(uint64_t *)buf = num; +#else + tuklib_memcpy_aligned(buf, &num, sizeof(num)); +#endif + return; +} + + +static inline uint16_t +aligned_read16be(const uint8_t *buf) +{ + uint16_t num = aligned_read16ne(buf); + return conv16be(num); +} + + +static inline uint16_t +aligned_read16le(const uint8_t *buf) +{ + uint16_t num = aligned_read16ne(buf); + return conv16le(num); +} + + +static inline uint32_t +aligned_read32be(const uint8_t *buf) +{ + uint32_t num = aligned_read32ne(buf); + return conv32be(num); +} + + +static inline uint32_t +aligned_read32le(const uint8_t *buf) +{ + uint32_t num = aligned_read32ne(buf); + return conv32le(num); +} + + +static inline uint64_t +aligned_read64be(const uint8_t *buf) +{ + uint64_t num = aligned_read64ne(buf); + return conv64be(num); +} + + +static inline uint64_t +aligned_read64le(const uint8_t *buf) +{ + uint64_t num = aligned_read64ne(buf); + return conv64le(num); +} + + +// These need to be macros like in the unaligned case. +#define aligned_write16be(buf, num) aligned_write16ne((buf), conv16be(num)) +#define aligned_write16le(buf, num) aligned_write16ne((buf), conv16le(num)) +#define aligned_write32be(buf, num) aligned_write32ne((buf), conv32be(num)) +#define aligned_write32le(buf, num) aligned_write32ne((buf), conv32le(num)) +#define aligned_write64be(buf, num) aligned_write64ne((buf), conv64be(num)) +#define aligned_write64le(buf, num) aligned_write64ne((buf), conv64le(num)) + + +//////////////////// +// Bit operations // +//////////////////// + +static inline uint32_t +bsr32(uint32_t n) +{ + // Check for ICC first, since it tends to define __GNUC__ too. +#if defined(__INTEL_COMPILER) + return _bit_scan_reverse(n); + +#elif (TUKLIB_GNUC_REQ(3, 4) || defined(__clang__)) && UINT_MAX == UINT32_MAX + // GCC >= 3.4 has __builtin_clz(), which gives good results on + // multiple architectures. On x86, __builtin_clz() ^ 31U becomes + // either plain BSR (so the XOR gets optimized away) or LZCNT and + // XOR (if -march indicates that SSE4a instructions are supported). + return (uint32_t)__builtin_clz(n) ^ 31U; + +#elif defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__)) + uint32_t i; + __asm__("bsrl %1, %0" : "=r" (i) : "rm" (n)); + return i; + +#elif defined(_MSC_VER) + unsigned long i; + _BitScanReverse(&i, n); + return i; + +#else + uint32_t i = 31; + + if ((n & 0xFFFF0000) == 0) { + n <<= 16; + i = 15; + } + + if ((n & 0xFF000000) == 0) { + n <<= 8; + i -= 8; + } + + if ((n & 0xF0000000) == 0) { + n <<= 4; + i -= 4; + } + + if ((n & 0xC0000000) == 0) { + n <<= 2; + i -= 2; + } + + if ((n & 0x80000000) == 0) + --i; + + return i; +#endif +} + + +static inline uint32_t +clz32(uint32_t n) +{ +#if defined(__INTEL_COMPILER) + return _bit_scan_reverse(n) ^ 31U; + +#elif (TUKLIB_GNUC_REQ(3, 4) || defined(__clang__)) && UINT_MAX == UINT32_MAX + return (uint32_t)__builtin_clz(n); + +#elif defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__)) + uint32_t i; + __asm__("bsrl %1, %0\n\t" + "xorl $31, %0" + : "=r" (i) : "rm" (n)); + return i; + +#elif defined(_MSC_VER) + unsigned long i; + _BitScanReverse(&i, n); + return i ^ 31U; + +#else + uint32_t i = 0; + + if ((n & 0xFFFF0000) == 0) { + n <<= 16; + i = 16; + } + + if ((n & 0xFF000000) == 0) { + n <<= 8; + i += 8; + } + + if ((n & 0xF0000000) == 0) { + n <<= 4; + i += 4; + } + + if ((n & 0xC0000000) == 0) { + n <<= 2; + i += 2; + } + + if ((n & 0x80000000) == 0) + ++i; + + return i; +#endif +} + + +static inline uint32_t +ctz32(uint32_t n) +{ +#if defined(__INTEL_COMPILER) + return _bit_scan_forward(n); + +#elif (TUKLIB_GNUC_REQ(3, 4) || defined(__clang__)) && UINT_MAX >= UINT32_MAX + return (uint32_t)__builtin_ctz(n); + +#elif defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__)) + uint32_t i; + __asm__("bsfl %1, %0" : "=r" (i) : "rm" (n)); + return i; + +#elif defined(_MSC_VER) + unsigned long i; + _BitScanForward(&i, n); + return i; + +#else + uint32_t i = 0; + + if ((n & 0x0000FFFF) == 0) { + n >>= 16; + i = 16; + } + + if ((n & 0x000000FF) == 0) { + n >>= 8; + i += 8; + } + + if ((n & 0x0000000F) == 0) { + n >>= 4; + i += 4; + } + + if ((n & 0x00000003) == 0) { + n >>= 2; + i += 2; + } + + if ((n & 0x00000001) == 0) + ++i; + + return i; +#endif +} + +#define bsf32 ctz32 + +#endif diff --git a/src/common/tuklib_mbstr.h b/src/common/tuklib_mbstr.h new file mode 100644 index 0000000000..5ac06eb35e --- /dev/null +++ b/src/common/tuklib_mbstr.h @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr.h +/// \brief Utility functions for handling multibyte strings +/// +/// If not enough multibyte string support is available in the C library, +/// these functions keep working with the assumption that all strings +/// are in a single-byte character set without combining characters, e.g. +/// US-ASCII or ISO-8859-*. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_MBSTR_H +#define TUKLIB_MBSTR_H + +#include "tuklib_common.h" +TUKLIB_DECLS_BEGIN + +#define tuklib_mbstr_width TUKLIB_SYMBOL(tuklib_mbstr_width) +extern size_t tuklib_mbstr_width(const char *str, size_t *bytes); +///< +/// \brief Get the number of columns needed for the multibyte string +/// +/// This is somewhat similar to wcswidth() but works on multibyte strings. +/// +/// \param str String whose width is to be calculated. +/// \param bytes If this is not NULL, *bytes is set to the +/// value returned by strlen(str) (even if an +/// error occurs when calculating the width). +/// +/// \return On success, the number of columns needed to display the +/// string e.g. in a terminal emulator is returned. On error, +/// (size_t)-1 is returned. Possible errors include invalid, +/// partial, or non-printable multibyte character in str. + +#define tuklib_mbstr_width_mem TUKLIB_SYMBOL(tuklib_mbstr_width_mem) +extern size_t tuklib_mbstr_width_mem(const char *str, size_t len); +///< +/// \brief Get the number of columns needed for the multibyte buffer +/// +/// This is like tuklib_mbstr_width() except that this takes the buffer +/// length in bytes as the second argument. This allows using the function +/// for buffers that aren't terminated with '\0'. +/// +/// \param str String whose width is to be calculated. +/// \param len Number of bytes to read from str. +/// +/// \return On success, the number of columns needed to display the +/// string e.g. in a terminal emulator is returned. On error, +/// (size_t)-1 is returned. Possible errors include invalid, +/// partial, or non-printable multibyte character in str. + +#define tuklib_mbstr_fw TUKLIB_SYMBOL(tuklib_mbstr_fw) +extern int tuklib_mbstr_fw(const char *str, int columns_min); +///< +/// \brief Get the field width for printf() e.g. to align table columns +/// +/// Printing simple tables to a terminal can be done using the field field +/// feature in the printf() format string, but it works only with single-byte +/// character sets. To do the same with multibyte strings, tuklib_mbstr_fw() +/// can be used to calculate appropriate field width. +/// +/// The behavior of this function is undefined, if +/// - str is NULL or not terminated with '\0'; +/// - columns_min <= 0; or +/// - the calculated field width exceeds INT_MAX. +/// +/// \return If tuklib_mbstr_width(str, NULL) fails, -1 is returned. +/// If str needs more columns than columns_min, zero is returned. +/// Otherwise a positive integer is returned, which can be +/// used as the field width, e.g. printf("%*s", fw, str). + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_mbstr_fw.c b/src/common/tuklib_mbstr_fw.c new file mode 100644 index 0000000000..22d883b569 --- /dev/null +++ b/src/common/tuklib_mbstr_fw.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr_fw.c +/// \brief Get the field width for printf() e.g. to align table columns +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_mbstr.h" + + +extern int +tuklib_mbstr_fw(const char *str, int columns_min) +{ + size_t len; + const size_t width = tuklib_mbstr_width(str, &len); + if (width == (size_t)-1) + return -1; + + if (width > (size_t)columns_min) + return 0; + + if (width < (size_t)columns_min) + len += (size_t)columns_min - width; + + return (int)len; +} diff --git a/src/common/tuklib_mbstr_nonprint.c b/src/common/tuklib_mbstr_nonprint.c new file mode 100644 index 0000000000..dc778757b1 --- /dev/null +++ b/src/common/tuklib_mbstr_nonprint.c @@ -0,0 +1,162 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr_nonprint.c +/// \brief Find and replace non-printable characters with question marks +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_mbstr_nonprint.h" +#include +#include +#include + +#ifdef HAVE_MBRTOWC +# include +# include +#else +# include +#endif + + +static bool +is_next_printable(const char *str, size_t len, size_t *next_len) +{ +#ifdef HAVE_MBRTOWC + // This assumes that character sets with locking shift states aren't + // used, and thus mbsinit() is never needed. + mbstate_t ps; + memset(&ps, 0, sizeof(ps)); + + wchar_t wc; + *next_len = mbrtowc(&wc, str, len, &ps); + + if (*next_len == (size_t)-2) { + // Incomplete multibyte sequence: Treat the whole sequence + // as a single non-printable multibyte character that ends + // the string. + *next_len = len; + return false; + } + + // Check more broadly than just ret == (size_t)-1 to be safe + // in case mbrtowc() returns something weird. This check + // covers (size_t)-1 (that is, SIZE_MAX) too because len is from + // strlen() and the terminating '\0' isn't part of the length. + if (*next_len < 1 || *next_len > len) { + // Invalid multibyte sequence: Treat the first byte as + // a non-printable single-byte character. Decoding will + // be restarted from the next byte on the next call to + // this function. + *next_len = 1; + return false; + } + +# if defined(_WIN32) && !defined(__CYGWIN__) + // On Windows, wchar_t stores UTF-16 code units, thus characters + // outside the Basic Multilingual Plane (BMP) don't fit into + // a single wchar_t. In an UTF-8 locale, UCRT's mbrtowc() returns + // successfully when the input is a non-BMP character but the + // output is the replacement character U+FFFD. + // + // iswprint() returns 0 for U+FFFD on Windows for some reason. Treat + // U+FFFD as printable and thus also all non-BMP chars as printable. + if (wc == 0xFFFD) + return true; +# endif + + return iswprint((wint_t)wc) != 0; +#else + (void)len; + *next_len = 1; + return isprint((unsigned char)str[0]) != 0; +#endif +} + + +static bool +has_nonprint(const char *str, size_t len) +{ + for (size_t i = 0; i < len; ) { + size_t next_len; + if (!is_next_printable(str + i, len - i, &next_len)) + return true; + + i += next_len; + } + + return false; +} + + +extern bool +tuklib_has_nonprint(const char *str) +{ + const int saved_errno = errno; + const bool ret = has_nonprint(str, strlen(str)); + errno = saved_errno; + return ret; +} + + +extern const char * +tuklib_mask_nonprint_r(const char *str, char **mem) +{ + const int saved_errno = errno; + + // Free the old string, if any. + free(*mem); + *mem = NULL; + + // If the whole input string contains only printable characters, + // return the input string. + const size_t len = strlen(str); + if (!has_nonprint(str, len)) { + errno = saved_errno; + return str; + } + + // Allocate memory for the masked string. Since we use the single-byte + // character '?' to mask non-printable characters, it's possible that + // a few bytes less memory would be needed in reality if multibyte + // characters are masked. + // + // If allocation fails, return "???" because it should be safer than + // returning the unmasked string. + *mem = malloc(len + 1); + if (*mem == NULL) { + errno = saved_errno; + return "???"; + } + + // Replace all non-printable characters with '?'. + char *dest = *mem; + + for (size_t i = 0; i < len; ) { + size_t next_len; + if (is_next_printable(str + i, len - i, &next_len)) { + memcpy(dest, str + i, next_len); + dest += next_len; + } else { + *dest++ = '?'; + } + + i += next_len; + } + + *dest = '\0'; + + errno = saved_errno; + return *mem; +} + + +extern const char * +tuklib_mask_nonprint(const char *str) +{ + static char *mem = NULL; + return tuklib_mask_nonprint_r(str, &mem); +} diff --git a/src/common/tuklib_mbstr_nonprint.h b/src/common/tuklib_mbstr_nonprint.h new file mode 100644 index 0000000000..6fc969109f --- /dev/null +++ b/src/common/tuklib_mbstr_nonprint.h @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr_nonprint.h +/// \brief Find and replace non-printable characters with question marks +/// +/// If mbrtowc(3) is available, it and iswprint(3) is used to check if all +/// characters are printable. Otherwise single-byte character set is assumed +/// and isprint(3) is used. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_MBSTR_NONPRINT_H +#define TUKLIB_MBSTR_NONPRINT_H + +#include "tuklib_common.h" +TUKLIB_DECLS_BEGIN + +#define tuklib_has_nonprint TUKLIB_SYMBOL(tuklib_has_nonprint) +extern bool tuklib_has_nonprint(const char *str); +///< +/// \brief Check if a string contains any non-printable characters +/// +/// \return false if str contains only valid multibyte characters and +/// iswprint(3) returns non-zero for all of them; true otherwise. +/// The value of errno is preserved. +/// +/// \note In case mbrtowc(3) isn't available, single-byte character set +/// is assumed and isprint(3) is used instead of iswprint(3). + +#define tuklib_mask_nonprint_r TUKLIB_SYMBOL(tuklib_mask_nonprint_r) +extern const char *tuklib_mask_nonprint_r(const char *str, char **mem); +///< +/// \brief Replace non-printable characters with question marks +/// +/// \param str Untrusted string, for example, a filename +/// \param mem This function always calls free(*mem) to free the old +/// allocation and then sets *mem = NULL. Before the first +/// call, *mem should be initialized to NULL. If this +/// function needs to allocate memory for a modified +/// string, a pointer to the allocated memory will be +/// stored to *mem. Otherwise *mem will remain NULL. +/// +/// \return If tuklib_has_nonprint(str) returns false, this function +/// returns str. Otherwise memory is allocated to hold a modified +/// string and a pointer to that is returned. The pointer to the +/// allocated memory is also stored to *mem. A modified string +/// has the problematic characters replaced by '?'. If memory +/// allocation fails, "???" is returned and *mem is NULL. +/// The value of errno is preserved. + +#define tuklib_mask_nonprint TUKLIB_SYMBOL(tuklib_mask_nonprint) +extern const char *tuklib_mask_nonprint(const char *str); +///< +/// \brief Replace non-printable characters with question marks +/// +/// This is a convenience function for single-threaded use. This calls +/// tuklib_mask_nonprint_r() using an internal static variable to hold +/// the possible allocation. +/// +/// \param str Untrusted string, for example, a filename +/// +/// \return See tuklib_mask_nonprint_r(). +/// +/// \note This function is not thread safe! + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_mbstr_width.c b/src/common/tuklib_mbstr_width.c new file mode 100644 index 0000000000..98c611d8f3 --- /dev/null +++ b/src/common/tuklib_mbstr_width.c @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr_width.c +/// \brief Calculate width of a multibyte string +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_mbstr.h" +#include + +#ifdef HAVE_MBRTOWC +# include +#endif + + +extern size_t +tuklib_mbstr_width(const char *str, size_t *bytes) +{ + const size_t len = strlen(str); + if (bytes != NULL) + *bytes = len; + + return tuklib_mbstr_width_mem(str, len); +} + + +extern size_t +tuklib_mbstr_width_mem(const char *str, size_t len) +{ +#ifndef HAVE_MBRTOWC + // In single-byte mode, the width of the string is the same + // as its length. + (void)str; + return len; + +#else + mbstate_t state; + memset(&state, 0, sizeof(state)); + + size_t width = 0; + size_t i = 0; + + // Convert one multibyte character at a time to wchar_t + // and get its width using wcwidth(). + while (i < len) { + wchar_t wc; + const size_t ret = mbrtowc(&wc, str + i, len - i, &state); + if (ret < 1 || ret > len - i) + return (size_t)-1; + + i += ret; + +#ifdef HAVE_WCWIDTH + const int wc_width = wcwidth(wc); + if (wc_width < 0) + return (size_t)-1; + + width += (size_t)wc_width; +#else + // Without wcwidth() (like in a native Windows build), + // assume that one multibyte char == one column. With + // UTF-8, this is less bad than one byte == one column. + // This way quite a few languages will be handled correctly + // in practice; CJK chars will be very wrong though. + ++width; +#endif + } + + // It's good to check that the string ended in the initial state. + // However, in practice this is redundant: + // + // - No one will use this code with character sets that have + // locking shift states. + // + // - We already checked that mbrtowc() didn't return (size_t)-2 + // which would indicate a partial multibyte character. + if (!mbsinit(&state)) + return (size_t)-1; + + return width; +#endif +} diff --git a/src/common/tuklib_mbstr_wrap.c b/src/common/tuklib_mbstr_wrap.c new file mode 100644 index 0000000000..8d906e004d --- /dev/null +++ b/src/common/tuklib_mbstr_wrap.c @@ -0,0 +1,294 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr_wrap.c +/// \brief Word wraps a string and prints it to a FILE stream +/// +/// This depends on tuklib_mbstr_width.c. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_mbstr.h" +#include "tuklib_mbstr_wrap.h" +#include +#include +#include +#include + + +extern int +tuklib_wraps(FILE *outfile, const struct tuklib_wrap_opt *opt, const char *str) +{ + // left_cont may be less than left_margin. In that case, if the first + // word is extremely long, it will stay on the first line even if + // the line then gets overlong. + // + // On the other hand, left2_cont < left2_margin isn't allowed because + // it could result in inconsistent behavior when a very long word + // comes right after a \v. + // + // It is fine to have left2_margin < left_margin although it would be + // an odd use case. + if (!(opt->left_margin < opt->right_margin + && opt->left_cont < opt->right_margin + && opt->left2_margin <= opt->left2_cont + && opt->left2_cont < opt->right_margin)) + return TUKLIB_WRAP_ERR_OPT; + + // This is set to TUKLIB_WRAP_WARN_OVERLONG if one or more + // output lines extend past opt->right_margin columns. + int warn_overlong = 0; + + // Indentation of the first output line after \n or \r. + // \v sets this to opt->left2_margin. + // \r resets this back to the original value. + size_t first_indent = opt->left_margin; + + // Indentation of the output lines that occur due to word wrapping. + // \v sets this to opt->left2_cont and \r back to the original value. + size_t cont_indent = opt->left_cont; + + // If word wrapping occurs, the newline isn't printed unless more + // text would be put on the continuation line. This is also used + // when \v needs to start on a new line. + bool pending_newline = false; + + // Spaces are printed only when there is something else to put + // after the spaces on the line. This avoids unwanted empty lines + // in the output and makes it possible to ignore possible spaces + // before a \v character. + size_t pending_spaces = first_indent; + + // Current output column. When cur_col == pending_spaces, nothing + // has been actually printed to the current output line. + size_t cur_col = pending_spaces; + + while (true) { + // Number of bytes until the *next* line-break opportunity. + size_t len = 0; + + // Number of columns until the *next* line-break opportunity. + size_t width = 0; + + // Text between a pair of \b characters is treated as + // an unbreakable block even if it contains spaces. + // It must not contain any control characters before + // the closing \b. + bool unbreakable = false; + + while (true) { + // Find the next character that we handle specially. + // In an unbreakable block, search only for the + // closing \b; if missing, the unbreakable block + // extends to the end of the string. + const size_t n = strcspn(str + len, + unbreakable ? "\b" : " \t\n\r\v\b"); + + // Calculate how many columns the characters need. + const size_t w = tuklib_mbstr_width_mem(str + len, n); + if (w == (size_t)-1) + return TUKLIB_WRAP_ERR_STR; + + width += w; + len += n; + + // \b isn't a line-break opportunity so it has to + // be handled here. For simplicity, empty blocks + // are treated as zero-width characters. + if (str[len] == '\b') { + ++len; + unbreakable = !unbreakable; + continue; + } + + break; + } + + // Determine if adding this chunk of text would make the + // current output line exceed opt->right_margin columns. + const bool too_long = cur_col + width > opt->right_margin; + + // Wrap the line if needed. However: + // + // - Don't wrap if the current column is less than where + // the continuation line would begin. In that case + // the chunk wouldn't fit on the next line either so + // we just have to produce an overlong line. + // + // - Don't wrap if so far the line only contains spaces. + // Wrapping in that case would leave a weird empty line. + // NOTE: This "only contains spaces" condition is the + // reason why left2_margin > left2_cont isn't allowed. + if (too_long && cur_col > cont_indent + && cur_col > pending_spaces) { + // There might be trailing spaces or zero-width spaces + // which need to be ignored to keep the output pretty. + // + // Spaces need to be ignored because in some + // writing styles there are two spaces after + // a full stop. Example string: + // + // "Foo bar. Abc def." + // ^ + // If the first space after the first full stop + // triggers word wrapping, both spaces must be + // ignored. Otherwise the next line would be + // indented too much. + // + // Zero-width spaces are ignored the same way + // because they are meaningless if an adjacent + // character is a space. + while (*str == ' ' || *str == '\t') + ++str; + + // Don't print the newline here; only mark it as + // pending. This avoids an unwanted empty line if + // there is a \n or \r or \0 after the spaces have + // been ignored. + pending_newline = true; + pending_spaces = cont_indent; + cur_col = pending_spaces; + + // Since str may have been incremented due to the + // ignored spaces, the loop needs to be restarted. + continue; + } + + // Print the current chunk of text before the next + // line-break opportunity. If the chunk was empty, + // don't print anything so that the pending newline + // and pending spaces aren't printed on their own. + if (len > 0) { + if (pending_newline) { + pending_newline = false; + if (putc('\n', outfile) == EOF) + return TUKLIB_WRAP_ERR_IO; + } + + while (pending_spaces > 0) { + if (putc(' ', outfile) == EOF) + return TUKLIB_WRAP_ERR_IO; + + --pending_spaces; + } + + for (size_t i = 0; i < len; ++i) { + // Ignore unbreakable block characters (\b). + const int c = (unsigned char)str[i]; + if (c != '\b' && putc(c, outfile) == EOF) + return TUKLIB_WRAP_ERR_IO; + } + + str += len; + cur_col += width; + + // Remember if the line got overlong. If no other + // errors occur, we return warn_overlong. It might + // help in catching problematic strings. + if (too_long) + warn_overlong = TUKLIB_WRAP_WARN_OVERLONG; + } + + // Handle the special character after the chunk of text. + switch (*str) { + case ' ': + // Regular space. + ++cur_col; + ++pending_spaces; + break; + + case '\v': + // Set the alternative indentation settings. + first_indent = opt->left2_margin; + cont_indent = opt->left2_cont; + + if (first_indent > cur_col) { + // Add one or more spaces to reach + // the column specified in first_indent. + pending_spaces += first_indent - cur_col; + } else { + // There is no room to add even one space + // before reaching the column first_indent. + pending_newline = true; + pending_spaces = first_indent; + } + + cur_col = first_indent; + break; + + case '\0': // Implicit newline at the end of the string. + case '\r': // Newline that also resets the effect of \v. + case '\n': // Newline without resetting the indentation mode. + if (putc('\n', outfile) == EOF) + return TUKLIB_WRAP_ERR_IO; + + if (*str == '\0') + return warn_overlong; + + if (*str == '\r') { + first_indent = opt->left_margin; + cont_indent = opt->left_cont; + } + + pending_newline = false; + pending_spaces = first_indent; + cur_col = first_indent; + break; + } + + // Skip the specially-handled character. + ++str; + } +} + + +extern int +tuklib_wrapf(FILE *stream, const struct tuklib_wrap_opt *opt, + const char *fmt, ...) +{ + va_list ap; + char *buf; + +#ifdef HAVE_VASPRINTF + va_start(ap, fmt); + +#ifdef __clang__ +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wformat-nonliteral" +#endif + const int n = vasprintf(&buf, fmt, ap); +#ifdef __clang__ +# pragma GCC diagnostic pop +#endif + + va_end(ap); + if (n == -1) + return TUKLIB_WRAP_ERR_FORMAT; +#else + // Fixed buffer size is dumb but in practice one shouldn't need + // huge strings for *formatted* output. This simple method is safe + // with pre-C99 vsnprintf() implementations too which don't return + // the required buffer size (they return -1 or buf_size - 1) or + // which might not null-terminate the buffer in case it's too small. + const size_t buf_size = 128 * 1024; + buf = malloc(buf_size); + if (buf == NULL) + return TUKLIB_WRAP_ERR_FORMAT; + + va_start(ap, fmt); + const int n = vsnprintf(buf, buf_size, fmt, ap); + va_end(ap); + + if (n <= 0 || n >= (int)(buf_size - 1)) { + free(buf); + return TUKLIB_WRAP_ERR_FORMAT; + } +#endif + + const int ret = tuklib_wraps(stream, opt, buf); + free(buf); + return ret; +} diff --git a/src/common/tuklib_mbstr_wrap.h b/src/common/tuklib_mbstr_wrap.h new file mode 100644 index 0000000000..4e2f297dab --- /dev/null +++ b/src/common/tuklib_mbstr_wrap.h @@ -0,0 +1,204 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_mbstr_wrap.h +/// \brief Word wrapping for multibyte strings +/// +/// The word wrapping functions are intended to be usable, for example, +/// for printing --help text in command line tools. While manually-wrapped +/// --help text allows precise formatting, such freedom requires translators +/// to count spaces and determine where line breaks should occur. It's +/// tedious and error prone, and experience has shown that only some +/// translators do it well. Automatic word wrapping is less flexible but +/// results in polished-enough look with less effort from everyone. +/// Right-to-left languages and languages that don't use spaces between +/// words will still need extra effort though. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_MBSTR_WRAP_H +#define TUKLIB_MBSTR_WRAP_H + +#include "tuklib_common.h" +#include + +TUKLIB_DECLS_BEGIN + +/// One or more output lines exceeded right_margin. +/// This only a warning; everything was still printed successfully. +#define TUKLIB_WRAP_WARN_OVERLONG 0x01 + +/// Error writing to to the output FILE. The error flag in the FILE +/// should have been set as well. +#define TUKLIB_WRAP_ERR_IO 0x02 + +/// Invalid options in struct tuklib_wrap_opt. +/// Nothing was printed. +#define TUKLIB_WRAP_ERR_OPT 0x04 + +/// Invalid or unsupported multibyte character in the input string: +/// either mbrtowc() failed or wcwidth() returned a negative value. +#define TUKLIB_WRAP_ERR_STR 0x08 + +/// Only tuklib_wrapf(): Error in converting the format string. +/// It's either a memory allocation failure or something bad with the +/// format string or arguments. +#define TUKLIB_WRAP_ERR_FORMAT 0x10 + +/// Options for tuklib_wraps() and tuklib_wrapf() +struct tuklib_wrap_opt { + /// Indentation of the first output line after `\n` or `\r`. + /// This can be anything less than right_margin. + unsigned short left_margin; + + /// Column where word-wrapped continuation lines start. + /// This can be anything less than right_margin. + unsigned short left_cont; + + /// Column where the text after `\v` will start, either on the current + /// line (when there is room to add at least one space) or on a new + /// empty line. + unsigned short left2_margin; + + /// Like left_cont but for text after a `\v`. However, this must + /// be greater than or equal to left2_margin in addition to being + /// less than right_margin. + unsigned short left2_cont; + + /// For 80-column terminals, it is recommended to use 79 here for + /// maximum portability. 80 will work most of the time but it will + /// result in unwanted empty lines in the rare case where a terminal + /// moves the cursor to the beginning of the next line immediately + /// when the last column has been used. + unsigned short right_margin; +}; + +#define tuklib_wraps TUKLIB_SYMBOL(tuklib_wraps) +extern int tuklib_wraps(FILE *stream, const struct tuklib_wrap_opt *opt, + const char *str); +///< +/// \brief Word wrap a multibyte string and write it to a FILE +/// +/// Word wrapping is done only at spaces and at the special control characters +/// described below. Multiple consecutive spaces are handled properly: strings +/// that have two (or more) spaces after a full sentence will look good even +/// when the spaces occur at a word wrapping boundary. Trailing spaces are +/// ignored at the end of a line or at the end of a string. +/// +/// The following control characters have been repurposed: +/// +/// - `\t` = Zero-width space allows a line break without producing any +/// output by itself. This can be useful after hard hyphens as +/// hyphens aren't otherwise used for line breaking. This can also +/// be useful in languages that don't use spaces between words. +/// (The Unicode character U+200B isn't supported.) +/// - `\b` = Text between a pair of `\b` characters is treated as an +/// unbreakable block (not wrapped even if there are spaces). +/// For example, a non-breaking space can be done like +/// in `"123\b \bMiB"`. Control characters (like `\n` or `\t`) +/// aren't allowed before the closing `\b`. If closing `\b` is +/// missing, the block extends to the end of the string. Empty +/// blocks are treated as zero-width characters. If line breaks +/// are possible around an empty block (like in `"foo \b\b bar"` +/// or `"foo \b"`), it can result in weird output. +/// - `\v` = Change to alternative indentation (left2_margin). +/// - `\r` = Reset back to the initial indentation and add a newline. +/// The next line will be indented by left_margin. +/// - `\n` = Add a newline without resetting the effect of `\v`. The +/// next line will be indented by left_margin or left2_margin +/// (not left_cont or left2_cont). +/// +/// Only `\n` should appear in translatable strings. `\t` works too but +/// even that might confuse some translators even if there is a TRANSLATORS +/// comment explaining its meaning. +/// +/// To use the other control characters in messages, one should use +/// tuklib_wrapf() with appropriate printf format string to combine +/// translatable strings with non-translatable portions. For example: +/// +/// \code{.c} +/// static const struct tuklib_wrap_opt wrap2 = { 2, 2, 22, 22, 79 }; +/// int e = 0; +/// ... +/// e |= tuklib_wrapf(stdout, &wrap2, +/// "-h, --help\v%s\r" +/// " --version\v%s", +/// W_("display this help and exit"), +/// W_("display version information and exit")); +/// ... +/// if (e != 0) { +/// // Handle warning or error. +/// ... +/// } +/// \endcode +/// +/// Control characters other than `\n` and `\t` are unusable in +/// translatable strings: +/// +/// - Gettext tools show annoying warnings if C escape sequences other +/// than `\n` or `\t` are seen. (Otherwise they still work perfectly +/// fine though.) +/// +/// - While at least Poedit and Lokalize support all escapes, some +/// editors only support `\n` and `\t`. +/// +/// - They could confuse some translators, resulting in broken +/// translations. +/// +/// Using non-control characters would solve some issues but it wouldn't +/// help with the unfortunate real-world issue that some translators would +/// likely have trouble understanding a new syntax. The Gettext manual +/// specifically warns about this, see the subheading "No unusual markup" +/// in `info (gettext)Preparing Strings`. (While using `\t` for zero-width +/// space is such custom markup, most translators will never need it.) +/// +/// Translators can use the Unicode character U+00A0 (or U+202F) if they +/// need a non-breaking space. For example, in French a non-breaking space +/// may be needed before colons and question marks (U+00A0 is common in +/// real-world French PO files). +/// +/// Using a non-ASCII char in a string in the C code (like `"123\u00A0MiB"`) +/// can work if one tells xgettext that input encoding is UTF-8, one +/// ensures that the C compiler uses UTF-8 as the input charset, and one +/// is certain that the program is *always* run under an UTF-8 locale. +/// Unfortunately a portable program cannot make this kind of assumptions, +/// which means that there is no pretty way to have a non-breaking space in +/// a translatable string. +/// +/// Optional: To tell translators which strings are automatically word +/// wrapped, see the macro `W_` in tuklib_gettext.h. +/// +/// \param stream Output FILE stream. For decent performance, it +/// should be in buffered mode because this function +/// writes the output one byte at a time with fputc(). +/// \param opt Word wrapping options. +/// \param str Null-terminated multibyte string that is in +/// the encoding used by the current locale. +/// +/// \return Returns 0 on success. If an error or warning occurs, one of +/// TUKLIB_WRAP_* codes is returned. Those codes are powers +/// of two. When warning/error detection can be delayed, the +/// return values can be accumulated from multiple calls using +/// bitwise-or into a single variable which can be checked after +/// all strings have (hopefully) been printed. + +#define tuklib_wrapf TUKLIB_SYMBOL(tuklib_wrapf) +tuklib_attr_format_printf(3, 4) +extern int tuklib_wrapf(FILE *stream, const struct tuklib_wrap_opt *opt, + const char *fmt, ...); +///< +/// \brief Format and word-wrap a multibyte string and write it to a FILE +/// +/// This is like tuklib_wraps() except that this takes a printf +/// format string. +/// +/// \note On platforms that lack vasprintf(), the intermediate +/// result from vsnprintf() must fit into a 128 KiB buffer. +/// TUKLIB_WRAP_ERR_FORMAT is returned if it doesn't but +/// only on platforms that lack vasprintf(). + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_open_stdxxx.c b/src/common/tuklib_open_stdxxx.c new file mode 100644 index 0000000000..b93e61d3b6 --- /dev/null +++ b/src/common/tuklib_open_stdxxx.c @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_open_stdxxx.c +/// \brief Make sure that file descriptors 0, 1, and 2 are open +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_open_stdxxx.h" + +#ifndef TUKLIB_DOSLIKE +# include +# include +# include +# include +#endif + + +extern void +tuklib_open_stdxxx(int err_status) +{ +#ifdef TUKLIB_DOSLIKE + // Do nothing, just silence warnings. + (void)err_status; + +#else + for (int i = 0; i <= 2; ++i) { + // We use fcntl() to check if the file descriptor is open. + if (fcntl(i, F_GETFD) == -1 && errno == EBADF) { + // With stdin, we could use /dev/full so that + // writing to stdin would fail. However, /dev/full + // is Linux specific, and if the program tries to + // write to stdin, there's already a problem anyway. + const int fd = open("/dev/null", O_NOCTTY + | (i == 0 ? O_WRONLY : O_RDONLY)); + + if (fd != i) { + if (fd != -1) + (void)close(fd); + + // Something went wrong. Exit with the + // exit status we were given. Don't try + // to print an error message, since stderr + // may very well be non-existent. This + // error should be extremely rare. + exit(err_status); + } + } + } +#endif + + return; +} diff --git a/src/common/tuklib_open_stdxxx.h b/src/common/tuklib_open_stdxxx.h new file mode 100644 index 0000000000..3ee3ade355 --- /dev/null +++ b/src/common/tuklib_open_stdxxx.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_open_stdxxx.h +/// \brief Make sure that file descriptors 0, 1, and 2 are open +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_OPEN_STDXXX_H +#define TUKLIB_OPEN_STDXXX_H + +#include "tuklib_common.h" +TUKLIB_DECLS_BEGIN + +#define tuklib_open_stdxx TUKLIB_SYMBOL(tuklib_open_stdxxx) +extern void tuklib_open_stdxxx(int err_status); + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_physmem.c b/src/common/tuklib_physmem.c new file mode 100644 index 0000000000..5988ba77a2 --- /dev/null +++ b/src/common/tuklib_physmem.c @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_physmem.c +/// \brief Get the amount of physical memory +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_physmem.h" + +// We want to use Windows-specific code on Cygwin, which also has memory +// information available via sysconf(), but on Cygwin 1.5 and older it +// gives wrong results (from our point of view). +#if defined(_WIN32) || defined(__CYGWIN__) +# ifndef _WIN32_WINNT +# define _WIN32_WINNT 0x0500 +# endif +# include + +#elif defined(__OS2__) +# define INCL_DOSMISC +# include + +#elif defined(__DJGPP__) +# include + +#elif defined(__VMS) +# include +# include +# include + +#elif defined(AMIGA) || defined(__AROS__) +# define __USE_INLINE__ +# include + +#elif defined(__QNX__) +# include +# include + +#elif defined(TUKLIB_PHYSMEM_AIX) +# include + +#elif defined(TUKLIB_PHYSMEM_SYSCONF) +# include + +#elif defined(TUKLIB_PHYSMEM_SYSCTL) +# ifdef HAVE_SYS_PARAM_H +# include +# endif +# include + +// Tru64 +#elif defined(TUKLIB_PHYSMEM_GETSYSINFO) +# include +# include + +// HP-UX +#elif defined(TUKLIB_PHYSMEM_PSTAT_GETSTATIC) +# include +# include + +// IRIX +#elif defined(TUKLIB_PHYSMEM_GETINVENT_R) +# include + +// This sysinfo() is Linux-specific. +#elif defined(TUKLIB_PHYSMEM_SYSINFO) +# include +#endif + + +extern uint64_t +tuklib_physmem(void) +{ + uint64_t ret = 0; + +#if defined(_WIN32) || defined(__CYGWIN__) + // This requires Windows 2000 or later. + MEMORYSTATUSEX meminfo; + meminfo.dwLength = sizeof(meminfo); + if (GlobalMemoryStatusEx(&meminfo)) + ret = meminfo.ullTotalPhys; + +/* + // Old version that is compatible with even Win95: + if ((GetVersion() & 0xFF) >= 5) { + // Windows 2000 and later have GlobalMemoryStatusEx() which + // supports reporting values greater than 4 GiB. To keep the + // code working also on older Windows versions, use + // GlobalMemoryStatusEx() conditionally. + HMODULE kernel32 = GetModuleHandleA("kernel32.dll"); + if (kernel32 != NULL) { + typedef BOOL (WINAPI *gmse_type)(LPMEMORYSTATUSEX); + gmse_type gmse = (gmse_type)GetProcAddress( + kernel32, "GlobalMemoryStatusEx"); + if (gmse != NULL) { + MEMORYSTATUSEX meminfo; + meminfo.dwLength = sizeof(meminfo); + if (gmse(&meminfo)) + ret = meminfo.ullTotalPhys; + } + } + } + + if (ret == 0) { + // GlobalMemoryStatus() is supported by Windows 95 and later, + // so it is fine to link against it unconditionally. Note that + // GlobalMemoryStatus() has no return value. + MEMORYSTATUS meminfo; + meminfo.dwLength = sizeof(meminfo); + GlobalMemoryStatus(&meminfo); + ret = meminfo.dwTotalPhys; + } +*/ + +#elif defined(__OS2__) + unsigned long mem; + if (DosQuerySysInfo(QSV_TOTPHYSMEM, QSV_TOTPHYSMEM, + &mem, sizeof(mem)) == 0) + ret = mem; + +#elif defined(__DJGPP__) + __dpmi_free_mem_info meminfo; + if (__dpmi_get_free_memory_information(&meminfo) == 0 + && meminfo.total_number_of_physical_pages + != (unsigned long)-1) + ret = (uint64_t)meminfo.total_number_of_physical_pages * 4096; + +#elif defined(__VMS) + int vms_mem; + int val = SYI$_MEMSIZE; + if (LIB$GETSYI(&val, &vms_mem, 0, 0, 0, 0) == SS$_NORMAL) + ret = (uint64_t)vms_mem * 8192; + +#elif defined(AMIGA) || defined(__AROS__) + ret = AvailMem(MEMF_TOTAL); + +#elif defined(__QNX__) + const struct asinfo_entry *entries = SYSPAGE_ENTRY(asinfo); + size_t count = SYSPAGE_ENTRY_SIZE(asinfo) / sizeof(struct asinfo_entry); + const char *strings = SYSPAGE_ENTRY(strings)->data; + + for (size_t i = 0; i < count; ++i) + if (strcmp(strings + entries[i].name, "ram") == 0) + ret += entries[i].end - entries[i].start + 1; + +#elif defined(TUKLIB_PHYSMEM_AIX) + ret = (uint64_t)_system_configuration.physmem; + +#elif defined(TUKLIB_PHYSMEM_SYSCONF) + const long pagesize = sysconf(_SC_PAGESIZE); + const long pages = sysconf(_SC_PHYS_PAGES); + if (pagesize != -1 && pages != -1) + // According to docs, pagesize * pages can overflow. + // Simple case is 32-bit box with 4 GiB or more RAM, + // which may report exactly 4 GiB of RAM, and "long" + // being 32-bit will overflow. Casting to uint64_t + // hopefully avoids overflows in the near future. + ret = (uint64_t)pagesize * (uint64_t)pages; + +#elif defined(TUKLIB_PHYSMEM_SYSCTL) + int name[2] = { + CTL_HW, +#ifdef HW_PHYSMEM64 + HW_PHYSMEM64 +#else + HW_PHYSMEM +#endif + }; + union { + uint32_t u32; + uint64_t u64; + } mem; + size_t mem_ptr_size = sizeof(mem.u64); + if (sysctl(name, 2, &mem.u64, &mem_ptr_size, NULL, 0) != -1) { + // IIRC, 64-bit "return value" is possible on some 64-bit + // BSD systems even with HW_PHYSMEM (instead of HW_PHYSMEM64), + // so support both. + if (mem_ptr_size == sizeof(mem.u64)) + ret = mem.u64; + else if (mem_ptr_size == sizeof(mem.u32)) + ret = mem.u32; + } + +#elif defined(TUKLIB_PHYSMEM_GETSYSINFO) + // Docs are unclear if "start" is needed, but it doesn't hurt + // much to have it. + int memkb; + int start = 0; + if (getsysinfo(GSI_PHYSMEM, (caddr_t)&memkb, sizeof(memkb), &start) + != -1) + ret = (uint64_t)memkb * 1024; + +#elif defined(TUKLIB_PHYSMEM_PSTAT_GETSTATIC) + struct pst_static pst; + if (pstat_getstatic(&pst, sizeof(pst), 1, 0) != -1) + ret = (uint64_t)pst.physical_memory * (uint64_t)pst.page_size; + +#elif defined(TUKLIB_PHYSMEM_GETINVENT_R) + inv_state_t *st = NULL; + if (setinvent_r(&st) != -1) { + inventory_t *i; + while ((i = getinvent_r(st)) != NULL) { + if (i->inv_class == INV_MEMORY + && i->inv_type == INV_MAIN_MB) { + ret = (uint64_t)i->inv_state << 20; + break; + } + } + + endinvent_r(st); + } + +#elif defined(TUKLIB_PHYSMEM_SYSINFO) + struct sysinfo si; + if (sysinfo(&si) == 0) + ret = (uint64_t)si.totalram * si.mem_unit; +#endif + + return ret; +} diff --git a/src/common/tuklib_physmem.h b/src/common/tuklib_physmem.h new file mode 100644 index 0000000000..f35bfbab9c --- /dev/null +++ b/src/common/tuklib_physmem.h @@ -0,0 +1,27 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_physmem.h +/// \brief Get the amount of physical memory +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_PHYSMEM_H +#define TUKLIB_PHYSMEM_H + +#include "tuklib_common.h" +TUKLIB_DECLS_BEGIN + +#define tuklib_physmem TUKLIB_SYMBOL(tuklib_physmem) +extern uint64_t tuklib_physmem(void); +///< +/// \brief Get the amount of physical memory in bytes +/// +/// \return Amount of physical memory in bytes. On error, zero is +/// returned. + +TUKLIB_DECLS_END +#endif diff --git a/src/common/tuklib_progname.c b/src/common/tuklib_progname.c new file mode 100644 index 0000000000..959c1270ce --- /dev/null +++ b/src/common/tuklib_progname.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_progname.c +/// \brief Program name to be displayed in messages +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "tuklib_progname.h" +#include + + +#ifndef HAVE_PROGRAM_INVOCATION_NAME +char *progname = NULL; +#endif + + +extern void +tuklib_progname_init(char **argv) +{ +#ifdef TUKLIB_DOSLIKE + // On these systems, argv[0] always has the full path and .exe + // suffix even if the user just types the plain program name. + // We modify argv[0] to make it nicer to read. + + // Strip the leading path. + char *p = argv[0] + strlen(argv[0]); + while (argv[0] < p && p[-1] != '/' && p[-1] != '\\') + --p; + + argv[0] = p; + + // Strip the .exe suffix. + p = strrchr(p, '.'); + if (p != NULL) + *p = '\0'; + + // Make it lowercase. + for (p = argv[0]; *p != '\0'; ++p) + if (*p >= 'A' && *p <= 'Z') + *p = *p - 'A' + 'a'; +#endif + + progname = argv[0]; + return; +} diff --git a/src/common/tuklib_progname.h b/src/common/tuklib_progname.h new file mode 100644 index 0000000000..a3d90cb1f2 --- /dev/null +++ b/src/common/tuklib_progname.h @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file tuklib_progname.h +/// \brief Program name to be displayed in messages +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef TUKLIB_PROGNAME_H +#define TUKLIB_PROGNAME_H + +#include "tuklib_common.h" +#include + +TUKLIB_DECLS_BEGIN + +#ifdef HAVE_PROGRAM_INVOCATION_NAME +# define progname program_invocation_name +#else +# define progname TUKLIB_SYMBOL(tuklib_progname) + extern char *progname; +#endif + +#define tuklib_progname_init TUKLIB_SYMBOL(tuklib_progname_init) +extern void tuklib_progname_init(char **argv); + +TUKLIB_DECLS_END +#endif diff --git a/src/common/w32_application.manifest b/src/common/w32_application.manifest new file mode 100644 index 0000000000..2f8750879b --- /dev/null +++ b/src/common/w32_application.manifest @@ -0,0 +1,28 @@ + + + + + + + + + + + + + + + + + + + + + + + + true + UTF-8 + + + diff --git a/src/common/w32_application.manifest.comments.txt b/src/common/w32_application.manifest.comments.txt new file mode 100644 index 0000000000..de5c2105ac --- /dev/null +++ b/src/common/w32_application.manifest.comments.txt @@ -0,0 +1,192 @@ + +Windows application manifest for UTF-8 and long paths +===================================================== + +The .manifest file is embedded as is in the executables, thus +the comments are here in a separate file. These comments were +written in context of XZ Utils but might be useful when porting +other command line tools from POSIX environments to Windows. + + NOTE: On Cygwin and MSYS2, command line arguments and file + system access aren't tied to a Windows code page. Cygwin + and MSYS2 include a default application manifest. Replacing + it doesn't seem useful and might even be harmful if Cygwin + and MSYS2 some day change their default manifest. + + +UTF-8 code page +--------------- + +On Windows, command line applications can use main() or wmain(). +With the Windows-specific wmain(), argv contains UTF-16 code units +which is the native encoding on Windows. With main(), argv uses the +system active code page by default. It typically is a legacy code +page like Windows-1252. + + NOTE: On POSIX, argv for main() is constructed by the calling + process. On Windows, argv is constructed by a new process + itself: a program receives the command line as a single string, + and the startup code splits it into individual arguments, + including quote removal and wildcard expansion. Then main() or + wmain() is called. + +This application manifest forces the process code page to UTF-8 +when the application runs on Windows 10 version 1903 or later. +This is useful for programs that use main(): + + * UTF-8 allows such programs to access files whose names contain + characters that don't exist in the current legacy code page. + However, filenames on Windows may contain unpaired surrogates + (invalid UTF-16). Such files cannot be accesses even with the + UTF-8 code page. + + * UTF-8 avoids a security issue in command line argument handling: + If a command line contains Unicode characters (for example, + filenames) that don't exist in the current legacy code page, + the characters are converted to similar-looking characters + with best-fit mapping. Some best-fit mappings result in ASCII + characters that change the meaning of the command line, which + can be exploited with malicious filenames. For example: + + - Double quote (") breaks quoting and makes argument + injection possible. + + - Question mark (?) is a wildcard character which may + expand to one or more filenames. + + - Forward slash (/) makes a directory traversal attack + possible. This character can appear in a dangerous way + even from a wildcard expansion; a look-alike character + doesn't need to be passed directly on the command line. + + UTF-8 avoids best-fit mappings. However, it's still not + perfect. Unpaired surrogates (invalid UTF-16) on the command + line (including those from wildcard expansion) are converted + to the replacement character U+FFFD. Thus, filenames with + different unpaired surrogates appear identical when converted + to the UTF-8 code page and aren't distinguishable from + filenames that contain the actual replacement character U+FFFD. + + FindFirstFileA() and FindFirstFileExA() also suffer from the above + issue where unpaired surrogates become U+FFFD. Another issue is + that filenames may require more bytes in UTF-8 than in a legacy + code page. In UTF-8, a very long filename may exceed MAX_PATH bytes + and thus these APIs cannot list such filenames anymore because + WIN32_FIND_DATAA has a member "CHAR cFileName[MAX_PATH]". + +If different programs use different code pages, compatibility issues +are possible. For example, if one program produces a list of +filenames and another program reads it, both programs should use +the same code page because the code page affects filenames in the +char-based file system APIs. + +If building with a MinGW-w64 toolchain, it is strongly recommended +to use UCRT instead of the old MSVCRT. For example, with the UTF-8 +code page, MSVCRT doesn't convert non-ASCII characters correctly +when writing to console with printf(). With UCRT it works. + + +Long path names +--------------- + +The manifest enables support for path names longer than 260 wide +characters (UTF-16 code units) if the feature has been enabled in +the Windows registry. Omit the longPathAware element from the manifest +if the application isn't compatible with it. For example, some uses +of MAX_PATH might be a sign of incompatibility. + +Note that UTF-8 encoded filenames can exceed MAX_PATH (260) bytes when +the UTF-16 form is still within MAX_PATH wide characters. In this +situation the application doesn't need to be long path aware: functions +like _open() work with UTF-8 names that exceed MAX_PATH bytes if the +wide character form stays within MAX_PATH wide characters. (MAX_PATH +includes the terminating null character.) + +Documentation of the registry setting: +https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later + + +Summary of the manifest contents +-------------------------------- + +See also Microsoft's documentation: +https://learn.microsoft.com/en-us/windows/win32/sbscs/application-manifests + +assemblyIdentity (omitted) + + This is documented as mandatory but not all apps in the real world + have it, and of those that do, not all put an up-to-date version + number there. Things seem to work correctly without + so let's keep this simpler and omit it. + +compatibility + + Declare the application compatible with different Windows versions. + Without this, Windows versions newer than Vista will run the + application using Vista as the Operating System Context. + +trustInfo + + Declare the application as UAC-compliant. This avoids file system + and registry virtualization that Windows otherwise does with 32-bit + executables to make some ancient applications work. UAC-compliancy + also stops Windows from using heuristics based on the filename + (like setup.exe) to guess when elevated privileges might be + needed which would then bring up the UAC prompt. + +longPathAware + + Declare the application as long path aware. This way many file + system operations aren't limited to MAX_PATH (260) wide characters + (including the terminating null character). The feature has to be + enabled in the Windows registry too. + +activeCodePage + + Force the process code page to UTF-8 on Windows 10 version 1903 + and later. For example: + + - main() gets the command line arguments in UTF-8 instead of + in a legacy code page. + + - File system APIs that take char-based strings use UTF-8 + instead of a legacy code page. + + - Text written to the console via stdio.h's stdout or stderr + (like calling printf()) are expected to be in UTF-8. + + +CMake notes +----------- + +As of CMake 3.30, one can add a .manifest file as a source file but +it only works with MSVC; it's ignored with MinGW-w64 toolchains. +Embedding the manifest with a resource file works with all +toolchains. However, then the default manifest needs to be +disabled with MSVC in CMakeLists.txt to avoid duplicate +manifests which would break the build. + +w32_application.manifest.rc: + + #include + CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST "w32_application.manifest" + +Or the same thing without the #include: + + 1 24 "w32_application.manifest" + +CMakeLists.txt: + + if(MSVC) + set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /MANIFEST:NO") + endif() + + add_executable(foo foo.c) + + # WIN32 isn't set on Cygwin or MSYS2, thus if(WIN32) is correct here. + if(WIN32) + target_sources(foo PRIVATE w32_application.manifest.rc) + set_source_files_properties(w32_application.manifest.rc PROPERTIES + OBJECT_DEPENDS w32_application.manifest + ) + endif() diff --git a/src/liblzma/Makefile.am b/src/liblzma/Makefile.am new file mode 100644 index 0000000000..d554702012 --- /dev/null +++ b/src/liblzma/Makefile.am @@ -0,0 +1,126 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +SUBDIRS = api + +EXTRA_DIST = +CLEANFILES = +doc_DATA = + +lib_LTLIBRARIES = liblzma.la +liblzma_la_SOURCES = +liblzma_la_CPPFLAGS = \ + -I$(top_srcdir)/src/liblzma/api \ + -I$(top_srcdir)/src/liblzma/common \ + -I$(top_srcdir)/src/liblzma/check \ + -I$(top_srcdir)/src/liblzma/lz \ + -I$(top_srcdir)/src/liblzma/rangecoder \ + -I$(top_srcdir)/src/liblzma/lzma \ + -I$(top_srcdir)/src/liblzma/delta \ + -I$(top_srcdir)/src/liblzma/simple \ + -I$(top_srcdir)/src/common \ + -DTUKLIB_SYMBOL_PREFIX=lzma_ +liblzma_la_LDFLAGS = -no-undefined -version-info 13:1:8 + +EXTRA_DIST += liblzma_generic.map liblzma_linux.map validate_map.sh +if COND_SYMVERS_GENERIC +liblzma_la_LDFLAGS += \ + -Wl,--version-script=$(top_srcdir)/src/liblzma/liblzma_generic.map +endif +if COND_SYMVERS_LINUX +liblzma_la_LDFLAGS += \ + -Wl,--version-script=$(top_srcdir)/src/liblzma/liblzma_linux.map +endif + +liblzma_la_SOURCES += ../common/tuklib_physmem.c + +if COND_THREADS +liblzma_la_SOURCES += ../common/tuklib_cpucores.c +endif + +include $(srcdir)/common/Makefile.inc +include $(srcdir)/check/Makefile.inc + +if COND_FILTER_LZ +include $(srcdir)/lz/Makefile.inc +endif + +if COND_FILTER_LZMA1 +include $(srcdir)/lzma/Makefile.inc +include $(srcdir)/rangecoder/Makefile.inc +endif + +if COND_FILTER_DELTA +include $(srcdir)/delta/Makefile.inc +endif + +if COND_FILTER_SIMPLE +include $(srcdir)/simple/Makefile.inc +endif + + +## Windows-specific stuff + +# Windows resource compiler support. libtool knows what to do with .rc +# files, but Automake (<= 1.11 at least) doesn't know. +# +# We want the resource file only in shared liblzma. To avoid linking it into +# static liblzma, we overwrite the static object file with an object file +# compiled from empty input. Note that GNU-specific features are OK here, +# because on Windows we are compiled with the GNU toolchain. +# +# The typedef in empty.c will prevent an empty translation unit, which is +# not allowed by the C standard. It results in a warning with +# -Wempty-translation-unit with Clang or -pedantic for GCC. +.rc.lo: + $(LIBTOOL) --mode=compile $(RC) $(DEFS) $(DEFAULT_INCLUDES) \ + $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(RCFLAGS) \ + -i $< -o $@ + echo "typedef void empty;" > empty.c + $(COMPILE) -c empty.c -o $(*D)/$(*F).o + +# Remove ordinals from the generated .def file. People must link by name, +# not by ordinal, because no one is going to track the ordinal numbers. +liblzma.def: liblzma.la liblzma.def.in + sed 's/ \+@ *[0-9]\+//' liblzma.def.in > liblzma.def + +# Creating liblzma.def.in is a side effect of linking the library. +liblzma.def.in: liblzma.la + +if COND_W32 +CLEANFILES += liblzma.def liblzma.def.in empty.c +liblzma_la_SOURCES += liblzma_w32res.rc +liblzma_la_LDFLAGS += -Xlinker --output-def -Xlinker liblzma.def.in + +## liblzma.def.in is created only when building shared liblzma, so don't +## try to create liblzma.def when not building shared liblzma. +if COND_SHARED +doc_DATA += liblzma.def +endif +endif + + +## pkg-config +pkgconfigdir = $(libdir)/pkgconfig +pkgconfig_DATA = liblzma.pc +EXTRA_DIST += liblzma.pc.in + +pc_verbose = $(pc_verbose_@AM_V@) +pc_verbose_ = $(pc_verbose_@AM_DEFAULT_V@) +pc_verbose_0 = @echo " PC " $@; + +liblzma.pc: $(srcdir)/liblzma.pc.in + $(AM_V_at)rm -f $@ + $(pc_verbose)sed \ + -e 's,@prefix[@],$(prefix),g' \ + -e 's,@exec_prefix[@],$(exec_prefix),g' \ + -e 's,@libdir[@],$(libdir),g' \ + -e 's,@includedir[@],$(includedir),g' \ + -e 's,@PACKAGE_URL[@],$(PACKAGE_URL),g' \ + -e 's,@PACKAGE_VERSION[@],$(PACKAGE_VERSION),g' \ + -e 's,@PTHREAD_CFLAGS[@],$(PTHREAD_CFLAGS),g' \ + -e 's,@LIBS[@],$(LIBS),g' \ + < $(srcdir)/liblzma.pc.in > $@ || { rm -f $@; exit 1; } + +clean-local: + rm -f liblzma.pc diff --git a/src/liblzma/Makefile.in b/src/liblzma/Makefile.in new file mode 100644 index 0000000000..edb280f01f --- /dev/null +++ b/src/liblzma/Makefile.in @@ -0,0 +1,2233 @@ +# Makefile.in generated by automake 1.17 from Makefile.am. +# @configure_input@ + +# Copyright (C) 1994-2024 Free Software Foundation, Inc. + +# This Makefile.in is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY, to the extent permitted by law; without +# even the implied warranty of MERCHANTABILITY or FITNESS FOR A +# PARTICULAR PURPOSE. + +@SET_MAKE@ + + +VPATH = @srcdir@ +am__is_gnu_make = { \ + if test -z '$(MAKELEVEL)'; then \ + false; \ + elif test -n '$(MAKE_HOST)'; then \ + true; \ + elif test -n '$(MAKE_VERSION)' && test -n '$(CURDIR)'; then \ + true; \ + else \ + false; \ + fi; \ +} +am__make_running_with_option = \ + case $${target_option-} in \ + ?) ;; \ + *) echo "am__make_running_with_option: internal error: invalid" \ + "target option '$${target_option-}' specified" >&2; \ + exit 1;; \ + esac; \ + has_opt=no; \ + sane_makeflags=$$MAKEFLAGS; \ + if $(am__is_gnu_make); then \ + sane_makeflags=$$MFLAGS; \ + else \ + case $$MAKEFLAGS in \ + *\\[\ \ ]*) \ + bs=\\; \ + sane_makeflags=`printf '%s\n' "$$MAKEFLAGS" \ + | sed "s/$$bs$$bs[$$bs $$bs ]*//g"`;; \ + esac; \ + fi; \ + skip_next=no; \ + strip_trailopt () \ + { \ + flg=`printf '%s\n' "$$flg" | sed "s/$$1.*$$//"`; \ + }; \ + for flg in $$sane_makeflags; do \ + test $$skip_next = yes && { skip_next=no; continue; }; \ + case $$flg in \ + *=*|--*) continue;; \ + -*I) strip_trailopt 'I'; skip_next=yes;; \ + -*I?*) strip_trailopt 'I';; \ + -*O) strip_trailopt 'O'; skip_next=yes;; \ + -*O?*) strip_trailopt 'O';; \ + -*l) strip_trailopt 'l'; skip_next=yes;; \ + -*l?*) strip_trailopt 'l';; \ + -[dEDm]) skip_next=yes;; \ + -[JT]) skip_next=yes;; \ + esac; \ + case $$flg in \ + *$$target_option*) has_opt=yes; break;; \ + esac; \ + done; \ + test $$has_opt = yes +am__make_dryrun = (target_option=n; $(am__make_running_with_option)) +am__make_keepgoing = (target_option=k; $(am__make_running_with_option)) +am__rm_f = rm -f $(am__rm_f_notfound) +am__rm_rf = rm -rf $(am__rm_f_notfound) +pkgdatadir = $(datadir)/@PACKAGE@ +pkgincludedir = $(includedir)/@PACKAGE@ +pkglibdir = $(libdir)/@PACKAGE@ +pkglibexecdir = $(libexecdir)/@PACKAGE@ +am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd +install_sh_DATA = $(install_sh) -c -m 644 +install_sh_PROGRAM = $(install_sh) -c +install_sh_SCRIPT = $(install_sh) -c +INSTALL_HEADER = $(INSTALL_DATA) +transform = $(program_transform_name) +NORMAL_INSTALL = : +PRE_INSTALL = : +POST_INSTALL = : +NORMAL_UNINSTALL = : +PRE_UNINSTALL = : +POST_UNINSTALL = : +build_triplet = @build@ +host_triplet = @host@ +@COND_SYMVERS_GENERIC_TRUE@am__append_1 = \ +@COND_SYMVERS_GENERIC_TRUE@ -Wl,--version-script=$(top_srcdir)/src/liblzma/liblzma_generic.map + +@COND_SYMVERS_LINUX_TRUE@am__append_2 = \ +@COND_SYMVERS_LINUX_TRUE@ -Wl,--version-script=$(top_srcdir)/src/liblzma/liblzma_linux.map + +@COND_THREADS_TRUE@am__append_3 = ../common/tuklib_cpucores.c +@COND_THREADS_TRUE@am__append_4 = \ +@COND_THREADS_TRUE@ common/hardware_cputhreads.c \ +@COND_THREADS_TRUE@ common/outqueue.c \ +@COND_THREADS_TRUE@ common/outqueue.h + +@COND_MAIN_ENCODER_TRUE@am__append_5 = \ +@COND_MAIN_ENCODER_TRUE@ common/alone_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/block_buffer_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/block_buffer_encoder.h \ +@COND_MAIN_ENCODER_TRUE@ common/block_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/block_encoder.h \ +@COND_MAIN_ENCODER_TRUE@ common/block_header_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/easy_buffer_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/easy_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/easy_encoder_memusage.c \ +@COND_MAIN_ENCODER_TRUE@ common/filter_buffer_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/filter_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/filter_encoder.h \ +@COND_MAIN_ENCODER_TRUE@ common/filter_flags_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/index_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/index_encoder.h \ +@COND_MAIN_ENCODER_TRUE@ common/stream_buffer_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/stream_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/stream_flags_encoder.c \ +@COND_MAIN_ENCODER_TRUE@ common/vli_encoder.c + +@COND_MAIN_ENCODER_TRUE@@COND_THREADS_TRUE@am__append_6 = \ +@COND_MAIN_ENCODER_TRUE@@COND_THREADS_TRUE@ common/stream_encoder_mt.c + +@COND_MAIN_ENCODER_TRUE@@COND_MICROLZMA_TRUE@am__append_7 = \ +@COND_MAIN_ENCODER_TRUE@@COND_MICROLZMA_TRUE@ common/microlzma_encoder.c + +@COND_MAIN_DECODER_TRUE@am__append_8 = \ +@COND_MAIN_DECODER_TRUE@ common/alone_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/alone_decoder.h \ +@COND_MAIN_DECODER_TRUE@ common/auto_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/block_buffer_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/block_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/block_decoder.h \ +@COND_MAIN_DECODER_TRUE@ common/block_header_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/easy_decoder_memusage.c \ +@COND_MAIN_DECODER_TRUE@ common/file_info.c \ +@COND_MAIN_DECODER_TRUE@ common/filter_buffer_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/filter_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/filter_decoder.h \ +@COND_MAIN_DECODER_TRUE@ common/filter_flags_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/index_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/index_decoder.h \ +@COND_MAIN_DECODER_TRUE@ common/index_hash.c \ +@COND_MAIN_DECODER_TRUE@ common/stream_buffer_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/stream_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/stream_decoder.h \ +@COND_MAIN_DECODER_TRUE@ common/stream_flags_decoder.c \ +@COND_MAIN_DECODER_TRUE@ common/vli_decoder.c + +@COND_MAIN_DECODER_TRUE@@COND_THREADS_TRUE@am__append_9 = \ +@COND_MAIN_DECODER_TRUE@@COND_THREADS_TRUE@ common/stream_decoder_mt.c + +@COND_MAIN_DECODER_TRUE@@COND_MICROLZMA_TRUE@am__append_10 = \ +@COND_MAIN_DECODER_TRUE@@COND_MICROLZMA_TRUE@ common/microlzma_decoder.c + +@COND_LZIP_DECODER_TRUE@@COND_MAIN_DECODER_TRUE@am__append_11 = \ +@COND_LZIP_DECODER_TRUE@@COND_MAIN_DECODER_TRUE@ common/lzip_decoder.c \ +@COND_LZIP_DECODER_TRUE@@COND_MAIN_DECODER_TRUE@ common/lzip_decoder.h + +@COND_SMALL_TRUE@am__append_12 = check/crc32_small.c +@COND_SMALL_FALSE@am__append_13 = \ +@COND_SMALL_FALSE@ check/crc32_fast.c \ +@COND_SMALL_FALSE@ check/crc32_table_le.h \ +@COND_SMALL_FALSE@ check/crc32_table_be.h + +@COND_ASM_X86_TRUE@@COND_SMALL_FALSE@am__append_14 = check/crc32_x86.S +@COND_CHECK_CRC64_TRUE@@COND_SMALL_TRUE@am__append_15 = check/crc64_small.c +@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@am__append_16 = \ +@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@ check/crc64_fast.c \ +@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@ check/crc64_table_le.h \ +@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@ check/crc64_table_be.h + +@COND_ASM_X86_TRUE@@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@am__append_17 = check/crc64_x86.S +@COND_CHECK_SHA256_TRUE@@COND_INTERNAL_SHA256_TRUE@am__append_18 = check/sha256.c +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@am__append_19 = \ +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_encoder.c \ +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_encoder.h \ +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_encoder_hash.h \ +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_encoder_hash_table.h \ +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_encoder_mf.c + +@COND_DECODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@am__append_20 = \ +@COND_DECODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_decoder.c \ +@COND_DECODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ lz/lz_decoder.h + +@COND_FILTER_LZMA1_TRUE@am__append_21 = lzma/fastpos_tablegen.c \ +@COND_FILTER_LZMA1_TRUE@ rangecoder/price_tablegen.c +@COND_FILTER_LZMA1_TRUE@am__append_22 = \ +@COND_FILTER_LZMA1_TRUE@ lzma/lzma_common.h \ +@COND_FILTER_LZMA1_TRUE@ lzma/lzma_encoder_presets.c + +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__append_23 = \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/fastpos.h \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_encoder.h \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_encoder.c \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_encoder_private.h \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_encoder_optimum_fast.c \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_encoder_optimum_normal.c + +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@@COND_SMALL_FALSE@am__append_24 = lzma/fastpos_table.c +@COND_DECODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__append_25 = \ +@COND_DECODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_decoder.c \ +@COND_DECODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma_decoder.h + +@COND_ENCODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@am__append_26 = \ +@COND_ENCODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma2_encoder.c \ +@COND_ENCODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma2_encoder.h + +@COND_DECODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@am__append_27 = \ +@COND_DECODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma2_decoder.c \ +@COND_DECODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@ lzma/lzma2_decoder.h + +@COND_FILTER_LZMA1_TRUE@am__append_28 = rangecoder/range_common.h +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__append_29 = \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ rangecoder/range_encoder.h \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ rangecoder/price.h \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ rangecoder/price_table.c + +@COND_DECODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__append_30 = rangecoder/range_decoder.h +@COND_FILTER_DELTA_TRUE@am__append_31 = \ +@COND_FILTER_DELTA_TRUE@ delta/delta_common.c \ +@COND_FILTER_DELTA_TRUE@ delta/delta_common.h \ +@COND_FILTER_DELTA_TRUE@ delta/delta_private.h + +@COND_ENCODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@am__append_32 = \ +@COND_ENCODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@ delta/delta_encoder.c \ +@COND_ENCODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@ delta/delta_encoder.h + +@COND_DECODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@am__append_33 = \ +@COND_DECODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@ delta/delta_decoder.c \ +@COND_DECODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@ delta/delta_decoder.h + +@COND_FILTER_SIMPLE_TRUE@am__append_34 = \ +@COND_FILTER_SIMPLE_TRUE@ simple/simple_coder.c \ +@COND_FILTER_SIMPLE_TRUE@ simple/simple_coder.h \ +@COND_FILTER_SIMPLE_TRUE@ simple/simple_private.h + +@COND_ENCODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_35 = \ +@COND_ENCODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@ simple/simple_encoder.c \ +@COND_ENCODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@ simple/simple_encoder.h + +@COND_DECODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_36 = \ +@COND_DECODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@ simple/simple_decoder.c \ +@COND_DECODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@ simple/simple_decoder.h + +@COND_FILTER_SIMPLE_TRUE@@COND_FILTER_X86_TRUE@am__append_37 = simple/x86.c +@COND_FILTER_POWERPC_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_38 = simple/powerpc.c +@COND_FILTER_IA64_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_39 = simple/ia64.c +@COND_FILTER_ARM_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_40 = simple/arm.c +@COND_FILTER_ARMTHUMB_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_41 = simple/armthumb.c +@COND_FILTER_ARM64_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_42 = simple/arm64.c +@COND_FILTER_SIMPLE_TRUE@@COND_FILTER_SPARC_TRUE@am__append_43 = simple/sparc.c +@COND_FILTER_RISCV_TRUE@@COND_FILTER_SIMPLE_TRUE@am__append_44 = simple/riscv.c +@COND_W32_TRUE@am__append_45 = liblzma.def liblzma.def.in empty.c +@COND_W32_TRUE@am__append_46 = liblzma_w32res.rc +@COND_W32_TRUE@am__append_47 = -Xlinker --output-def -Xlinker liblzma.def.in +@COND_SHARED_TRUE@@COND_W32_TRUE@am__append_48 = liblzma.def +subdir = src/liblzma +ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 +am__aclocal_m4_deps = $(top_srcdir)/m4/ax_pthread.m4 \ + $(top_srcdir)/m4/build-to-host.m4 $(top_srcdir)/m4/getopt.m4 \ + $(top_srcdir)/m4/gettext.m4 $(top_srcdir)/m4/host-cpu-c-abi.m4 \ + $(top_srcdir)/m4/iconv.m4 $(top_srcdir)/m4/intlmacosx.m4 \ + $(top_srcdir)/m4/lib-ld.m4 $(top_srcdir)/m4/lib-link.m4 \ + $(top_srcdir)/m4/lib-prefix.m4 $(top_srcdir)/m4/libtool.m4 \ + $(top_srcdir)/m4/ltoptions.m4 $(top_srcdir)/m4/ltsugar.m4 \ + $(top_srcdir)/m4/ltversion.m4 $(top_srcdir)/m4/lt~obsolete.m4 \ + $(top_srcdir)/m4/nls.m4 $(top_srcdir)/m4/po.m4 \ + $(top_srcdir)/m4/posix-shell.m4 $(top_srcdir)/m4/progtest.m4 \ + $(top_srcdir)/m4/tuklib_common.m4 \ + $(top_srcdir)/m4/tuklib_cpucores.m4 \ + $(top_srcdir)/m4/tuklib_integer.m4 \ + $(top_srcdir)/m4/tuklib_mbstr.m4 \ + $(top_srcdir)/m4/tuklib_physmem.m4 \ + $(top_srcdir)/m4/tuklib_progname.m4 \ + $(top_srcdir)/m4/visibility.m4 $(top_srcdir)/configure.ac +am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \ + $(ACLOCAL_M4) +DIST_COMMON = $(srcdir)/Makefile.am $(am__DIST_COMMON) +mkinstalldirs = $(install_sh) -d +CONFIG_HEADER = $(top_builddir)/config.h +CONFIG_CLEAN_FILES = +CONFIG_CLEAN_VPATH_FILES = +am__vpath_adj_setup = srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; +am__vpath_adj = case $$p in \ + $(srcdir)/*) f=`echo "$$p" | sed "s|^$$srcdirstrip/||"`;; \ + *) f=$$p;; \ + esac; +am__strip_dir = f=`echo $$p | sed -e 's|^.*/||'`; +am__install_max = 40 +am__nobase_strip_setup = \ + srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*|]/\\\\&/g'` +am__nobase_strip = \ + for p in $$list; do echo "$$p"; done | sed -e "s|$$srcdirstrip/||" +am__nobase_list = $(am__nobase_strip_setup); \ + for p in $$list; do echo "$$p $$p"; done | \ + sed "s| $$srcdirstrip/| |;"' / .*\//!s/ .*/ ./; s,\( .*\)/[^/]*$$,\1,' | \ + $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \ + if (++n[$$2] == $(am__install_max)) \ + { print $$2, files[$$2]; n[$$2] = 0; files[$$2] = "" } } \ + END { for (dir in files) print dir, files[dir] }' +am__base_list = \ + sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \ + sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g' +am__uninstall_files_from_dir = { \ + { test ! -d "$$dir" && test ! -f "$$dir" && test ! -r "$$dir"; } \ + || { echo " ( cd '$$dir' && rm -f" $$files ")"; \ + $(am__cd) "$$dir" && echo $$files | $(am__xargs_n) 40 $(am__rm_f); }; \ + } +am__installdirs = "$(DESTDIR)$(libdir)" "$(DESTDIR)$(docdir)" \ + "$(DESTDIR)$(pkgconfigdir)" +LTLIBRARIES = $(lib_LTLIBRARIES) +liblzma_la_LIBADD = +am__liblzma_la_SOURCES_DIST = ../common/tuklib_physmem.c \ + ../common/tuklib_cpucores.c common/common.c common/common.h \ + common/memcmplen.h common/block_util.c common/easy_preset.c \ + common/easy_preset.h common/filter_common.c \ + common/filter_common.h common/hardware_physmem.c \ + common/index.c common/index.h common/stream_flags_common.c \ + common/stream_flags_common.h common/string_conversion.c \ + common/vli_size.c common/hardware_cputhreads.c \ + common/outqueue.c common/outqueue.h common/alone_encoder.c \ + common/block_buffer_encoder.c common/block_buffer_encoder.h \ + common/block_encoder.c common/block_encoder.h \ + common/block_header_encoder.c common/easy_buffer_encoder.c \ + common/easy_encoder.c common/easy_encoder_memusage.c \ + common/filter_buffer_encoder.c common/filter_encoder.c \ + common/filter_encoder.h common/filter_flags_encoder.c \ + common/index_encoder.c common/index_encoder.h \ + common/stream_buffer_encoder.c common/stream_encoder.c \ + common/stream_flags_encoder.c common/vli_encoder.c \ + common/stream_encoder_mt.c common/microlzma_encoder.c \ + common/alone_decoder.c common/alone_decoder.h \ + common/auto_decoder.c common/block_buffer_decoder.c \ + common/block_decoder.c common/block_decoder.h \ + common/block_header_decoder.c common/easy_decoder_memusage.c \ + common/file_info.c common/filter_buffer_decoder.c \ + common/filter_decoder.c common/filter_decoder.h \ + common/filter_flags_decoder.c common/index_decoder.c \ + common/index_decoder.h common/index_hash.c \ + common/stream_buffer_decoder.c common/stream_decoder.c \ + common/stream_decoder.h common/stream_flags_decoder.c \ + common/vli_decoder.c common/stream_decoder_mt.c \ + common/microlzma_decoder.c common/lzip_decoder.c \ + common/lzip_decoder.h check/check.c check/check.h \ + check/crc_common.h check/crc_x86_clmul.h check/crc32_arm64.h \ + check/crc32_loongarch.h check/crc32_small.c check/crc32_fast.c \ + check/crc32_table_le.h check/crc32_table_be.h \ + check/crc32_x86.S check/crc64_small.c check/crc64_fast.c \ + check/crc64_table_le.h check/crc64_table_be.h \ + check/crc64_x86.S check/sha256.c lz/lz_encoder.c \ + lz/lz_encoder.h lz/lz_encoder_hash.h \ + lz/lz_encoder_hash_table.h lz/lz_encoder_mf.c lz/lz_decoder.c \ + lz/lz_decoder.h lzma/lzma_common.h lzma/lzma_encoder_presets.c \ + lzma/fastpos.h lzma/lzma_encoder.h lzma/lzma_encoder.c \ + lzma/lzma_encoder_private.h lzma/lzma_encoder_optimum_fast.c \ + lzma/lzma_encoder_optimum_normal.c lzma/fastpos_table.c \ + lzma/lzma_decoder.c lzma/lzma_decoder.h lzma/lzma2_encoder.c \ + lzma/lzma2_encoder.h lzma/lzma2_decoder.c lzma/lzma2_decoder.h \ + rangecoder/range_common.h rangecoder/range_encoder.h \ + rangecoder/price.h rangecoder/price_table.c \ + rangecoder/range_decoder.h delta/delta_common.c \ + delta/delta_common.h delta/delta_private.h \ + delta/delta_encoder.c delta/delta_encoder.h \ + delta/delta_decoder.c delta/delta_decoder.h \ + simple/simple_coder.c simple/simple_coder.h \ + simple/simple_private.h simple/simple_encoder.c \ + simple/simple_encoder.h simple/simple_decoder.c \ + simple/simple_decoder.h simple/x86.c simple/powerpc.c \ + simple/ia64.c simple/arm.c simple/armthumb.c simple/arm64.c \ + simple/sparc.c simple/riscv.c liblzma_w32res.rc +@COND_THREADS_TRUE@am__objects_1 = liblzma_la-tuklib_cpucores.lo +@COND_THREADS_TRUE@am__objects_2 = liblzma_la-hardware_cputhreads.lo \ +@COND_THREADS_TRUE@ liblzma_la-outqueue.lo +@COND_MAIN_ENCODER_TRUE@am__objects_3 = liblzma_la-alone_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-block_buffer_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-block_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-block_header_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-easy_buffer_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-easy_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-easy_encoder_memusage.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-filter_buffer_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-filter_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-filter_flags_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-index_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-stream_buffer_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-stream_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-stream_flags_encoder.lo \ +@COND_MAIN_ENCODER_TRUE@ liblzma_la-vli_encoder.lo +@COND_MAIN_ENCODER_TRUE@@COND_THREADS_TRUE@am__objects_4 = liblzma_la-stream_encoder_mt.lo +@COND_MAIN_ENCODER_TRUE@@COND_MICROLZMA_TRUE@am__objects_5 = liblzma_la-microlzma_encoder.lo +@COND_MAIN_DECODER_TRUE@am__objects_6 = liblzma_la-alone_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-auto_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-block_buffer_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-block_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-block_header_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-easy_decoder_memusage.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-file_info.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-filter_buffer_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-filter_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-filter_flags_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-index_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-index_hash.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-stream_buffer_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-stream_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-stream_flags_decoder.lo \ +@COND_MAIN_DECODER_TRUE@ liblzma_la-vli_decoder.lo +@COND_MAIN_DECODER_TRUE@@COND_THREADS_TRUE@am__objects_7 = liblzma_la-stream_decoder_mt.lo +@COND_MAIN_DECODER_TRUE@@COND_MICROLZMA_TRUE@am__objects_8 = liblzma_la-microlzma_decoder.lo +@COND_LZIP_DECODER_TRUE@@COND_MAIN_DECODER_TRUE@am__objects_9 = liblzma_la-lzip_decoder.lo +@COND_SMALL_TRUE@am__objects_10 = liblzma_la-crc32_small.lo +@COND_SMALL_FALSE@am__objects_11 = liblzma_la-crc32_fast.lo +@COND_ASM_X86_TRUE@@COND_SMALL_FALSE@am__objects_12 = \ +@COND_ASM_X86_TRUE@@COND_SMALL_FALSE@ liblzma_la-crc32_x86.lo +@COND_CHECK_CRC64_TRUE@@COND_SMALL_TRUE@am__objects_13 = liblzma_la-crc64_small.lo +@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@am__objects_14 = liblzma_la-crc64_fast.lo +@COND_ASM_X86_TRUE@@COND_CHECK_CRC64_TRUE@@COND_SMALL_FALSE@am__objects_15 = liblzma_la-crc64_x86.lo +@COND_CHECK_SHA256_TRUE@@COND_INTERNAL_SHA256_TRUE@am__objects_16 = liblzma_la-sha256.lo +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@am__objects_17 = liblzma_la-lz_encoder.lo \ +@COND_ENCODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@ liblzma_la-lz_encoder_mf.lo +@COND_DECODER_LZ_TRUE@@COND_FILTER_LZ_TRUE@am__objects_18 = liblzma_la-lz_decoder.lo +@COND_FILTER_LZMA1_TRUE@am__objects_19 = \ +@COND_FILTER_LZMA1_TRUE@ liblzma_la-lzma_encoder_presets.lo +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__objects_20 = liblzma_la-lzma_encoder.lo \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ liblzma_la-lzma_encoder_optimum_fast.lo \ +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@ liblzma_la-lzma_encoder_optimum_normal.lo +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@@COND_SMALL_FALSE@am__objects_21 = liblzma_la-fastpos_table.lo +@COND_DECODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__objects_22 = liblzma_la-lzma_decoder.lo +@COND_ENCODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@am__objects_23 = liblzma_la-lzma2_encoder.lo +@COND_DECODER_LZMA2_TRUE@@COND_FILTER_LZMA1_TRUE@am__objects_24 = liblzma_la-lzma2_decoder.lo +am__objects_25 = +@COND_ENCODER_LZMA1_TRUE@@COND_FILTER_LZMA1_TRUE@am__objects_26 = liblzma_la-price_table.lo +@COND_FILTER_DELTA_TRUE@am__objects_27 = liblzma_la-delta_common.lo +@COND_ENCODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@am__objects_28 = liblzma_la-delta_encoder.lo +@COND_DECODER_DELTA_TRUE@@COND_FILTER_DELTA_TRUE@am__objects_29 = liblzma_la-delta_decoder.lo +@COND_FILTER_SIMPLE_TRUE@am__objects_30 = liblzma_la-simple_coder.lo +@COND_ENCODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_31 = liblzma_la-simple_encoder.lo +@COND_DECODER_SIMPLE_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_32 = liblzma_la-simple_decoder.lo +@COND_FILTER_SIMPLE_TRUE@@COND_FILTER_X86_TRUE@am__objects_33 = liblzma_la-x86.lo +@COND_FILTER_POWERPC_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_34 = liblzma_la-powerpc.lo +@COND_FILTER_IA64_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_35 = liblzma_la-ia64.lo +@COND_FILTER_ARM_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_36 = liblzma_la-arm.lo +@COND_FILTER_ARMTHUMB_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_37 = liblzma_la-armthumb.lo +@COND_FILTER_ARM64_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_38 = liblzma_la-arm64.lo +@COND_FILTER_SIMPLE_TRUE@@COND_FILTER_SPARC_TRUE@am__objects_39 = liblzma_la-sparc.lo +@COND_FILTER_RISCV_TRUE@@COND_FILTER_SIMPLE_TRUE@am__objects_40 = liblzma_la-riscv.lo +@COND_W32_TRUE@am__objects_41 = liblzma_w32res.lo +am_liblzma_la_OBJECTS = liblzma_la-tuklib_physmem.lo $(am__objects_1) \ + liblzma_la-common.lo liblzma_la-block_util.lo \ + liblzma_la-easy_preset.lo liblzma_la-filter_common.lo \ + liblzma_la-hardware_physmem.lo liblzma_la-index.lo \ + liblzma_la-stream_flags_common.lo \ + liblzma_la-string_conversion.lo liblzma_la-vli_size.lo \ + $(am__objects_2) $(am__objects_3) $(am__objects_4) \ + $(am__objects_5) $(am__objects_6) $(am__objects_7) \ + $(am__objects_8) $(am__objects_9) liblzma_la-check.lo \ + $(am__objects_10) $(am__objects_11) $(am__objects_12) \ + $(am__objects_13) $(am__objects_14) $(am__objects_15) \ + $(am__objects_16) $(am__objects_17) $(am__objects_18) \ + $(am__objects_19) $(am__objects_20) $(am__objects_21) \ + $(am__objects_22) $(am__objects_23) $(am__objects_24) \ + $(am__objects_25) $(am__objects_26) $(am__objects_25) \ + $(am__objects_27) $(am__objects_28) $(am__objects_29) \ + $(am__objects_30) $(am__objects_31) $(am__objects_32) \ + $(am__objects_33) $(am__objects_34) $(am__objects_35) \ + $(am__objects_36) $(am__objects_37) $(am__objects_38) \ + $(am__objects_39) $(am__objects_40) $(am__objects_41) +liblzma_la_OBJECTS = $(am_liblzma_la_OBJECTS) +AM_V_lt = $(am__v_lt_@AM_V@) +am__v_lt_ = $(am__v_lt_@AM_DEFAULT_V@) +am__v_lt_0 = --silent +am__v_lt_1 = +liblzma_la_LINK = $(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \ + $(LIBTOOLFLAGS) --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) \ + $(liblzma_la_LDFLAGS) $(LDFLAGS) -o $@ +AM_V_P = $(am__v_P_@AM_V@) +am__v_P_ = $(am__v_P_@AM_DEFAULT_V@) +am__v_P_0 = false +am__v_P_1 = : +AM_V_GEN = $(am__v_GEN_@AM_V@) +am__v_GEN_ = $(am__v_GEN_@AM_DEFAULT_V@) +am__v_GEN_0 = @echo " GEN " $@; +am__v_GEN_1 = +AM_V_at = $(am__v_at_@AM_V@) +am__v_at_ = $(am__v_at_@AM_DEFAULT_V@) +am__v_at_0 = @ +am__v_at_1 = +DEFAULT_INCLUDES = -I.@am__isrc@ -I$(top_builddir) +depcomp = $(SHELL) $(top_srcdir)/build-aux/depcomp +am__maybe_remake_depfiles = depfiles +am__depfiles_remade = ./$(DEPDIR)/liblzma_la-alone_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-alone_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-arm.Plo \ + ./$(DEPDIR)/liblzma_la-arm64.Plo \ + ./$(DEPDIR)/liblzma_la-armthumb.Plo \ + ./$(DEPDIR)/liblzma_la-auto_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_buffer_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_buffer_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_header_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_header_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-block_util.Plo \ + ./$(DEPDIR)/liblzma_la-check.Plo \ + ./$(DEPDIR)/liblzma_la-common.Plo \ + ./$(DEPDIR)/liblzma_la-crc32_fast.Plo \ + ./$(DEPDIR)/liblzma_la-crc32_small.Plo \ + ./$(DEPDIR)/liblzma_la-crc32_x86.Plo \ + ./$(DEPDIR)/liblzma_la-crc64_fast.Plo \ + ./$(DEPDIR)/liblzma_la-crc64_small.Plo \ + ./$(DEPDIR)/liblzma_la-crc64_x86.Plo \ + ./$(DEPDIR)/liblzma_la-delta_common.Plo \ + ./$(DEPDIR)/liblzma_la-delta_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-delta_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-easy_buffer_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-easy_decoder_memusage.Plo \ + ./$(DEPDIR)/liblzma_la-easy_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-easy_encoder_memusage.Plo \ + ./$(DEPDIR)/liblzma_la-easy_preset.Plo \ + ./$(DEPDIR)/liblzma_la-fastpos_table.Plo \ + ./$(DEPDIR)/liblzma_la-file_info.Plo \ + ./$(DEPDIR)/liblzma_la-filter_buffer_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-filter_buffer_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-filter_common.Plo \ + ./$(DEPDIR)/liblzma_la-filter_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-filter_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-filter_flags_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-filter_flags_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-hardware_cputhreads.Plo \ + ./$(DEPDIR)/liblzma_la-hardware_physmem.Plo \ + ./$(DEPDIR)/liblzma_la-ia64.Plo \ + ./$(DEPDIR)/liblzma_la-index.Plo \ + ./$(DEPDIR)/liblzma_la-index_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-index_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-index_hash.Plo \ + ./$(DEPDIR)/liblzma_la-lz_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-lz_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-lz_encoder_mf.Plo \ + ./$(DEPDIR)/liblzma_la-lzip_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-lzma2_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-lzma2_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-lzma_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-lzma_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Plo \ + ./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Plo \ + ./$(DEPDIR)/liblzma_la-lzma_encoder_presets.Plo \ + ./$(DEPDIR)/liblzma_la-microlzma_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-microlzma_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-outqueue.Plo \ + ./$(DEPDIR)/liblzma_la-powerpc.Plo \ + ./$(DEPDIR)/liblzma_la-price_table.Plo \ + ./$(DEPDIR)/liblzma_la-riscv.Plo \ + ./$(DEPDIR)/liblzma_la-sha256.Plo \ + ./$(DEPDIR)/liblzma_la-simple_coder.Plo \ + ./$(DEPDIR)/liblzma_la-simple_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-simple_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-sparc.Plo \ + ./$(DEPDIR)/liblzma_la-stream_buffer_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-stream_buffer_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-stream_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-stream_decoder_mt.Plo \ + ./$(DEPDIR)/liblzma_la-stream_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-stream_encoder_mt.Plo \ + ./$(DEPDIR)/liblzma_la-stream_flags_common.Plo \ + ./$(DEPDIR)/liblzma_la-stream_flags_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-stream_flags_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-string_conversion.Plo \ + ./$(DEPDIR)/liblzma_la-tuklib_cpucores.Plo \ + ./$(DEPDIR)/liblzma_la-tuklib_physmem.Plo \ + ./$(DEPDIR)/liblzma_la-vli_decoder.Plo \ + ./$(DEPDIR)/liblzma_la-vli_encoder.Plo \ + ./$(DEPDIR)/liblzma_la-vli_size.Plo \ + ./$(DEPDIR)/liblzma_la-x86.Plo +am__mv = mv -f +CPPASCOMPILE = $(CCAS) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \ + $(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CCASFLAGS) $(CCASFLAGS) +LTCPPASCOMPILE = $(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \ + $(LIBTOOLFLAGS) --mode=compile $(CCAS) $(DEFS) \ + $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) $(CPPFLAGS) \ + $(AM_CCASFLAGS) $(CCASFLAGS) +AM_V_CPPAS = $(am__v_CPPAS_@AM_V@) +am__v_CPPAS_ = $(am__v_CPPAS_@AM_DEFAULT_V@) +am__v_CPPAS_0 = @echo " CPPAS " $@; +am__v_CPPAS_1 = +COMPILE = $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) \ + $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) +LTCOMPILE = $(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \ + $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) \ + $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) $(CPPFLAGS) \ + $(AM_CFLAGS) $(CFLAGS) +AM_V_CC = $(am__v_CC_@AM_V@) +am__v_CC_ = $(am__v_CC_@AM_DEFAULT_V@) +am__v_CC_0 = @echo " CC " $@; +am__v_CC_1 = +CCLD = $(CC) +LINK = $(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \ + $(LIBTOOLFLAGS) --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) \ + $(AM_LDFLAGS) $(LDFLAGS) -o $@ +AM_V_CCLD = $(am__v_CCLD_@AM_V@) +am__v_CCLD_ = $(am__v_CCLD_@AM_DEFAULT_V@) +am__v_CCLD_0 = @echo " CCLD " $@; +am__v_CCLD_1 = +SOURCES = $(liblzma_la_SOURCES) +DIST_SOURCES = $(am__liblzma_la_SOURCES_DIST) +RECURSIVE_TARGETS = all-recursive check-recursive cscopelist-recursive \ + ctags-recursive dvi-recursive html-recursive info-recursive \ + install-data-recursive install-dvi-recursive \ + install-exec-recursive install-html-recursive \ + install-info-recursive install-pdf-recursive \ + install-ps-recursive install-recursive installcheck-recursive \ + installdirs-recursive pdf-recursive ps-recursive \ + tags-recursive uninstall-recursive +am__can_run_installinfo = \ + case $$AM_UPDATE_INFO_DIR in \ + n|no|NO) false;; \ + *) (install-info --version) >/dev/null 2>&1;; \ + esac +DATA = $(doc_DATA) $(pkgconfig_DATA) +RECURSIVE_CLEAN_TARGETS = mostlyclean-recursive clean-recursive \ + distclean-recursive maintainer-clean-recursive +am__recursive_targets = \ + $(RECURSIVE_TARGETS) \ + $(RECURSIVE_CLEAN_TARGETS) \ + $(am__extra_recursive_targets) +AM_RECURSIVE_TARGETS = $(am__recursive_targets:-recursive=) TAGS CTAGS \ + distdir distdir-am +am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) +# Read a list of newline-separated strings from the standard input, +# and print each of them once, without duplicates. Input order is +# *not* preserved. +am__uniquify_input = $(AWK) '\ + BEGIN { nonempty = 0; } \ + { items[$$0] = 1; nonempty = 1; } \ + END { if (nonempty) { for (i in items) print i; }; } \ +' +# Make sure the list of sources is unique. This is necessary because, +# e.g., the same source file might be shared among _SOURCES variables +# for different programs/libraries. +am__define_uniq_tagged_files = \ + list='$(am__tagged_files)'; \ + unique=`for i in $$list; do \ + if test -f "$$i"; then echo $$i; else echo $(srcdir)/$$i; fi; \ + done | $(am__uniquify_input)` +DIST_SUBDIRS = $(SUBDIRS) +am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/check/Makefile.inc \ + $(srcdir)/common/Makefile.inc $(srcdir)/delta/Makefile.inc \ + $(srcdir)/lz/Makefile.inc $(srcdir)/lzma/Makefile.inc \ + $(srcdir)/rangecoder/Makefile.inc \ + $(srcdir)/simple/Makefile.inc $(top_srcdir)/build-aux/depcomp +DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) +am__relativize = \ + dir0=`pwd`; \ + sed_first='s,^\([^/]*\)/.*$$,\1,'; \ + sed_rest='s,^[^/]*/*,,'; \ + sed_last='s,^.*/\([^/]*\)$$,\1,'; \ + sed_butlast='s,/*[^/]*$$,,'; \ + while test -n "$$dir1"; do \ + first=`echo "$$dir1" | sed -e "$$sed_first"`; \ + if test "$$first" != "."; then \ + if test "$$first" = ".."; then \ + dir2=`echo "$$dir0" | sed -e "$$sed_last"`/"$$dir2"; \ + dir0=`echo "$$dir0" | sed -e "$$sed_butlast"`; \ + else \ + first2=`echo "$$dir2" | sed -e "$$sed_first"`; \ + if test "$$first2" = "$$first"; then \ + dir2=`echo "$$dir2" | sed -e "$$sed_rest"`; \ + else \ + dir2="../$$dir2"; \ + fi; \ + dir0="$$dir0"/"$$first"; \ + fi; \ + fi; \ + dir1=`echo "$$dir1" | sed -e "$$sed_rest"`; \ + done; \ + reldir="$$dir2" +ACLOCAL = @ACLOCAL@ +AMTAR = @AMTAR@ +AM_CFLAGS = @AM_CFLAGS@ +AM_DEFAULT_VERBOSITY = @AM_DEFAULT_VERBOSITY@ +AR = @AR@ +AS = @AS@ +AUTOCONF = @AUTOCONF@ +AUTOHEADER = @AUTOHEADER@ +AUTOMAKE = @AUTOMAKE@ +AWK = @AWK@ +CC = @CC@ +CCAS = @CCAS@ +CCASDEPMODE = @CCASDEPMODE@ +CCASFLAGS = @CCASFLAGS@ +CCDEPMODE = @CCDEPMODE@ +CFLAGS = @CFLAGS@ +CFLAG_VISIBILITY = @CFLAG_VISIBILITY@ +CPP = @CPP@ +CPPFLAGS = @CPPFLAGS@ +CSCOPE = @CSCOPE@ +CTAGS = @CTAGS@ +CYGPATH_W = @CYGPATH_W@ +DEFS = @DEFS@ +DEPDIR = @DEPDIR@ +DLLTOOL = @DLLTOOL@ +DSYMUTIL = @DSYMUTIL@ +DUMPBIN = @DUMPBIN@ +ECHO_C = @ECHO_C@ +ECHO_N = @ECHO_N@ +ECHO_T = @ECHO_T@ +EGREP = @EGREP@ +ETAGS = @ETAGS@ +EXEEXT = @EXEEXT@ +FGREP = @FGREP@ +FILECMD = @FILECMD@ +GETOPT_H = @GETOPT_H@ +GETTEXT_MACRO_VERSION = @GETTEXT_MACRO_VERSION@ +GMSGFMT = @GMSGFMT@ +GMSGFMT_015 = @GMSGFMT_015@ +GREP = @GREP@ +HAVE_VISIBILITY = @HAVE_VISIBILITY@ +INSTALL = @INSTALL@ +INSTALL_DATA = @INSTALL_DATA@ +INSTALL_PROGRAM = @INSTALL_PROGRAM@ +INSTALL_SCRIPT = @INSTALL_SCRIPT@ +INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@ +INTLLIBS = @INTLLIBS@ +INTL_MACOSX_LIBS = @INTL_MACOSX_LIBS@ +LD = @LD@ +LDFLAGS = @LDFLAGS@ +LIBICONV = @LIBICONV@ +LIBINTL = @LIBINTL@ +LIBOBJS = @LIBOBJS@ +LIBS = @LIBS@ +LIBTOOL = @LIBTOOL@ +LIPO = @LIPO@ +LN_EXEEXT = @LN_EXEEXT@ +LN_S = @LN_S@ +LTLIBICONV = @LTLIBICONV@ +LTLIBINTL = @LTLIBINTL@ +LTLIBOBJS = @LTLIBOBJS@ +LT_SYS_LIBRARY_PATH = @LT_SYS_LIBRARY_PATH@ +MAKEINFO = @MAKEINFO@ +MANIFEST_TOOL = @MANIFEST_TOOL@ +MKDIR_P = @MKDIR_P@ +MSGFMT = @MSGFMT@ +MSGMERGE = @MSGMERGE@ +MSGMERGE_FOR_MSGFMT_OPTION = @MSGMERGE_FOR_MSGFMT_OPTION@ +NM = @NM@ +NMEDIT = @NMEDIT@ +OBJDUMP = @OBJDUMP@ +OBJEXT = @OBJEXT@ +OTOOL = @OTOOL@ +OTOOL64 = @OTOOL64@ +PACKAGE = @PACKAGE@ +PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@ +PACKAGE_NAME = @PACKAGE_NAME@ +PACKAGE_STRING = @PACKAGE_STRING@ +PACKAGE_TARNAME = @PACKAGE_TARNAME@ +PACKAGE_URL = @PACKAGE_URL@ +PACKAGE_VERSION = @PACKAGE_VERSION@ +PATH_SEPARATOR = @PATH_SEPARATOR@ +POSIX_SHELL = @POSIX_SHELL@ +POSUB = @POSUB@ +PREFERABLY_POSIX_SHELL = @PREFERABLY_POSIX_SHELL@ +PTHREAD_CC = @PTHREAD_CC@ +PTHREAD_CFLAGS = @PTHREAD_CFLAGS@ +PTHREAD_CXX = @PTHREAD_CXX@ +PTHREAD_LIBS = @PTHREAD_LIBS@ +RANLIB = @RANLIB@ +RC = @RC@ +SED = @SED@ +SET_MAKE = @SET_MAKE@ +SHELL = @SHELL@ +STRIP = @STRIP@ +USE_NLS = @USE_NLS@ +VERSION = @VERSION@ +XGETTEXT = @XGETTEXT@ +XGETTEXT_015 = @XGETTEXT_015@ +XGETTEXT_EXTRA_OPTIONS = @XGETTEXT_EXTRA_OPTIONS@ +abs_builddir = @abs_builddir@ +abs_srcdir = @abs_srcdir@ +abs_top_builddir = @abs_top_builddir@ +abs_top_srcdir = @abs_top_srcdir@ +ac_ct_AR = @ac_ct_AR@ +ac_ct_CC = @ac_ct_CC@ +ac_ct_DUMPBIN = @ac_ct_DUMPBIN@ +am__include = @am__include@ +am__leading_dot = @am__leading_dot@ +am__quote = @am__quote@ +am__rm_f_notfound = @am__rm_f_notfound@ +am__tar = @am__tar@ +am__untar = @am__untar@ +am__xargs_n = @am__xargs_n@ +ax_pthread_config = @ax_pthread_config@ +bindir = @bindir@ +build = @build@ +build_alias = @build_alias@ +build_cpu = @build_cpu@ +build_os = @build_os@ +build_vendor = @build_vendor@ +builddir = @builddir@ +datadir = @datadir@ +datarootdir = @datarootdir@ +docdir = @docdir@ +dvidir = @dvidir@ +enable_path_for_scripts = @enable_path_for_scripts@ +exec_prefix = @exec_prefix@ +host = @host@ +host_alias = @host_alias@ +host_cpu = @host_cpu@ +host_os = @host_os@ +host_vendor = @host_vendor@ +htmldir = @htmldir@ +includedir = @includedir@ +infodir = @infodir@ +install_sh = @install_sh@ +libdir = @libdir@ +libexecdir = @libexecdir@ +localedir = @localedir@ +localedir_c = @localedir_c@ +localedir_c_make = @localedir_c_make@ +localstatedir = @localstatedir@ +mandir = @mandir@ +mkdir_p = @mkdir_p@ +oldincludedir = @oldincludedir@ +pdfdir = @pdfdir@ +prefix = @prefix@ +program_transform_name = @program_transform_name@ +psdir = @psdir@ +runstatedir = @runstatedir@ +sbindir = @sbindir@ +sharedstatedir = @sharedstatedir@ +srcdir = @srcdir@ +sysconfdir = @sysconfdir@ +target_alias = @target_alias@ +top_build_prefix = @top_build_prefix@ +top_builddir = @top_builddir@ +top_srcdir = @top_srcdir@ +xz = @xz@ +SUBDIRS = api +EXTRA_DIST = liblzma_generic.map liblzma_linux.map validate_map.sh \ + check/crc_clmul_consts_gen.c check/crc32_tablegen.c \ + check/crc64_tablegen.c $(am__append_21) liblzma.pc.in +CLEANFILES = $(am__append_45) +doc_DATA = $(am__append_48) +lib_LTLIBRARIES = liblzma.la +liblzma_la_SOURCES = ../common/tuklib_physmem.c $(am__append_3) \ + common/common.c common/common.h common/memcmplen.h \ + common/block_util.c common/easy_preset.c common/easy_preset.h \ + common/filter_common.c common/filter_common.h \ + common/hardware_physmem.c common/index.c common/index.h \ + common/stream_flags_common.c common/stream_flags_common.h \ + common/string_conversion.c common/vli_size.c $(am__append_4) \ + $(am__append_5) $(am__append_6) $(am__append_7) \ + $(am__append_8) $(am__append_9) $(am__append_10) \ + $(am__append_11) check/check.c check/check.h \ + check/crc_common.h check/crc_x86_clmul.h check/crc32_arm64.h \ + check/crc32_loongarch.h $(am__append_12) $(am__append_13) \ + $(am__append_14) $(am__append_15) $(am__append_16) \ + $(am__append_17) $(am__append_18) $(am__append_19) \ + $(am__append_20) $(am__append_22) $(am__append_23) \ + $(am__append_24) $(am__append_25) $(am__append_26) \ + $(am__append_27) $(am__append_28) $(am__append_29) \ + $(am__append_30) $(am__append_31) $(am__append_32) \ + $(am__append_33) $(am__append_34) $(am__append_35) \ + $(am__append_36) $(am__append_37) $(am__append_38) \ + $(am__append_39) $(am__append_40) $(am__append_41) \ + $(am__append_42) $(am__append_43) $(am__append_44) \ + $(am__append_46) +liblzma_la_CPPFLAGS = \ + -I$(top_srcdir)/src/liblzma/api \ + -I$(top_srcdir)/src/liblzma/common \ + -I$(top_srcdir)/src/liblzma/check \ + -I$(top_srcdir)/src/liblzma/lz \ + -I$(top_srcdir)/src/liblzma/rangecoder \ + -I$(top_srcdir)/src/liblzma/lzma \ + -I$(top_srcdir)/src/liblzma/delta \ + -I$(top_srcdir)/src/liblzma/simple \ + -I$(top_srcdir)/src/common \ + -DTUKLIB_SYMBOL_PREFIX=lzma_ + +liblzma_la_LDFLAGS = -no-undefined -version-info 13:1:8 \ + $(am__append_1) $(am__append_2) $(am__append_47) +pkgconfigdir = $(libdir)/pkgconfig +pkgconfig_DATA = liblzma.pc +pc_verbose = $(pc_verbose_@AM_V@) +pc_verbose_ = $(pc_verbose_@AM_DEFAULT_V@) +pc_verbose_0 = @echo " PC " $@; +all: all-recursive + +.SUFFIXES: +.SUFFIXES: .S .c .lo .o .obj .rc +$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/common/Makefile.inc $(srcdir)/check/Makefile.inc $(srcdir)/lz/Makefile.inc $(srcdir)/lzma/Makefile.inc $(srcdir)/rangecoder/Makefile.inc $(srcdir)/delta/Makefile.inc $(srcdir)/simple/Makefile.inc $(am__configure_deps) + @for dep in $?; do \ + case '$(am__configure_deps)' in \ + *$$dep*) \ + ( cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh ) \ + && { if test -f $@; then exit 0; else break; fi; }; \ + exit 1;; \ + esac; \ + done; \ + echo ' cd $(top_srcdir) && $(AUTOMAKE) --foreign src/liblzma/Makefile'; \ + $(am__cd) $(top_srcdir) && \ + $(AUTOMAKE) --foreign src/liblzma/Makefile +Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status + @case '$?' in \ + *config.status*) \ + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh;; \ + *) \ + echo ' cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__maybe_remake_depfiles)'; \ + cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__maybe_remake_depfiles);; \ + esac; +$(srcdir)/common/Makefile.inc $(srcdir)/check/Makefile.inc $(srcdir)/lz/Makefile.inc $(srcdir)/lzma/Makefile.inc $(srcdir)/rangecoder/Makefile.inc $(srcdir)/delta/Makefile.inc $(srcdir)/simple/Makefile.inc $(am__empty): + +$(top_builddir)/config.status: $(top_srcdir)/configure $(CONFIG_STATUS_DEPENDENCIES) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh + +$(top_srcdir)/configure: $(am__configure_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(ACLOCAL_M4): $(am__aclocal_m4_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(am__aclocal_m4_deps): + +install-libLTLIBRARIES: $(lib_LTLIBRARIES) + @$(NORMAL_INSTALL) + @list='$(lib_LTLIBRARIES)'; test -n "$(libdir)" || list=; \ + list2=; for p in $$list; do \ + if test -f $$p; then \ + list2="$$list2 $$p"; \ + else :; fi; \ + done; \ + test -z "$$list2" || { \ + echo " $(MKDIR_P) '$(DESTDIR)$(libdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(libdir)" || exit 1; \ + echo " $(LIBTOOL) $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=install $(INSTALL) $(INSTALL_STRIP_FLAG) $$list2 '$(DESTDIR)$(libdir)'"; \ + $(LIBTOOL) $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=install $(INSTALL) $(INSTALL_STRIP_FLAG) $$list2 "$(DESTDIR)$(libdir)"; \ + } + +uninstall-libLTLIBRARIES: + @$(NORMAL_UNINSTALL) + @list='$(lib_LTLIBRARIES)'; test -n "$(libdir)" || list=; \ + for p in $$list; do \ + $(am__strip_dir) \ + echo " $(LIBTOOL) $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=uninstall rm -f '$(DESTDIR)$(libdir)/$$f'"; \ + $(LIBTOOL) $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=uninstall rm -f "$(DESTDIR)$(libdir)/$$f"; \ + done + +clean-libLTLIBRARIES: + -$(am__rm_f) $(lib_LTLIBRARIES) + @list='$(lib_LTLIBRARIES)'; \ + locs=`for p in $$list; do echo $$p; done | \ + sed 's|^[^/]*$$|.|; s|/[^/]*$$||; s|$$|/so_locations|' | \ + sort -u`; \ + echo rm -f $${locs}; \ + $(am__rm_f) $${locs} + +liblzma.la: $(liblzma_la_OBJECTS) $(liblzma_la_DEPENDENCIES) $(EXTRA_liblzma_la_DEPENDENCIES) + $(AM_V_CCLD)$(liblzma_la_LINK) -rpath $(libdir) $(liblzma_la_OBJECTS) $(liblzma_la_LIBADD) $(LIBS) + +mostlyclean-compile: + -rm -f *.$(OBJEXT) + +distclean-compile: + -rm -f *.tab.c + +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-alone_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-alone_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-arm.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-arm64.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-armthumb.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-auto_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_buffer_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_buffer_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_header_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_header_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-block_util.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-check.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-common.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-crc32_fast.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-crc32_small.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-crc32_x86.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-crc64_fast.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-crc64_small.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-crc64_x86.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-delta_common.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-delta_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-delta_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-easy_buffer_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-easy_decoder_memusage.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-easy_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-easy_encoder_memusage.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-easy_preset.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-fastpos_table.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-file_info.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_buffer_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_buffer_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_common.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_flags_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-filter_flags_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-hardware_cputhreads.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-hardware_physmem.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-ia64.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-index.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-index_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-index_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-index_hash.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lz_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lz_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lz_encoder_mf.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzip_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma2_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma2_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-lzma_encoder_presets.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-microlzma_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-microlzma_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-outqueue.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-powerpc.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-price_table.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-riscv.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-sha256.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-simple_coder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-simple_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-simple_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-sparc.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_buffer_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_buffer_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_decoder_mt.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_encoder_mt.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_flags_common.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_flags_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-stream_flags_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-string_conversion.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-tuklib_cpucores.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-tuklib_physmem.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-vli_decoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-vli_encoder.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-vli_size.Plo@am__quote@ # am--include-marker +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/liblzma_la-x86.Plo@am__quote@ # am--include-marker + +$(am__depfiles_remade): + @$(MKDIR_P) $(@D) + @: >>$@ + +am--depfiles: $(am__depfiles_remade) + +.S.o: +@am__fastdepCCAS_TRUE@ $(AM_V_CPPAS)$(CPPASCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ $< +@am__fastdepCCAS_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS)source='$<' object='$@' libtool=no @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ DEPDIR=$(DEPDIR) $(CCASDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS@am__nodep@)$(CPPASCOMPILE) -c -o $@ $< + +.S.obj: +@am__fastdepCCAS_TRUE@ $(AM_V_CPPAS)$(CPPASCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ `$(CYGPATH_W) '$<'` +@am__fastdepCCAS_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS)source='$<' object='$@' libtool=no @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ DEPDIR=$(DEPDIR) $(CCASDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS@am__nodep@)$(CPPASCOMPILE) -c -o $@ `$(CYGPATH_W) '$<'` + +.S.lo: +@am__fastdepCCAS_TRUE@ $(AM_V_CPPAS)$(LTCPPASCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ $< +@am__fastdepCCAS_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Plo +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS)source='$<' object='$@' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ DEPDIR=$(DEPDIR) $(CCASDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS@am__nodep@)$(LTCPPASCOMPILE) -c -o $@ $< + +liblzma_la-crc32_x86.lo: check/crc32_x86.S +@am__fastdepCCAS_TRUE@ $(AM_V_CPPAS)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CCAS) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CCASFLAGS) $(CCASFLAGS) -MT liblzma_la-crc32_x86.lo -MD -MP -MF $(DEPDIR)/liblzma_la-crc32_x86.Tpo -c -o liblzma_la-crc32_x86.lo `test -f 'check/crc32_x86.S' || echo '$(srcdir)/'`check/crc32_x86.S +@am__fastdepCCAS_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-crc32_x86.Tpo $(DEPDIR)/liblzma_la-crc32_x86.Plo +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS)source='check/crc32_x86.S' object='liblzma_la-crc32_x86.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ DEPDIR=$(DEPDIR) $(CCASDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CCAS) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CCASFLAGS) $(CCASFLAGS) -c -o liblzma_la-crc32_x86.lo `test -f 'check/crc32_x86.S' || echo '$(srcdir)/'`check/crc32_x86.S + +liblzma_la-crc64_x86.lo: check/crc64_x86.S +@am__fastdepCCAS_TRUE@ $(AM_V_CPPAS)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CCAS) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CCASFLAGS) $(CCASFLAGS) -MT liblzma_la-crc64_x86.lo -MD -MP -MF $(DEPDIR)/liblzma_la-crc64_x86.Tpo -c -o liblzma_la-crc64_x86.lo `test -f 'check/crc64_x86.S' || echo '$(srcdir)/'`check/crc64_x86.S +@am__fastdepCCAS_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-crc64_x86.Tpo $(DEPDIR)/liblzma_la-crc64_x86.Plo +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS)source='check/crc64_x86.S' object='liblzma_la-crc64_x86.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCCAS_FALSE@ DEPDIR=$(DEPDIR) $(CCASDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCCAS_FALSE@ $(AM_V_CPPAS@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CCAS) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CCASFLAGS) $(CCASFLAGS) -c -o liblzma_la-crc64_x86.lo `test -f 'check/crc64_x86.S' || echo '$(srcdir)/'`check/crc64_x86.S + +.c.o: +@am__fastdepCC_TRUE@ $(AM_V_CC)$(COMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ $< +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='$<' object='$@' libtool=no @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(COMPILE) -c -o $@ $< + +.c.obj: +@am__fastdepCC_TRUE@ $(AM_V_CC)$(COMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ `$(CYGPATH_W) '$<'` +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='$<' object='$@' libtool=no @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(COMPILE) -c -o $@ `$(CYGPATH_W) '$<'` + +.c.lo: +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LTCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ $< +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='$<' object='$@' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LTCOMPILE) -c -o $@ $< + +liblzma_la-tuklib_physmem.lo: ../common/tuklib_physmem.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-tuklib_physmem.lo -MD -MP -MF $(DEPDIR)/liblzma_la-tuklib_physmem.Tpo -c -o liblzma_la-tuklib_physmem.lo `test -f '../common/tuklib_physmem.c' || echo '$(srcdir)/'`../common/tuklib_physmem.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-tuklib_physmem.Tpo $(DEPDIR)/liblzma_la-tuklib_physmem.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='../common/tuklib_physmem.c' object='liblzma_la-tuklib_physmem.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-tuklib_physmem.lo `test -f '../common/tuklib_physmem.c' || echo '$(srcdir)/'`../common/tuklib_physmem.c + +liblzma_la-tuklib_cpucores.lo: ../common/tuklib_cpucores.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-tuklib_cpucores.lo -MD -MP -MF $(DEPDIR)/liblzma_la-tuklib_cpucores.Tpo -c -o liblzma_la-tuklib_cpucores.lo `test -f '../common/tuklib_cpucores.c' || echo '$(srcdir)/'`../common/tuklib_cpucores.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-tuklib_cpucores.Tpo $(DEPDIR)/liblzma_la-tuklib_cpucores.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='../common/tuklib_cpucores.c' object='liblzma_la-tuklib_cpucores.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-tuklib_cpucores.lo `test -f '../common/tuklib_cpucores.c' || echo '$(srcdir)/'`../common/tuklib_cpucores.c + +liblzma_la-common.lo: common/common.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-common.lo -MD -MP -MF $(DEPDIR)/liblzma_la-common.Tpo -c -o liblzma_la-common.lo `test -f 'common/common.c' || echo '$(srcdir)/'`common/common.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-common.Tpo $(DEPDIR)/liblzma_la-common.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/common.c' object='liblzma_la-common.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-common.lo `test -f 'common/common.c' || echo '$(srcdir)/'`common/common.c + +liblzma_la-block_util.lo: common/block_util.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_util.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_util.Tpo -c -o liblzma_la-block_util.lo `test -f 'common/block_util.c' || echo '$(srcdir)/'`common/block_util.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_util.Tpo $(DEPDIR)/liblzma_la-block_util.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_util.c' object='liblzma_la-block_util.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_util.lo `test -f 'common/block_util.c' || echo '$(srcdir)/'`common/block_util.c + +liblzma_la-easy_preset.lo: common/easy_preset.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-easy_preset.lo -MD -MP -MF $(DEPDIR)/liblzma_la-easy_preset.Tpo -c -o liblzma_la-easy_preset.lo `test -f 'common/easy_preset.c' || echo '$(srcdir)/'`common/easy_preset.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-easy_preset.Tpo $(DEPDIR)/liblzma_la-easy_preset.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/easy_preset.c' object='liblzma_la-easy_preset.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-easy_preset.lo `test -f 'common/easy_preset.c' || echo '$(srcdir)/'`common/easy_preset.c + +liblzma_la-filter_common.lo: common/filter_common.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_common.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_common.Tpo -c -o liblzma_la-filter_common.lo `test -f 'common/filter_common.c' || echo '$(srcdir)/'`common/filter_common.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_common.Tpo $(DEPDIR)/liblzma_la-filter_common.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_common.c' object='liblzma_la-filter_common.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_common.lo `test -f 'common/filter_common.c' || echo '$(srcdir)/'`common/filter_common.c + +liblzma_la-hardware_physmem.lo: common/hardware_physmem.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-hardware_physmem.lo -MD -MP -MF $(DEPDIR)/liblzma_la-hardware_physmem.Tpo -c -o liblzma_la-hardware_physmem.lo `test -f 'common/hardware_physmem.c' || echo '$(srcdir)/'`common/hardware_physmem.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-hardware_physmem.Tpo $(DEPDIR)/liblzma_la-hardware_physmem.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/hardware_physmem.c' object='liblzma_la-hardware_physmem.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-hardware_physmem.lo `test -f 'common/hardware_physmem.c' || echo '$(srcdir)/'`common/hardware_physmem.c + +liblzma_la-index.lo: common/index.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-index.lo -MD -MP -MF $(DEPDIR)/liblzma_la-index.Tpo -c -o liblzma_la-index.lo `test -f 'common/index.c' || echo '$(srcdir)/'`common/index.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-index.Tpo $(DEPDIR)/liblzma_la-index.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/index.c' object='liblzma_la-index.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-index.lo `test -f 'common/index.c' || echo '$(srcdir)/'`common/index.c + +liblzma_la-stream_flags_common.lo: common/stream_flags_common.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_flags_common.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_flags_common.Tpo -c -o liblzma_la-stream_flags_common.lo `test -f 'common/stream_flags_common.c' || echo '$(srcdir)/'`common/stream_flags_common.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_flags_common.Tpo $(DEPDIR)/liblzma_la-stream_flags_common.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_flags_common.c' object='liblzma_la-stream_flags_common.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_flags_common.lo `test -f 'common/stream_flags_common.c' || echo '$(srcdir)/'`common/stream_flags_common.c + +liblzma_la-string_conversion.lo: common/string_conversion.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-string_conversion.lo -MD -MP -MF $(DEPDIR)/liblzma_la-string_conversion.Tpo -c -o liblzma_la-string_conversion.lo `test -f 'common/string_conversion.c' || echo '$(srcdir)/'`common/string_conversion.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-string_conversion.Tpo $(DEPDIR)/liblzma_la-string_conversion.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/string_conversion.c' object='liblzma_la-string_conversion.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-string_conversion.lo `test -f 'common/string_conversion.c' || echo '$(srcdir)/'`common/string_conversion.c + +liblzma_la-vli_size.lo: common/vli_size.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-vli_size.lo -MD -MP -MF $(DEPDIR)/liblzma_la-vli_size.Tpo -c -o liblzma_la-vli_size.lo `test -f 'common/vli_size.c' || echo '$(srcdir)/'`common/vli_size.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-vli_size.Tpo $(DEPDIR)/liblzma_la-vli_size.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/vli_size.c' object='liblzma_la-vli_size.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-vli_size.lo `test -f 'common/vli_size.c' || echo '$(srcdir)/'`common/vli_size.c + +liblzma_la-hardware_cputhreads.lo: common/hardware_cputhreads.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-hardware_cputhreads.lo -MD -MP -MF $(DEPDIR)/liblzma_la-hardware_cputhreads.Tpo -c -o liblzma_la-hardware_cputhreads.lo `test -f 'common/hardware_cputhreads.c' || echo '$(srcdir)/'`common/hardware_cputhreads.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-hardware_cputhreads.Tpo $(DEPDIR)/liblzma_la-hardware_cputhreads.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/hardware_cputhreads.c' object='liblzma_la-hardware_cputhreads.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-hardware_cputhreads.lo `test -f 'common/hardware_cputhreads.c' || echo '$(srcdir)/'`common/hardware_cputhreads.c + +liblzma_la-outqueue.lo: common/outqueue.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-outqueue.lo -MD -MP -MF $(DEPDIR)/liblzma_la-outqueue.Tpo -c -o liblzma_la-outqueue.lo `test -f 'common/outqueue.c' || echo '$(srcdir)/'`common/outqueue.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-outqueue.Tpo $(DEPDIR)/liblzma_la-outqueue.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/outqueue.c' object='liblzma_la-outqueue.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-outqueue.lo `test -f 'common/outqueue.c' || echo '$(srcdir)/'`common/outqueue.c + +liblzma_la-alone_encoder.lo: common/alone_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-alone_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-alone_encoder.Tpo -c -o liblzma_la-alone_encoder.lo `test -f 'common/alone_encoder.c' || echo '$(srcdir)/'`common/alone_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-alone_encoder.Tpo $(DEPDIR)/liblzma_la-alone_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/alone_encoder.c' object='liblzma_la-alone_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-alone_encoder.lo `test -f 'common/alone_encoder.c' || echo '$(srcdir)/'`common/alone_encoder.c + +liblzma_la-block_buffer_encoder.lo: common/block_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_buffer_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_buffer_encoder.Tpo -c -o liblzma_la-block_buffer_encoder.lo `test -f 'common/block_buffer_encoder.c' || echo '$(srcdir)/'`common/block_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_buffer_encoder.Tpo $(DEPDIR)/liblzma_la-block_buffer_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_buffer_encoder.c' object='liblzma_la-block_buffer_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_buffer_encoder.lo `test -f 'common/block_buffer_encoder.c' || echo '$(srcdir)/'`common/block_buffer_encoder.c + +liblzma_la-block_encoder.lo: common/block_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_encoder.Tpo -c -o liblzma_la-block_encoder.lo `test -f 'common/block_encoder.c' || echo '$(srcdir)/'`common/block_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_encoder.Tpo $(DEPDIR)/liblzma_la-block_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_encoder.c' object='liblzma_la-block_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_encoder.lo `test -f 'common/block_encoder.c' || echo '$(srcdir)/'`common/block_encoder.c + +liblzma_la-block_header_encoder.lo: common/block_header_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_header_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_header_encoder.Tpo -c -o liblzma_la-block_header_encoder.lo `test -f 'common/block_header_encoder.c' || echo '$(srcdir)/'`common/block_header_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_header_encoder.Tpo $(DEPDIR)/liblzma_la-block_header_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_header_encoder.c' object='liblzma_la-block_header_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_header_encoder.lo `test -f 'common/block_header_encoder.c' || echo '$(srcdir)/'`common/block_header_encoder.c + +liblzma_la-easy_buffer_encoder.lo: common/easy_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-easy_buffer_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-easy_buffer_encoder.Tpo -c -o liblzma_la-easy_buffer_encoder.lo `test -f 'common/easy_buffer_encoder.c' || echo '$(srcdir)/'`common/easy_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-easy_buffer_encoder.Tpo $(DEPDIR)/liblzma_la-easy_buffer_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/easy_buffer_encoder.c' object='liblzma_la-easy_buffer_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-easy_buffer_encoder.lo `test -f 'common/easy_buffer_encoder.c' || echo '$(srcdir)/'`common/easy_buffer_encoder.c + +liblzma_la-easy_encoder.lo: common/easy_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-easy_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-easy_encoder.Tpo -c -o liblzma_la-easy_encoder.lo `test -f 'common/easy_encoder.c' || echo '$(srcdir)/'`common/easy_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-easy_encoder.Tpo $(DEPDIR)/liblzma_la-easy_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/easy_encoder.c' object='liblzma_la-easy_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-easy_encoder.lo `test -f 'common/easy_encoder.c' || echo '$(srcdir)/'`common/easy_encoder.c + +liblzma_la-easy_encoder_memusage.lo: common/easy_encoder_memusage.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-easy_encoder_memusage.lo -MD -MP -MF $(DEPDIR)/liblzma_la-easy_encoder_memusage.Tpo -c -o liblzma_la-easy_encoder_memusage.lo `test -f 'common/easy_encoder_memusage.c' || echo '$(srcdir)/'`common/easy_encoder_memusage.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-easy_encoder_memusage.Tpo $(DEPDIR)/liblzma_la-easy_encoder_memusage.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/easy_encoder_memusage.c' object='liblzma_la-easy_encoder_memusage.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-easy_encoder_memusage.lo `test -f 'common/easy_encoder_memusage.c' || echo '$(srcdir)/'`common/easy_encoder_memusage.c + +liblzma_la-filter_buffer_encoder.lo: common/filter_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_buffer_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_buffer_encoder.Tpo -c -o liblzma_la-filter_buffer_encoder.lo `test -f 'common/filter_buffer_encoder.c' || echo '$(srcdir)/'`common/filter_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_buffer_encoder.Tpo $(DEPDIR)/liblzma_la-filter_buffer_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_buffer_encoder.c' object='liblzma_la-filter_buffer_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_buffer_encoder.lo `test -f 'common/filter_buffer_encoder.c' || echo '$(srcdir)/'`common/filter_buffer_encoder.c + +liblzma_la-filter_encoder.lo: common/filter_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_encoder.Tpo -c -o liblzma_la-filter_encoder.lo `test -f 'common/filter_encoder.c' || echo '$(srcdir)/'`common/filter_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_encoder.Tpo $(DEPDIR)/liblzma_la-filter_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_encoder.c' object='liblzma_la-filter_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_encoder.lo `test -f 'common/filter_encoder.c' || echo '$(srcdir)/'`common/filter_encoder.c + +liblzma_la-filter_flags_encoder.lo: common/filter_flags_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_flags_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_flags_encoder.Tpo -c -o liblzma_la-filter_flags_encoder.lo `test -f 'common/filter_flags_encoder.c' || echo '$(srcdir)/'`common/filter_flags_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_flags_encoder.Tpo $(DEPDIR)/liblzma_la-filter_flags_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_flags_encoder.c' object='liblzma_la-filter_flags_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_flags_encoder.lo `test -f 'common/filter_flags_encoder.c' || echo '$(srcdir)/'`common/filter_flags_encoder.c + +liblzma_la-index_encoder.lo: common/index_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-index_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-index_encoder.Tpo -c -o liblzma_la-index_encoder.lo `test -f 'common/index_encoder.c' || echo '$(srcdir)/'`common/index_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-index_encoder.Tpo $(DEPDIR)/liblzma_la-index_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/index_encoder.c' object='liblzma_la-index_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-index_encoder.lo `test -f 'common/index_encoder.c' || echo '$(srcdir)/'`common/index_encoder.c + +liblzma_la-stream_buffer_encoder.lo: common/stream_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_buffer_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_buffer_encoder.Tpo -c -o liblzma_la-stream_buffer_encoder.lo `test -f 'common/stream_buffer_encoder.c' || echo '$(srcdir)/'`common/stream_buffer_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_buffer_encoder.Tpo $(DEPDIR)/liblzma_la-stream_buffer_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_buffer_encoder.c' object='liblzma_la-stream_buffer_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_buffer_encoder.lo `test -f 'common/stream_buffer_encoder.c' || echo '$(srcdir)/'`common/stream_buffer_encoder.c + +liblzma_la-stream_encoder.lo: common/stream_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_encoder.Tpo -c -o liblzma_la-stream_encoder.lo `test -f 'common/stream_encoder.c' || echo '$(srcdir)/'`common/stream_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_encoder.Tpo $(DEPDIR)/liblzma_la-stream_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_encoder.c' object='liblzma_la-stream_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_encoder.lo `test -f 'common/stream_encoder.c' || echo '$(srcdir)/'`common/stream_encoder.c + +liblzma_la-stream_flags_encoder.lo: common/stream_flags_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_flags_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_flags_encoder.Tpo -c -o liblzma_la-stream_flags_encoder.lo `test -f 'common/stream_flags_encoder.c' || echo '$(srcdir)/'`common/stream_flags_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_flags_encoder.Tpo $(DEPDIR)/liblzma_la-stream_flags_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_flags_encoder.c' object='liblzma_la-stream_flags_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_flags_encoder.lo `test -f 'common/stream_flags_encoder.c' || echo '$(srcdir)/'`common/stream_flags_encoder.c + +liblzma_la-vli_encoder.lo: common/vli_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-vli_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-vli_encoder.Tpo -c -o liblzma_la-vli_encoder.lo `test -f 'common/vli_encoder.c' || echo '$(srcdir)/'`common/vli_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-vli_encoder.Tpo $(DEPDIR)/liblzma_la-vli_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/vli_encoder.c' object='liblzma_la-vli_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-vli_encoder.lo `test -f 'common/vli_encoder.c' || echo '$(srcdir)/'`common/vli_encoder.c + +liblzma_la-stream_encoder_mt.lo: common/stream_encoder_mt.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_encoder_mt.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_encoder_mt.Tpo -c -o liblzma_la-stream_encoder_mt.lo `test -f 'common/stream_encoder_mt.c' || echo '$(srcdir)/'`common/stream_encoder_mt.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_encoder_mt.Tpo $(DEPDIR)/liblzma_la-stream_encoder_mt.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_encoder_mt.c' object='liblzma_la-stream_encoder_mt.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_encoder_mt.lo `test -f 'common/stream_encoder_mt.c' || echo '$(srcdir)/'`common/stream_encoder_mt.c + +liblzma_la-microlzma_encoder.lo: common/microlzma_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-microlzma_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-microlzma_encoder.Tpo -c -o liblzma_la-microlzma_encoder.lo `test -f 'common/microlzma_encoder.c' || echo '$(srcdir)/'`common/microlzma_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-microlzma_encoder.Tpo $(DEPDIR)/liblzma_la-microlzma_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/microlzma_encoder.c' object='liblzma_la-microlzma_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-microlzma_encoder.lo `test -f 'common/microlzma_encoder.c' || echo '$(srcdir)/'`common/microlzma_encoder.c + +liblzma_la-alone_decoder.lo: common/alone_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-alone_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-alone_decoder.Tpo -c -o liblzma_la-alone_decoder.lo `test -f 'common/alone_decoder.c' || echo '$(srcdir)/'`common/alone_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-alone_decoder.Tpo $(DEPDIR)/liblzma_la-alone_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/alone_decoder.c' object='liblzma_la-alone_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-alone_decoder.lo `test -f 'common/alone_decoder.c' || echo '$(srcdir)/'`common/alone_decoder.c + +liblzma_la-auto_decoder.lo: common/auto_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-auto_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-auto_decoder.Tpo -c -o liblzma_la-auto_decoder.lo `test -f 'common/auto_decoder.c' || echo '$(srcdir)/'`common/auto_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-auto_decoder.Tpo $(DEPDIR)/liblzma_la-auto_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/auto_decoder.c' object='liblzma_la-auto_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-auto_decoder.lo `test -f 'common/auto_decoder.c' || echo '$(srcdir)/'`common/auto_decoder.c + +liblzma_la-block_buffer_decoder.lo: common/block_buffer_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_buffer_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_buffer_decoder.Tpo -c -o liblzma_la-block_buffer_decoder.lo `test -f 'common/block_buffer_decoder.c' || echo '$(srcdir)/'`common/block_buffer_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_buffer_decoder.Tpo $(DEPDIR)/liblzma_la-block_buffer_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_buffer_decoder.c' object='liblzma_la-block_buffer_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_buffer_decoder.lo `test -f 'common/block_buffer_decoder.c' || echo '$(srcdir)/'`common/block_buffer_decoder.c + +liblzma_la-block_decoder.lo: common/block_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_decoder.Tpo -c -o liblzma_la-block_decoder.lo `test -f 'common/block_decoder.c' || echo '$(srcdir)/'`common/block_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_decoder.Tpo $(DEPDIR)/liblzma_la-block_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_decoder.c' object='liblzma_la-block_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_decoder.lo `test -f 'common/block_decoder.c' || echo '$(srcdir)/'`common/block_decoder.c + +liblzma_la-block_header_decoder.lo: common/block_header_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-block_header_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-block_header_decoder.Tpo -c -o liblzma_la-block_header_decoder.lo `test -f 'common/block_header_decoder.c' || echo '$(srcdir)/'`common/block_header_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-block_header_decoder.Tpo $(DEPDIR)/liblzma_la-block_header_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/block_header_decoder.c' object='liblzma_la-block_header_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-block_header_decoder.lo `test -f 'common/block_header_decoder.c' || echo '$(srcdir)/'`common/block_header_decoder.c + +liblzma_la-easy_decoder_memusage.lo: common/easy_decoder_memusage.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-easy_decoder_memusage.lo -MD -MP -MF $(DEPDIR)/liblzma_la-easy_decoder_memusage.Tpo -c -o liblzma_la-easy_decoder_memusage.lo `test -f 'common/easy_decoder_memusage.c' || echo '$(srcdir)/'`common/easy_decoder_memusage.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-easy_decoder_memusage.Tpo $(DEPDIR)/liblzma_la-easy_decoder_memusage.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/easy_decoder_memusage.c' object='liblzma_la-easy_decoder_memusage.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-easy_decoder_memusage.lo `test -f 'common/easy_decoder_memusage.c' || echo '$(srcdir)/'`common/easy_decoder_memusage.c + +liblzma_la-file_info.lo: common/file_info.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-file_info.lo -MD -MP -MF $(DEPDIR)/liblzma_la-file_info.Tpo -c -o liblzma_la-file_info.lo `test -f 'common/file_info.c' || echo '$(srcdir)/'`common/file_info.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-file_info.Tpo $(DEPDIR)/liblzma_la-file_info.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/file_info.c' object='liblzma_la-file_info.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-file_info.lo `test -f 'common/file_info.c' || echo '$(srcdir)/'`common/file_info.c + +liblzma_la-filter_buffer_decoder.lo: common/filter_buffer_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_buffer_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_buffer_decoder.Tpo -c -o liblzma_la-filter_buffer_decoder.lo `test -f 'common/filter_buffer_decoder.c' || echo '$(srcdir)/'`common/filter_buffer_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_buffer_decoder.Tpo $(DEPDIR)/liblzma_la-filter_buffer_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_buffer_decoder.c' object='liblzma_la-filter_buffer_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_buffer_decoder.lo `test -f 'common/filter_buffer_decoder.c' || echo '$(srcdir)/'`common/filter_buffer_decoder.c + +liblzma_la-filter_decoder.lo: common/filter_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_decoder.Tpo -c -o liblzma_la-filter_decoder.lo `test -f 'common/filter_decoder.c' || echo '$(srcdir)/'`common/filter_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_decoder.Tpo $(DEPDIR)/liblzma_la-filter_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_decoder.c' object='liblzma_la-filter_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_decoder.lo `test -f 'common/filter_decoder.c' || echo '$(srcdir)/'`common/filter_decoder.c + +liblzma_la-filter_flags_decoder.lo: common/filter_flags_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-filter_flags_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-filter_flags_decoder.Tpo -c -o liblzma_la-filter_flags_decoder.lo `test -f 'common/filter_flags_decoder.c' || echo '$(srcdir)/'`common/filter_flags_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-filter_flags_decoder.Tpo $(DEPDIR)/liblzma_la-filter_flags_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/filter_flags_decoder.c' object='liblzma_la-filter_flags_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-filter_flags_decoder.lo `test -f 'common/filter_flags_decoder.c' || echo '$(srcdir)/'`common/filter_flags_decoder.c + +liblzma_la-index_decoder.lo: common/index_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-index_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-index_decoder.Tpo -c -o liblzma_la-index_decoder.lo `test -f 'common/index_decoder.c' || echo '$(srcdir)/'`common/index_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-index_decoder.Tpo $(DEPDIR)/liblzma_la-index_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/index_decoder.c' object='liblzma_la-index_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-index_decoder.lo `test -f 'common/index_decoder.c' || echo '$(srcdir)/'`common/index_decoder.c + +liblzma_la-index_hash.lo: common/index_hash.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-index_hash.lo -MD -MP -MF $(DEPDIR)/liblzma_la-index_hash.Tpo -c -o liblzma_la-index_hash.lo `test -f 'common/index_hash.c' || echo '$(srcdir)/'`common/index_hash.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-index_hash.Tpo $(DEPDIR)/liblzma_la-index_hash.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/index_hash.c' object='liblzma_la-index_hash.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-index_hash.lo `test -f 'common/index_hash.c' || echo '$(srcdir)/'`common/index_hash.c + +liblzma_la-stream_buffer_decoder.lo: common/stream_buffer_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_buffer_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_buffer_decoder.Tpo -c -o liblzma_la-stream_buffer_decoder.lo `test -f 'common/stream_buffer_decoder.c' || echo '$(srcdir)/'`common/stream_buffer_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_buffer_decoder.Tpo $(DEPDIR)/liblzma_la-stream_buffer_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_buffer_decoder.c' object='liblzma_la-stream_buffer_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_buffer_decoder.lo `test -f 'common/stream_buffer_decoder.c' || echo '$(srcdir)/'`common/stream_buffer_decoder.c + +liblzma_la-stream_decoder.lo: common/stream_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_decoder.Tpo -c -o liblzma_la-stream_decoder.lo `test -f 'common/stream_decoder.c' || echo '$(srcdir)/'`common/stream_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_decoder.Tpo $(DEPDIR)/liblzma_la-stream_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_decoder.c' object='liblzma_la-stream_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_decoder.lo `test -f 'common/stream_decoder.c' || echo '$(srcdir)/'`common/stream_decoder.c + +liblzma_la-stream_flags_decoder.lo: common/stream_flags_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_flags_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_flags_decoder.Tpo -c -o liblzma_la-stream_flags_decoder.lo `test -f 'common/stream_flags_decoder.c' || echo '$(srcdir)/'`common/stream_flags_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_flags_decoder.Tpo $(DEPDIR)/liblzma_la-stream_flags_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_flags_decoder.c' object='liblzma_la-stream_flags_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_flags_decoder.lo `test -f 'common/stream_flags_decoder.c' || echo '$(srcdir)/'`common/stream_flags_decoder.c + +liblzma_la-vli_decoder.lo: common/vli_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-vli_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-vli_decoder.Tpo -c -o liblzma_la-vli_decoder.lo `test -f 'common/vli_decoder.c' || echo '$(srcdir)/'`common/vli_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-vli_decoder.Tpo $(DEPDIR)/liblzma_la-vli_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/vli_decoder.c' object='liblzma_la-vli_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-vli_decoder.lo `test -f 'common/vli_decoder.c' || echo '$(srcdir)/'`common/vli_decoder.c + +liblzma_la-stream_decoder_mt.lo: common/stream_decoder_mt.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-stream_decoder_mt.lo -MD -MP -MF $(DEPDIR)/liblzma_la-stream_decoder_mt.Tpo -c -o liblzma_la-stream_decoder_mt.lo `test -f 'common/stream_decoder_mt.c' || echo '$(srcdir)/'`common/stream_decoder_mt.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-stream_decoder_mt.Tpo $(DEPDIR)/liblzma_la-stream_decoder_mt.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/stream_decoder_mt.c' object='liblzma_la-stream_decoder_mt.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-stream_decoder_mt.lo `test -f 'common/stream_decoder_mt.c' || echo '$(srcdir)/'`common/stream_decoder_mt.c + +liblzma_la-microlzma_decoder.lo: common/microlzma_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-microlzma_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-microlzma_decoder.Tpo -c -o liblzma_la-microlzma_decoder.lo `test -f 'common/microlzma_decoder.c' || echo '$(srcdir)/'`common/microlzma_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-microlzma_decoder.Tpo $(DEPDIR)/liblzma_la-microlzma_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/microlzma_decoder.c' object='liblzma_la-microlzma_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-microlzma_decoder.lo `test -f 'common/microlzma_decoder.c' || echo '$(srcdir)/'`common/microlzma_decoder.c + +liblzma_la-lzip_decoder.lo: common/lzip_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzip_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzip_decoder.Tpo -c -o liblzma_la-lzip_decoder.lo `test -f 'common/lzip_decoder.c' || echo '$(srcdir)/'`common/lzip_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzip_decoder.Tpo $(DEPDIR)/liblzma_la-lzip_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='common/lzip_decoder.c' object='liblzma_la-lzip_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzip_decoder.lo `test -f 'common/lzip_decoder.c' || echo '$(srcdir)/'`common/lzip_decoder.c + +liblzma_la-check.lo: check/check.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-check.lo -MD -MP -MF $(DEPDIR)/liblzma_la-check.Tpo -c -o liblzma_la-check.lo `test -f 'check/check.c' || echo '$(srcdir)/'`check/check.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-check.Tpo $(DEPDIR)/liblzma_la-check.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='check/check.c' object='liblzma_la-check.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-check.lo `test -f 'check/check.c' || echo '$(srcdir)/'`check/check.c + +liblzma_la-crc32_small.lo: check/crc32_small.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-crc32_small.lo -MD -MP -MF $(DEPDIR)/liblzma_la-crc32_small.Tpo -c -o liblzma_la-crc32_small.lo `test -f 'check/crc32_small.c' || echo '$(srcdir)/'`check/crc32_small.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-crc32_small.Tpo $(DEPDIR)/liblzma_la-crc32_small.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='check/crc32_small.c' object='liblzma_la-crc32_small.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-crc32_small.lo `test -f 'check/crc32_small.c' || echo '$(srcdir)/'`check/crc32_small.c + +liblzma_la-crc32_fast.lo: check/crc32_fast.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-crc32_fast.lo -MD -MP -MF $(DEPDIR)/liblzma_la-crc32_fast.Tpo -c -o liblzma_la-crc32_fast.lo `test -f 'check/crc32_fast.c' || echo '$(srcdir)/'`check/crc32_fast.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-crc32_fast.Tpo $(DEPDIR)/liblzma_la-crc32_fast.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='check/crc32_fast.c' object='liblzma_la-crc32_fast.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-crc32_fast.lo `test -f 'check/crc32_fast.c' || echo '$(srcdir)/'`check/crc32_fast.c + +liblzma_la-crc64_small.lo: check/crc64_small.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-crc64_small.lo -MD -MP -MF $(DEPDIR)/liblzma_la-crc64_small.Tpo -c -o liblzma_la-crc64_small.lo `test -f 'check/crc64_small.c' || echo '$(srcdir)/'`check/crc64_small.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-crc64_small.Tpo $(DEPDIR)/liblzma_la-crc64_small.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='check/crc64_small.c' object='liblzma_la-crc64_small.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-crc64_small.lo `test -f 'check/crc64_small.c' || echo '$(srcdir)/'`check/crc64_small.c + +liblzma_la-crc64_fast.lo: check/crc64_fast.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-crc64_fast.lo -MD -MP -MF $(DEPDIR)/liblzma_la-crc64_fast.Tpo -c -o liblzma_la-crc64_fast.lo `test -f 'check/crc64_fast.c' || echo '$(srcdir)/'`check/crc64_fast.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-crc64_fast.Tpo $(DEPDIR)/liblzma_la-crc64_fast.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='check/crc64_fast.c' object='liblzma_la-crc64_fast.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-crc64_fast.lo `test -f 'check/crc64_fast.c' || echo '$(srcdir)/'`check/crc64_fast.c + +liblzma_la-sha256.lo: check/sha256.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-sha256.lo -MD -MP -MF $(DEPDIR)/liblzma_la-sha256.Tpo -c -o liblzma_la-sha256.lo `test -f 'check/sha256.c' || echo '$(srcdir)/'`check/sha256.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-sha256.Tpo $(DEPDIR)/liblzma_la-sha256.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='check/sha256.c' object='liblzma_la-sha256.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-sha256.lo `test -f 'check/sha256.c' || echo '$(srcdir)/'`check/sha256.c + +liblzma_la-lz_encoder.lo: lz/lz_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lz_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lz_encoder.Tpo -c -o liblzma_la-lz_encoder.lo `test -f 'lz/lz_encoder.c' || echo '$(srcdir)/'`lz/lz_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lz_encoder.Tpo $(DEPDIR)/liblzma_la-lz_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lz/lz_encoder.c' object='liblzma_la-lz_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lz_encoder.lo `test -f 'lz/lz_encoder.c' || echo '$(srcdir)/'`lz/lz_encoder.c + +liblzma_la-lz_encoder_mf.lo: lz/lz_encoder_mf.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lz_encoder_mf.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lz_encoder_mf.Tpo -c -o liblzma_la-lz_encoder_mf.lo `test -f 'lz/lz_encoder_mf.c' || echo '$(srcdir)/'`lz/lz_encoder_mf.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lz_encoder_mf.Tpo $(DEPDIR)/liblzma_la-lz_encoder_mf.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lz/lz_encoder_mf.c' object='liblzma_la-lz_encoder_mf.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lz_encoder_mf.lo `test -f 'lz/lz_encoder_mf.c' || echo '$(srcdir)/'`lz/lz_encoder_mf.c + +liblzma_la-lz_decoder.lo: lz/lz_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lz_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lz_decoder.Tpo -c -o liblzma_la-lz_decoder.lo `test -f 'lz/lz_decoder.c' || echo '$(srcdir)/'`lz/lz_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lz_decoder.Tpo $(DEPDIR)/liblzma_la-lz_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lz/lz_decoder.c' object='liblzma_la-lz_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lz_decoder.lo `test -f 'lz/lz_decoder.c' || echo '$(srcdir)/'`lz/lz_decoder.c + +liblzma_la-lzma_encoder_presets.lo: lzma/lzma_encoder_presets.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma_encoder_presets.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma_encoder_presets.Tpo -c -o liblzma_la-lzma_encoder_presets.lo `test -f 'lzma/lzma_encoder_presets.c' || echo '$(srcdir)/'`lzma/lzma_encoder_presets.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma_encoder_presets.Tpo $(DEPDIR)/liblzma_la-lzma_encoder_presets.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma_encoder_presets.c' object='liblzma_la-lzma_encoder_presets.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma_encoder_presets.lo `test -f 'lzma/lzma_encoder_presets.c' || echo '$(srcdir)/'`lzma/lzma_encoder_presets.c + +liblzma_la-lzma_encoder.lo: lzma/lzma_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma_encoder.Tpo -c -o liblzma_la-lzma_encoder.lo `test -f 'lzma/lzma_encoder.c' || echo '$(srcdir)/'`lzma/lzma_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma_encoder.Tpo $(DEPDIR)/liblzma_la-lzma_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma_encoder.c' object='liblzma_la-lzma_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma_encoder.lo `test -f 'lzma/lzma_encoder.c' || echo '$(srcdir)/'`lzma/lzma_encoder.c + +liblzma_la-lzma_encoder_optimum_fast.lo: lzma/lzma_encoder_optimum_fast.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma_encoder_optimum_fast.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Tpo -c -o liblzma_la-lzma_encoder_optimum_fast.lo `test -f 'lzma/lzma_encoder_optimum_fast.c' || echo '$(srcdir)/'`lzma/lzma_encoder_optimum_fast.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Tpo $(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma_encoder_optimum_fast.c' object='liblzma_la-lzma_encoder_optimum_fast.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma_encoder_optimum_fast.lo `test -f 'lzma/lzma_encoder_optimum_fast.c' || echo '$(srcdir)/'`lzma/lzma_encoder_optimum_fast.c + +liblzma_la-lzma_encoder_optimum_normal.lo: lzma/lzma_encoder_optimum_normal.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma_encoder_optimum_normal.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Tpo -c -o liblzma_la-lzma_encoder_optimum_normal.lo `test -f 'lzma/lzma_encoder_optimum_normal.c' || echo '$(srcdir)/'`lzma/lzma_encoder_optimum_normal.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Tpo $(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma_encoder_optimum_normal.c' object='liblzma_la-lzma_encoder_optimum_normal.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma_encoder_optimum_normal.lo `test -f 'lzma/lzma_encoder_optimum_normal.c' || echo '$(srcdir)/'`lzma/lzma_encoder_optimum_normal.c + +liblzma_la-fastpos_table.lo: lzma/fastpos_table.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-fastpos_table.lo -MD -MP -MF $(DEPDIR)/liblzma_la-fastpos_table.Tpo -c -o liblzma_la-fastpos_table.lo `test -f 'lzma/fastpos_table.c' || echo '$(srcdir)/'`lzma/fastpos_table.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-fastpos_table.Tpo $(DEPDIR)/liblzma_la-fastpos_table.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/fastpos_table.c' object='liblzma_la-fastpos_table.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-fastpos_table.lo `test -f 'lzma/fastpos_table.c' || echo '$(srcdir)/'`lzma/fastpos_table.c + +liblzma_la-lzma_decoder.lo: lzma/lzma_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma_decoder.Tpo -c -o liblzma_la-lzma_decoder.lo `test -f 'lzma/lzma_decoder.c' || echo '$(srcdir)/'`lzma/lzma_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma_decoder.Tpo $(DEPDIR)/liblzma_la-lzma_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma_decoder.c' object='liblzma_la-lzma_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma_decoder.lo `test -f 'lzma/lzma_decoder.c' || echo '$(srcdir)/'`lzma/lzma_decoder.c + +liblzma_la-lzma2_encoder.lo: lzma/lzma2_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma2_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma2_encoder.Tpo -c -o liblzma_la-lzma2_encoder.lo `test -f 'lzma/lzma2_encoder.c' || echo '$(srcdir)/'`lzma/lzma2_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma2_encoder.Tpo $(DEPDIR)/liblzma_la-lzma2_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma2_encoder.c' object='liblzma_la-lzma2_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma2_encoder.lo `test -f 'lzma/lzma2_encoder.c' || echo '$(srcdir)/'`lzma/lzma2_encoder.c + +liblzma_la-lzma2_decoder.lo: lzma/lzma2_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-lzma2_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-lzma2_decoder.Tpo -c -o liblzma_la-lzma2_decoder.lo `test -f 'lzma/lzma2_decoder.c' || echo '$(srcdir)/'`lzma/lzma2_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-lzma2_decoder.Tpo $(DEPDIR)/liblzma_la-lzma2_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='lzma/lzma2_decoder.c' object='liblzma_la-lzma2_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-lzma2_decoder.lo `test -f 'lzma/lzma2_decoder.c' || echo '$(srcdir)/'`lzma/lzma2_decoder.c + +liblzma_la-price_table.lo: rangecoder/price_table.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-price_table.lo -MD -MP -MF $(DEPDIR)/liblzma_la-price_table.Tpo -c -o liblzma_la-price_table.lo `test -f 'rangecoder/price_table.c' || echo '$(srcdir)/'`rangecoder/price_table.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-price_table.Tpo $(DEPDIR)/liblzma_la-price_table.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='rangecoder/price_table.c' object='liblzma_la-price_table.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-price_table.lo `test -f 'rangecoder/price_table.c' || echo '$(srcdir)/'`rangecoder/price_table.c + +liblzma_la-delta_common.lo: delta/delta_common.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-delta_common.lo -MD -MP -MF $(DEPDIR)/liblzma_la-delta_common.Tpo -c -o liblzma_la-delta_common.lo `test -f 'delta/delta_common.c' || echo '$(srcdir)/'`delta/delta_common.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-delta_common.Tpo $(DEPDIR)/liblzma_la-delta_common.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='delta/delta_common.c' object='liblzma_la-delta_common.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-delta_common.lo `test -f 'delta/delta_common.c' || echo '$(srcdir)/'`delta/delta_common.c + +liblzma_la-delta_encoder.lo: delta/delta_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-delta_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-delta_encoder.Tpo -c -o liblzma_la-delta_encoder.lo `test -f 'delta/delta_encoder.c' || echo '$(srcdir)/'`delta/delta_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-delta_encoder.Tpo $(DEPDIR)/liblzma_la-delta_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='delta/delta_encoder.c' object='liblzma_la-delta_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-delta_encoder.lo `test -f 'delta/delta_encoder.c' || echo '$(srcdir)/'`delta/delta_encoder.c + +liblzma_la-delta_decoder.lo: delta/delta_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-delta_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-delta_decoder.Tpo -c -o liblzma_la-delta_decoder.lo `test -f 'delta/delta_decoder.c' || echo '$(srcdir)/'`delta/delta_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-delta_decoder.Tpo $(DEPDIR)/liblzma_la-delta_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='delta/delta_decoder.c' object='liblzma_la-delta_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-delta_decoder.lo `test -f 'delta/delta_decoder.c' || echo '$(srcdir)/'`delta/delta_decoder.c + +liblzma_la-simple_coder.lo: simple/simple_coder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-simple_coder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-simple_coder.Tpo -c -o liblzma_la-simple_coder.lo `test -f 'simple/simple_coder.c' || echo '$(srcdir)/'`simple/simple_coder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-simple_coder.Tpo $(DEPDIR)/liblzma_la-simple_coder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/simple_coder.c' object='liblzma_la-simple_coder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-simple_coder.lo `test -f 'simple/simple_coder.c' || echo '$(srcdir)/'`simple/simple_coder.c + +liblzma_la-simple_encoder.lo: simple/simple_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-simple_encoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-simple_encoder.Tpo -c -o liblzma_la-simple_encoder.lo `test -f 'simple/simple_encoder.c' || echo '$(srcdir)/'`simple/simple_encoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-simple_encoder.Tpo $(DEPDIR)/liblzma_la-simple_encoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/simple_encoder.c' object='liblzma_la-simple_encoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-simple_encoder.lo `test -f 'simple/simple_encoder.c' || echo '$(srcdir)/'`simple/simple_encoder.c + +liblzma_la-simple_decoder.lo: simple/simple_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-simple_decoder.lo -MD -MP -MF $(DEPDIR)/liblzma_la-simple_decoder.Tpo -c -o liblzma_la-simple_decoder.lo `test -f 'simple/simple_decoder.c' || echo '$(srcdir)/'`simple/simple_decoder.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-simple_decoder.Tpo $(DEPDIR)/liblzma_la-simple_decoder.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/simple_decoder.c' object='liblzma_la-simple_decoder.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-simple_decoder.lo `test -f 'simple/simple_decoder.c' || echo '$(srcdir)/'`simple/simple_decoder.c + +liblzma_la-x86.lo: simple/x86.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-x86.lo -MD -MP -MF $(DEPDIR)/liblzma_la-x86.Tpo -c -o liblzma_la-x86.lo `test -f 'simple/x86.c' || echo '$(srcdir)/'`simple/x86.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-x86.Tpo $(DEPDIR)/liblzma_la-x86.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/x86.c' object='liblzma_la-x86.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-x86.lo `test -f 'simple/x86.c' || echo '$(srcdir)/'`simple/x86.c + +liblzma_la-powerpc.lo: simple/powerpc.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-powerpc.lo -MD -MP -MF $(DEPDIR)/liblzma_la-powerpc.Tpo -c -o liblzma_la-powerpc.lo `test -f 'simple/powerpc.c' || echo '$(srcdir)/'`simple/powerpc.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-powerpc.Tpo $(DEPDIR)/liblzma_la-powerpc.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/powerpc.c' object='liblzma_la-powerpc.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-powerpc.lo `test -f 'simple/powerpc.c' || echo '$(srcdir)/'`simple/powerpc.c + +liblzma_la-ia64.lo: simple/ia64.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-ia64.lo -MD -MP -MF $(DEPDIR)/liblzma_la-ia64.Tpo -c -o liblzma_la-ia64.lo `test -f 'simple/ia64.c' || echo '$(srcdir)/'`simple/ia64.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-ia64.Tpo $(DEPDIR)/liblzma_la-ia64.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/ia64.c' object='liblzma_la-ia64.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-ia64.lo `test -f 'simple/ia64.c' || echo '$(srcdir)/'`simple/ia64.c + +liblzma_la-arm.lo: simple/arm.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-arm.lo -MD -MP -MF $(DEPDIR)/liblzma_la-arm.Tpo -c -o liblzma_la-arm.lo `test -f 'simple/arm.c' || echo '$(srcdir)/'`simple/arm.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-arm.Tpo $(DEPDIR)/liblzma_la-arm.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/arm.c' object='liblzma_la-arm.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-arm.lo `test -f 'simple/arm.c' || echo '$(srcdir)/'`simple/arm.c + +liblzma_la-armthumb.lo: simple/armthumb.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-armthumb.lo -MD -MP -MF $(DEPDIR)/liblzma_la-armthumb.Tpo -c -o liblzma_la-armthumb.lo `test -f 'simple/armthumb.c' || echo '$(srcdir)/'`simple/armthumb.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-armthumb.Tpo $(DEPDIR)/liblzma_la-armthumb.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/armthumb.c' object='liblzma_la-armthumb.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-armthumb.lo `test -f 'simple/armthumb.c' || echo '$(srcdir)/'`simple/armthumb.c + +liblzma_la-arm64.lo: simple/arm64.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-arm64.lo -MD -MP -MF $(DEPDIR)/liblzma_la-arm64.Tpo -c -o liblzma_la-arm64.lo `test -f 'simple/arm64.c' || echo '$(srcdir)/'`simple/arm64.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-arm64.Tpo $(DEPDIR)/liblzma_la-arm64.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/arm64.c' object='liblzma_la-arm64.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-arm64.lo `test -f 'simple/arm64.c' || echo '$(srcdir)/'`simple/arm64.c + +liblzma_la-sparc.lo: simple/sparc.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-sparc.lo -MD -MP -MF $(DEPDIR)/liblzma_la-sparc.Tpo -c -o liblzma_la-sparc.lo `test -f 'simple/sparc.c' || echo '$(srcdir)/'`simple/sparc.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-sparc.Tpo $(DEPDIR)/liblzma_la-sparc.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/sparc.c' object='liblzma_la-sparc.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-sparc.lo `test -f 'simple/sparc.c' || echo '$(srcdir)/'`simple/sparc.c + +liblzma_la-riscv.lo: simple/riscv.c +@am__fastdepCC_TRUE@ $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -MT liblzma_la-riscv.lo -MD -MP -MF $(DEPDIR)/liblzma_la-riscv.Tpo -c -o liblzma_la-riscv.lo `test -f 'simple/riscv.c' || echo '$(srcdir)/'`simple/riscv.c +@am__fastdepCC_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/liblzma_la-riscv.Tpo $(DEPDIR)/liblzma_la-riscv.Plo +@AMDEP_TRUE@@am__fastdepCC_FALSE@ $(AM_V_CC)source='simple/riscv.c' object='liblzma_la-riscv.lo' libtool=yes @AMDEPBACKSLASH@ +@AMDEP_TRUE@@am__fastdepCC_FALSE@ DEPDIR=$(DEPDIR) $(CCDEPMODE) $(depcomp) @AMDEPBACKSLASH@ +@am__fastdepCC_FALSE@ $(AM_V_CC@am__nodep@)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c -o liblzma_la-riscv.lo `test -f 'simple/riscv.c' || echo '$(srcdir)/'`simple/riscv.c + +mostlyclean-libtool: + -rm -f *.lo + +clean-libtool: + -rm -rf .libs _libs +install-docDATA: $(doc_DATA) + @$(NORMAL_INSTALL) + @list='$(doc_DATA)'; test -n "$(docdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(docdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(docdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(docdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(docdir)" || exit $$?; \ + done + +uninstall-docDATA: + @$(NORMAL_UNINSTALL) + @list='$(doc_DATA)'; test -n "$(docdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(docdir)'; $(am__uninstall_files_from_dir) +install-pkgconfigDATA: $(pkgconfig_DATA) + @$(NORMAL_INSTALL) + @list='$(pkgconfig_DATA)'; test -n "$(pkgconfigdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pkgconfigdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pkgconfigdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(pkgconfigdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(pkgconfigdir)" || exit $$?; \ + done + +uninstall-pkgconfigDATA: + @$(NORMAL_UNINSTALL) + @list='$(pkgconfig_DATA)'; test -n "$(pkgconfigdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(pkgconfigdir)'; $(am__uninstall_files_from_dir) + +# This directory's subdirectories are mostly independent; you can cd +# into them and run 'make' without going through this Makefile. +# To change the values of 'make' variables: instead of editing Makefiles, +# (1) if the variable is set in 'config.status', edit 'config.status' +# (which will cause the Makefiles to be regenerated when you run 'make'); +# (2) otherwise, pass the desired values on the 'make' command line. +$(am__recursive_targets): + @fail=; \ + if $(am__make_keepgoing); then \ + failcom='fail=yes'; \ + else \ + failcom='exit 1'; \ + fi; \ + dot_seen=no; \ + target=`echo $@ | sed s/-recursive//`; \ + case "$@" in \ + distclean-* | maintainer-clean-*) list='$(DIST_SUBDIRS)' ;; \ + *) list='$(SUBDIRS)' ;; \ + esac; \ + for subdir in $$list; do \ + echo "Making $$target in $$subdir"; \ + if test "$$subdir" = "."; then \ + dot_seen=yes; \ + local_target="$$target-am"; \ + else \ + local_target="$$target"; \ + fi; \ + ($(am__cd) $$subdir && $(MAKE) $(AM_MAKEFLAGS) $$local_target) \ + || eval $$failcom; \ + done; \ + if test "$$dot_seen" = "no"; then \ + $(MAKE) $(AM_MAKEFLAGS) "$$target-am" || exit 1; \ + fi; test -z "$$fail" + +ID: $(am__tagged_files) + $(am__define_uniq_tagged_files); mkid -fID $$unique +tags: tags-recursive +TAGS: tags + +tags-am: $(TAGS_DEPENDENCIES) $(am__tagged_files) + set x; \ + here=`pwd`; \ + if ($(ETAGS) --etags-include --version) >/dev/null 2>&1; then \ + include_option=--etags-include; \ + empty_fix=.; \ + else \ + include_option=--include; \ + empty_fix=; \ + fi; \ + list='$(SUBDIRS)'; for subdir in $$list; do \ + if test "$$subdir" = .; then :; else \ + test ! -f $$subdir/TAGS || \ + set "$$@" "$$include_option=$$here/$$subdir/TAGS"; \ + fi; \ + done; \ + $(am__define_uniq_tagged_files); \ + shift; \ + if test -z "$(ETAGS_ARGS)$$*$$unique"; then :; else \ + test -n "$$unique" || unique=$$empty_fix; \ + if test $$# -gt 0; then \ + $(ETAGS) $(ETAGSFLAGS) $(AM_ETAGSFLAGS) $(ETAGS_ARGS) \ + "$$@" $$unique; \ + else \ + $(ETAGS) $(ETAGSFLAGS) $(AM_ETAGSFLAGS) $(ETAGS_ARGS) \ + $$unique; \ + fi; \ + fi +ctags: ctags-recursive + +CTAGS: ctags +ctags-am: $(TAGS_DEPENDENCIES) $(am__tagged_files) + $(am__define_uniq_tagged_files); \ + test -z "$(CTAGS_ARGS)$$unique" \ + || $(CTAGS) $(CTAGSFLAGS) $(AM_CTAGSFLAGS) $(CTAGS_ARGS) \ + $$unique + +GTAGS: + here=`$(am__cd) $(top_builddir) && pwd` \ + && $(am__cd) $(top_srcdir) \ + && gtags -i $(GTAGS_ARGS) "$$here" +cscopelist: cscopelist-recursive + +cscopelist-am: $(am__tagged_files) + list='$(am__tagged_files)'; \ + case "$(srcdir)" in \ + [\\/]* | ?:[\\/]*) sdir="$(srcdir)" ;; \ + *) sdir=$(subdir)/$(srcdir) ;; \ + esac; \ + for i in $$list; do \ + if test -f "$$i"; then \ + echo "$(subdir)/$$i"; \ + else \ + echo "$$sdir/$$i"; \ + fi; \ + done >> $(top_builddir)/cscope.files + +distclean-tags: + -rm -f TAGS ID GTAGS GRTAGS GSYMS GPATH tags +distdir: $(BUILT_SOURCES) + $(MAKE) $(AM_MAKEFLAGS) distdir-am + +distdir-am: $(DISTFILES) + @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + list='$(DISTFILES)'; \ + dist_files=`for file in $$list; do echo $$file; done | \ + sed -e "s|^$$srcdirstrip/||;t" \ + -e "s|^$$topsrcdirstrip/|$(top_builddir)/|;t"`; \ + case $$dist_files in \ + */*) $(MKDIR_P) `echo "$$dist_files" | \ + sed '/\//!d;s|^|$(distdir)/|;s,/[^/]*$$,,' | \ + sort -u` ;; \ + esac; \ + for file in $$dist_files; do \ + if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \ + if test -d $$d/$$file; then \ + dir=`echo "/$$file" | sed -e 's,/[^/]*$$,,'`; \ + if test -d "$(distdir)/$$file"; then \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \ + cp -fpR $(srcdir)/$$file "$(distdir)$$dir" || exit 1; \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + cp -fpR $$d/$$file "$(distdir)$$dir" || exit 1; \ + else \ + test -f "$(distdir)/$$file" \ + || cp -p $$d/$$file "$(distdir)/$$file" \ + || exit 1; \ + fi; \ + done + @list='$(DIST_SUBDIRS)'; for subdir in $$list; do \ + if test "$$subdir" = .; then :; else \ + $(am__make_dryrun) \ + || test -d "$(distdir)/$$subdir" \ + || $(MKDIR_P) "$(distdir)/$$subdir" \ + || exit 1; \ + dir1=$$subdir; dir2="$(distdir)/$$subdir"; \ + $(am__relativize); \ + new_distdir=$$reldir; \ + dir1=$$subdir; dir2="$(top_distdir)"; \ + $(am__relativize); \ + new_top_distdir=$$reldir; \ + echo " (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) top_distdir="$$new_top_distdir" distdir="$$new_distdir" \\"; \ + echo " am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)"; \ + ($(am__cd) $$subdir && \ + $(MAKE) $(AM_MAKEFLAGS) \ + top_distdir="$$new_top_distdir" \ + distdir="$$new_distdir" \ + am__remove_distdir=: \ + am__skip_length_check=: \ + am__skip_mode_fix=: \ + distdir) \ + || exit 1; \ + fi; \ + done +check-am: all-am +check: check-recursive +all-am: Makefile $(LTLIBRARIES) $(DATA) +installdirs: installdirs-recursive +installdirs-am: + for dir in "$(DESTDIR)$(libdir)" "$(DESTDIR)$(docdir)" "$(DESTDIR)$(pkgconfigdir)"; do \ + test -z "$$dir" || $(MKDIR_P) "$$dir"; \ + done +install: install-recursive +install-exec: install-exec-recursive +install-data: install-data-recursive +uninstall: uninstall-recursive + +install-am: all-am + @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am + +installcheck: installcheck-recursive +install-strip: + if test -z '$(STRIP)'; then \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + install; \ + else \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + "INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'" install; \ + fi +mostlyclean-generic: + +clean-generic: + -$(am__rm_f) $(CLEANFILES) + +distclean-generic: + -$(am__rm_f) $(CONFIG_CLEAN_FILES) + -test . = "$(srcdir)" || $(am__rm_f) $(CONFIG_CLEAN_VPATH_FILES) + +maintainer-clean-generic: + @echo "This command is intended for maintainers to use" + @echo "it deletes files that may require special tools to rebuild." +clean: clean-recursive + +clean-am: clean-generic clean-libLTLIBRARIES clean-libtool clean-local \ + mostlyclean-am + +distclean: distclean-recursive + -rm -f ./$(DEPDIR)/liblzma_la-alone_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-alone_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-arm.Plo + -rm -f ./$(DEPDIR)/liblzma_la-arm64.Plo + -rm -f ./$(DEPDIR)/liblzma_la-armthumb.Plo + -rm -f ./$(DEPDIR)/liblzma_la-auto_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_buffer_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_header_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_header_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_util.Plo + -rm -f ./$(DEPDIR)/liblzma_la-check.Plo + -rm -f ./$(DEPDIR)/liblzma_la-common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc32_fast.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc32_small.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc32_x86.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc64_fast.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc64_small.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc64_x86.Plo + -rm -f ./$(DEPDIR)/liblzma_la-delta_common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-delta_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-delta_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_decoder_memusage.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_encoder_memusage.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_preset.Plo + -rm -f ./$(DEPDIR)/liblzma_la-fastpos_table.Plo + -rm -f ./$(DEPDIR)/liblzma_la-file_info.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_buffer_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_flags_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_flags_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-hardware_cputhreads.Plo + -rm -f ./$(DEPDIR)/liblzma_la-hardware_physmem.Plo + -rm -f ./$(DEPDIR)/liblzma_la-ia64.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index_hash.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lz_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lz_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lz_encoder_mf.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzip_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma2_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma2_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder_presets.Plo + -rm -f ./$(DEPDIR)/liblzma_la-microlzma_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-microlzma_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-outqueue.Plo + -rm -f ./$(DEPDIR)/liblzma_la-powerpc.Plo + -rm -f ./$(DEPDIR)/liblzma_la-price_table.Plo + -rm -f ./$(DEPDIR)/liblzma_la-riscv.Plo + -rm -f ./$(DEPDIR)/liblzma_la-sha256.Plo + -rm -f ./$(DEPDIR)/liblzma_la-simple_coder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-simple_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-simple_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-sparc.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_buffer_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_decoder_mt.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_encoder_mt.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_flags_common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_flags_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_flags_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-string_conversion.Plo + -rm -f ./$(DEPDIR)/liblzma_la-tuklib_cpucores.Plo + -rm -f ./$(DEPDIR)/liblzma_la-tuklib_physmem.Plo + -rm -f ./$(DEPDIR)/liblzma_la-vli_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-vli_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-vli_size.Plo + -rm -f ./$(DEPDIR)/liblzma_la-x86.Plo + -rm -f Makefile +distclean-am: clean-am distclean-compile distclean-generic \ + distclean-tags + +dvi: dvi-recursive + +dvi-am: + +html: html-recursive + +html-am: + +info: info-recursive + +info-am: + +install-data-am: install-docDATA install-pkgconfigDATA + +install-dvi: install-dvi-recursive + +install-dvi-am: + +install-exec-am: install-libLTLIBRARIES + +install-html: install-html-recursive + +install-html-am: + +install-info: install-info-recursive + +install-info-am: + +install-man: + +install-pdf: install-pdf-recursive + +install-pdf-am: + +install-ps: install-ps-recursive + +install-ps-am: + +installcheck-am: + +maintainer-clean: maintainer-clean-recursive + -rm -f ./$(DEPDIR)/liblzma_la-alone_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-alone_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-arm.Plo + -rm -f ./$(DEPDIR)/liblzma_la-arm64.Plo + -rm -f ./$(DEPDIR)/liblzma_la-armthumb.Plo + -rm -f ./$(DEPDIR)/liblzma_la-auto_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_buffer_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_header_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_header_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-block_util.Plo + -rm -f ./$(DEPDIR)/liblzma_la-check.Plo + -rm -f ./$(DEPDIR)/liblzma_la-common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc32_fast.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc32_small.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc32_x86.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc64_fast.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc64_small.Plo + -rm -f ./$(DEPDIR)/liblzma_la-crc64_x86.Plo + -rm -f ./$(DEPDIR)/liblzma_la-delta_common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-delta_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-delta_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_decoder_memusage.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_encoder_memusage.Plo + -rm -f ./$(DEPDIR)/liblzma_la-easy_preset.Plo + -rm -f ./$(DEPDIR)/liblzma_la-fastpos_table.Plo + -rm -f ./$(DEPDIR)/liblzma_la-file_info.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_buffer_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_flags_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-filter_flags_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-hardware_cputhreads.Plo + -rm -f ./$(DEPDIR)/liblzma_la-hardware_physmem.Plo + -rm -f ./$(DEPDIR)/liblzma_la-ia64.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-index_hash.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lz_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lz_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lz_encoder_mf.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzip_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma2_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma2_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_fast.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder_optimum_normal.Plo + -rm -f ./$(DEPDIR)/liblzma_la-lzma_encoder_presets.Plo + -rm -f ./$(DEPDIR)/liblzma_la-microlzma_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-microlzma_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-outqueue.Plo + -rm -f ./$(DEPDIR)/liblzma_la-powerpc.Plo + -rm -f ./$(DEPDIR)/liblzma_la-price_table.Plo + -rm -f ./$(DEPDIR)/liblzma_la-riscv.Plo + -rm -f ./$(DEPDIR)/liblzma_la-sha256.Plo + -rm -f ./$(DEPDIR)/liblzma_la-simple_coder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-simple_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-simple_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-sparc.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_buffer_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_buffer_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_decoder_mt.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_encoder_mt.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_flags_common.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_flags_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-stream_flags_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-string_conversion.Plo + -rm -f ./$(DEPDIR)/liblzma_la-tuklib_cpucores.Plo + -rm -f ./$(DEPDIR)/liblzma_la-tuklib_physmem.Plo + -rm -f ./$(DEPDIR)/liblzma_la-vli_decoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-vli_encoder.Plo + -rm -f ./$(DEPDIR)/liblzma_la-vli_size.Plo + -rm -f ./$(DEPDIR)/liblzma_la-x86.Plo + -rm -f Makefile +maintainer-clean-am: distclean-am maintainer-clean-generic + +mostlyclean: mostlyclean-recursive + +mostlyclean-am: mostlyclean-compile mostlyclean-generic \ + mostlyclean-libtool + +pdf: pdf-recursive + +pdf-am: + +ps: ps-recursive + +ps-am: + +uninstall-am: uninstall-docDATA uninstall-libLTLIBRARIES \ + uninstall-pkgconfigDATA + +.MAKE: $(am__recursive_targets) install-am install-strip + +.PHONY: $(am__recursive_targets) CTAGS GTAGS TAGS all all-am \ + am--depfiles check check-am clean clean-generic \ + clean-libLTLIBRARIES clean-libtool clean-local cscopelist-am \ + ctags ctags-am distclean distclean-compile distclean-generic \ + distclean-libtool distclean-tags distdir dvi dvi-am html \ + html-am info info-am install install-am install-data \ + install-data-am install-docDATA install-dvi install-dvi-am \ + install-exec install-exec-am install-html install-html-am \ + install-info install-info-am install-libLTLIBRARIES \ + install-man install-pdf install-pdf-am install-pkgconfigDATA \ + install-ps install-ps-am install-strip installcheck \ + installcheck-am installdirs installdirs-am maintainer-clean \ + maintainer-clean-generic mostlyclean mostlyclean-compile \ + mostlyclean-generic mostlyclean-libtool pdf pdf-am ps ps-am \ + tags tags-am uninstall uninstall-am uninstall-docDATA \ + uninstall-libLTLIBRARIES uninstall-pkgconfigDATA + +.PRECIOUS: Makefile + + +# Windows resource compiler support. libtool knows what to do with .rc +# files, but Automake (<= 1.11 at least) doesn't know. +# +# We want the resource file only in shared liblzma. To avoid linking it into +# static liblzma, we overwrite the static object file with an object file +# compiled from empty input. Note that GNU-specific features are OK here, +# because on Windows we are compiled with the GNU toolchain. +# +# The typedef in empty.c will prevent an empty translation unit, which is +# not allowed by the C standard. It results in a warning with +# -Wempty-translation-unit with Clang or -pedantic for GCC. +.rc.lo: + $(LIBTOOL) --mode=compile $(RC) $(DEFS) $(DEFAULT_INCLUDES) \ + $(INCLUDES) $(liblzma_la_CPPFLAGS) $(CPPFLAGS) $(RCFLAGS) \ + -i $< -o $@ + echo "typedef void empty;" > empty.c + $(COMPILE) -c empty.c -o $(*D)/$(*F).o + +# Remove ordinals from the generated .def file. People must link by name, +# not by ordinal, because no one is going to track the ordinal numbers. +liblzma.def: liblzma.la liblzma.def.in + sed 's/ \+@ *[0-9]\+//' liblzma.def.in > liblzma.def + +# Creating liblzma.def.in is a side effect of linking the library. +liblzma.def.in: liblzma.la + +liblzma.pc: $(srcdir)/liblzma.pc.in + $(AM_V_at)rm -f $@ + $(pc_verbose)sed \ + -e 's,@prefix[@],$(prefix),g' \ + -e 's,@exec_prefix[@],$(exec_prefix),g' \ + -e 's,@libdir[@],$(libdir),g' \ + -e 's,@includedir[@],$(includedir),g' \ + -e 's,@PACKAGE_URL[@],$(PACKAGE_URL),g' \ + -e 's,@PACKAGE_VERSION[@],$(PACKAGE_VERSION),g' \ + -e 's,@PTHREAD_CFLAGS[@],$(PTHREAD_CFLAGS),g' \ + -e 's,@LIBS[@],$(LIBS),g' \ + < $(srcdir)/liblzma.pc.in > $@ || { rm -f $@; exit 1; } + +clean-local: + rm -f liblzma.pc + +# Tell versions [3.59,3.63) of GNU make to not export all variables. +# Otherwise a system limit (for SysV at least) may be exceeded. +.NOEXPORT: + +# Tell GNU make to disable its built-in pattern rules. +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% diff --git a/src/liblzma/api/Makefile.am b/src/liblzma/api/Makefile.am new file mode 100644 index 0000000000..4f91c77a60 --- /dev/null +++ b/src/liblzma/api/Makefile.am @@ -0,0 +1,38 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +nobase_include_HEADERS = \ + lzma.h \ + lzma/base.h \ + lzma/bcj.h \ + lzma/block.h \ + lzma/check.h \ + lzma/container.h \ + lzma/delta.h \ + lzma/filter.h \ + lzma/hardware.h \ + lzma/index.h \ + lzma/index_hash.h \ + lzma/lzma12.h \ + lzma/stream_flags.h \ + lzma/version.h \ + lzma/vli.h + +if COND_DOXYGEN +$(top_builddir)/doc/api/index.html: $(top_srcdir)/doxygen/update-doxygen $(top_srcdir)/doxygen/Doxyfile $(nobase_include_HEADERS) + $(MKDIR_P) "$(top_builddir)/doc" + "$(top_srcdir)/doxygen/update-doxygen" api \ + "$(top_srcdir)" "$(top_builddir)/doc" + +all-local: $(top_builddir)/doc/api/index.html + +install-data-local: + $(MKDIR_P) "$(DESTDIR)$(docdir)/api" + $(INSTALL_DATA) "$(top_builddir)"/doc/api/* "$(DESTDIR)$(docdir)/api" + +uninstall-local: + rm -rf "$(DESTDIR)$(docdir)/api" + +clean-local: + rm -rf "$(top_builddir)/doc/api" +endif diff --git a/src/liblzma/api/Makefile.in b/src/liblzma/api/Makefile.in new file mode 100644 index 0000000000..49d7433838 --- /dev/null +++ b/src/liblzma/api/Makefile.in @@ -0,0 +1,663 @@ +# Makefile.in generated by automake 1.17 from Makefile.am. +# @configure_input@ + +# Copyright (C) 1994-2024 Free Software Foundation, Inc. + +# This Makefile.in is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY, to the extent permitted by law; without +# even the implied warranty of MERCHANTABILITY or FITNESS FOR A +# PARTICULAR PURPOSE. + +@SET_MAKE@ + +VPATH = @srcdir@ +am__is_gnu_make = { \ + if test -z '$(MAKELEVEL)'; then \ + false; \ + elif test -n '$(MAKE_HOST)'; then \ + true; \ + elif test -n '$(MAKE_VERSION)' && test -n '$(CURDIR)'; then \ + true; \ + else \ + false; \ + fi; \ +} +am__make_running_with_option = \ + case $${target_option-} in \ + ?) ;; \ + *) echo "am__make_running_with_option: internal error: invalid" \ + "target option '$${target_option-}' specified" >&2; \ + exit 1;; \ + esac; \ + has_opt=no; \ + sane_makeflags=$$MAKEFLAGS; \ + if $(am__is_gnu_make); then \ + sane_makeflags=$$MFLAGS; \ + else \ + case $$MAKEFLAGS in \ + *\\[\ \ ]*) \ + bs=\\; \ + sane_makeflags=`printf '%s\n' "$$MAKEFLAGS" \ + | sed "s/$$bs$$bs[$$bs $$bs ]*//g"`;; \ + esac; \ + fi; \ + skip_next=no; \ + strip_trailopt () \ + { \ + flg=`printf '%s\n' "$$flg" | sed "s/$$1.*$$//"`; \ + }; \ + for flg in $$sane_makeflags; do \ + test $$skip_next = yes && { skip_next=no; continue; }; \ + case $$flg in \ + *=*|--*) continue;; \ + -*I) strip_trailopt 'I'; skip_next=yes;; \ + -*I?*) strip_trailopt 'I';; \ + -*O) strip_trailopt 'O'; skip_next=yes;; \ + -*O?*) strip_trailopt 'O';; \ + -*l) strip_trailopt 'l'; skip_next=yes;; \ + -*l?*) strip_trailopt 'l';; \ + -[dEDm]) skip_next=yes;; \ + -[JT]) skip_next=yes;; \ + esac; \ + case $$flg in \ + *$$target_option*) has_opt=yes; break;; \ + esac; \ + done; \ + test $$has_opt = yes +am__make_dryrun = (target_option=n; $(am__make_running_with_option)) +am__make_keepgoing = (target_option=k; $(am__make_running_with_option)) +am__rm_f = rm -f $(am__rm_f_notfound) +am__rm_rf = rm -rf $(am__rm_f_notfound) +pkgdatadir = $(datadir)/@PACKAGE@ +pkgincludedir = $(includedir)/@PACKAGE@ +pkglibdir = $(libdir)/@PACKAGE@ +pkglibexecdir = $(libexecdir)/@PACKAGE@ +am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd +install_sh_DATA = $(install_sh) -c -m 644 +install_sh_PROGRAM = $(install_sh) -c +install_sh_SCRIPT = $(install_sh) -c +INSTALL_HEADER = $(INSTALL_DATA) +transform = $(program_transform_name) +NORMAL_INSTALL = : +PRE_INSTALL = : +POST_INSTALL = : +NORMAL_UNINSTALL = : +PRE_UNINSTALL = : +POST_UNINSTALL = : +build_triplet = @build@ +host_triplet = @host@ +subdir = src/liblzma/api +ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 +am__aclocal_m4_deps = $(top_srcdir)/m4/ax_pthread.m4 \ + $(top_srcdir)/m4/build-to-host.m4 $(top_srcdir)/m4/getopt.m4 \ + $(top_srcdir)/m4/gettext.m4 $(top_srcdir)/m4/host-cpu-c-abi.m4 \ + $(top_srcdir)/m4/iconv.m4 $(top_srcdir)/m4/intlmacosx.m4 \ + $(top_srcdir)/m4/lib-ld.m4 $(top_srcdir)/m4/lib-link.m4 \ + $(top_srcdir)/m4/lib-prefix.m4 $(top_srcdir)/m4/libtool.m4 \ + $(top_srcdir)/m4/ltoptions.m4 $(top_srcdir)/m4/ltsugar.m4 \ + $(top_srcdir)/m4/ltversion.m4 $(top_srcdir)/m4/lt~obsolete.m4 \ + $(top_srcdir)/m4/nls.m4 $(top_srcdir)/m4/po.m4 \ + $(top_srcdir)/m4/posix-shell.m4 $(top_srcdir)/m4/progtest.m4 \ + $(top_srcdir)/m4/tuklib_common.m4 \ + $(top_srcdir)/m4/tuklib_cpucores.m4 \ + $(top_srcdir)/m4/tuklib_integer.m4 \ + $(top_srcdir)/m4/tuklib_mbstr.m4 \ + $(top_srcdir)/m4/tuklib_physmem.m4 \ + $(top_srcdir)/m4/tuklib_progname.m4 \ + $(top_srcdir)/m4/visibility.m4 $(top_srcdir)/configure.ac +am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \ + $(ACLOCAL_M4) +DIST_COMMON = $(srcdir)/Makefile.am $(nobase_include_HEADERS) \ + $(am__DIST_COMMON) +mkinstalldirs = $(install_sh) -d +CONFIG_HEADER = $(top_builddir)/config.h +CONFIG_CLEAN_FILES = +CONFIG_CLEAN_VPATH_FILES = +AM_V_P = $(am__v_P_@AM_V@) +am__v_P_ = $(am__v_P_@AM_DEFAULT_V@) +am__v_P_0 = false +am__v_P_1 = : +AM_V_GEN = $(am__v_GEN_@AM_V@) +am__v_GEN_ = $(am__v_GEN_@AM_DEFAULT_V@) +am__v_GEN_0 = @echo " GEN " $@; +am__v_GEN_1 = +AM_V_at = $(am__v_at_@AM_V@) +am__v_at_ = $(am__v_at_@AM_DEFAULT_V@) +am__v_at_0 = @ +am__v_at_1 = +SOURCES = +DIST_SOURCES = +am__can_run_installinfo = \ + case $$AM_UPDATE_INFO_DIR in \ + n|no|NO) false;; \ + *) (install-info --version) >/dev/null 2>&1;; \ + esac +am__vpath_adj_setup = srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; +am__vpath_adj = case $$p in \ + $(srcdir)/*) f=`echo "$$p" | sed "s|^$$srcdirstrip/||"`;; \ + *) f=$$p;; \ + esac; +am__strip_dir = f=`echo $$p | sed -e 's|^.*/||'`; +am__install_max = 40 +am__nobase_strip_setup = \ + srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*|]/\\\\&/g'` +am__nobase_strip = \ + for p in $$list; do echo "$$p"; done | sed -e "s|$$srcdirstrip/||" +am__nobase_list = $(am__nobase_strip_setup); \ + for p in $$list; do echo "$$p $$p"; done | \ + sed "s| $$srcdirstrip/| |;"' / .*\//!s/ .*/ ./; s,\( .*\)/[^/]*$$,\1,' | \ + $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \ + if (++n[$$2] == $(am__install_max)) \ + { print $$2, files[$$2]; n[$$2] = 0; files[$$2] = "" } } \ + END { for (dir in files) print dir, files[dir] }' +am__base_list = \ + sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \ + sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g' +am__uninstall_files_from_dir = { \ + { test ! -d "$$dir" && test ! -f "$$dir" && test ! -r "$$dir"; } \ + || { echo " ( cd '$$dir' && rm -f" $$files ")"; \ + $(am__cd) "$$dir" && echo $$files | $(am__xargs_n) 40 $(am__rm_f); }; \ + } +am__installdirs = "$(DESTDIR)$(includedir)" +HEADERS = $(nobase_include_HEADERS) +am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) +# Read a list of newline-separated strings from the standard input, +# and print each of them once, without duplicates. Input order is +# *not* preserved. +am__uniquify_input = $(AWK) '\ + BEGIN { nonempty = 0; } \ + { items[$$0] = 1; nonempty = 1; } \ + END { if (nonempty) { for (i in items) print i; }; } \ +' +# Make sure the list of sources is unique. This is necessary because, +# e.g., the same source file might be shared among _SOURCES variables +# for different programs/libraries. +am__define_uniq_tagged_files = \ + list='$(am__tagged_files)'; \ + unique=`for i in $$list; do \ + if test -f "$$i"; then echo $$i; else echo $(srcdir)/$$i; fi; \ + done | $(am__uniquify_input)` +am__DIST_COMMON = $(srcdir)/Makefile.in +DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) +ACLOCAL = @ACLOCAL@ +AMTAR = @AMTAR@ +AM_CFLAGS = @AM_CFLAGS@ +AM_DEFAULT_VERBOSITY = @AM_DEFAULT_VERBOSITY@ +AR = @AR@ +AS = @AS@ +AUTOCONF = @AUTOCONF@ +AUTOHEADER = @AUTOHEADER@ +AUTOMAKE = @AUTOMAKE@ +AWK = @AWK@ +CC = @CC@ +CCAS = @CCAS@ +CCASDEPMODE = @CCASDEPMODE@ +CCASFLAGS = @CCASFLAGS@ +CCDEPMODE = @CCDEPMODE@ +CFLAGS = @CFLAGS@ +CFLAG_VISIBILITY = @CFLAG_VISIBILITY@ +CPP = @CPP@ +CPPFLAGS = @CPPFLAGS@ +CSCOPE = @CSCOPE@ +CTAGS = @CTAGS@ +CYGPATH_W = @CYGPATH_W@ +DEFS = @DEFS@ +DEPDIR = @DEPDIR@ +DLLTOOL = @DLLTOOL@ +DSYMUTIL = @DSYMUTIL@ +DUMPBIN = @DUMPBIN@ +ECHO_C = @ECHO_C@ +ECHO_N = @ECHO_N@ +ECHO_T = @ECHO_T@ +EGREP = @EGREP@ +ETAGS = @ETAGS@ +EXEEXT = @EXEEXT@ +FGREP = @FGREP@ +FILECMD = @FILECMD@ +GETOPT_H = @GETOPT_H@ +GETTEXT_MACRO_VERSION = @GETTEXT_MACRO_VERSION@ +GMSGFMT = @GMSGFMT@ +GMSGFMT_015 = @GMSGFMT_015@ +GREP = @GREP@ +HAVE_VISIBILITY = @HAVE_VISIBILITY@ +INSTALL = @INSTALL@ +INSTALL_DATA = @INSTALL_DATA@ +INSTALL_PROGRAM = @INSTALL_PROGRAM@ +INSTALL_SCRIPT = @INSTALL_SCRIPT@ +INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@ +INTLLIBS = @INTLLIBS@ +INTL_MACOSX_LIBS = @INTL_MACOSX_LIBS@ +LD = @LD@ +LDFLAGS = @LDFLAGS@ +LIBICONV = @LIBICONV@ +LIBINTL = @LIBINTL@ +LIBOBJS = @LIBOBJS@ +LIBS = @LIBS@ +LIBTOOL = @LIBTOOL@ +LIPO = @LIPO@ +LN_EXEEXT = @LN_EXEEXT@ +LN_S = @LN_S@ +LTLIBICONV = @LTLIBICONV@ +LTLIBINTL = @LTLIBINTL@ +LTLIBOBJS = @LTLIBOBJS@ +LT_SYS_LIBRARY_PATH = @LT_SYS_LIBRARY_PATH@ +MAKEINFO = @MAKEINFO@ +MANIFEST_TOOL = @MANIFEST_TOOL@ +MKDIR_P = @MKDIR_P@ +MSGFMT = @MSGFMT@ +MSGMERGE = @MSGMERGE@ +MSGMERGE_FOR_MSGFMT_OPTION = @MSGMERGE_FOR_MSGFMT_OPTION@ +NM = @NM@ +NMEDIT = @NMEDIT@ +OBJDUMP = @OBJDUMP@ +OBJEXT = @OBJEXT@ +OTOOL = @OTOOL@ +OTOOL64 = @OTOOL64@ +PACKAGE = @PACKAGE@ +PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@ +PACKAGE_NAME = @PACKAGE_NAME@ +PACKAGE_STRING = @PACKAGE_STRING@ +PACKAGE_TARNAME = @PACKAGE_TARNAME@ +PACKAGE_URL = @PACKAGE_URL@ +PACKAGE_VERSION = @PACKAGE_VERSION@ +PATH_SEPARATOR = @PATH_SEPARATOR@ +POSIX_SHELL = @POSIX_SHELL@ +POSUB = @POSUB@ +PREFERABLY_POSIX_SHELL = @PREFERABLY_POSIX_SHELL@ +PTHREAD_CC = @PTHREAD_CC@ +PTHREAD_CFLAGS = @PTHREAD_CFLAGS@ +PTHREAD_CXX = @PTHREAD_CXX@ +PTHREAD_LIBS = @PTHREAD_LIBS@ +RANLIB = @RANLIB@ +RC = @RC@ +SED = @SED@ +SET_MAKE = @SET_MAKE@ +SHELL = @SHELL@ +STRIP = @STRIP@ +USE_NLS = @USE_NLS@ +VERSION = @VERSION@ +XGETTEXT = @XGETTEXT@ +XGETTEXT_015 = @XGETTEXT_015@ +XGETTEXT_EXTRA_OPTIONS = @XGETTEXT_EXTRA_OPTIONS@ +abs_builddir = @abs_builddir@ +abs_srcdir = @abs_srcdir@ +abs_top_builddir = @abs_top_builddir@ +abs_top_srcdir = @abs_top_srcdir@ +ac_ct_AR = @ac_ct_AR@ +ac_ct_CC = @ac_ct_CC@ +ac_ct_DUMPBIN = @ac_ct_DUMPBIN@ +am__include = @am__include@ +am__leading_dot = @am__leading_dot@ +am__quote = @am__quote@ +am__rm_f_notfound = @am__rm_f_notfound@ +am__tar = @am__tar@ +am__untar = @am__untar@ +am__xargs_n = @am__xargs_n@ +ax_pthread_config = @ax_pthread_config@ +bindir = @bindir@ +build = @build@ +build_alias = @build_alias@ +build_cpu = @build_cpu@ +build_os = @build_os@ +build_vendor = @build_vendor@ +builddir = @builddir@ +datadir = @datadir@ +datarootdir = @datarootdir@ +docdir = @docdir@ +dvidir = @dvidir@ +enable_path_for_scripts = @enable_path_for_scripts@ +exec_prefix = @exec_prefix@ +host = @host@ +host_alias = @host_alias@ +host_cpu = @host_cpu@ +host_os = @host_os@ +host_vendor = @host_vendor@ +htmldir = @htmldir@ +includedir = @includedir@ +infodir = @infodir@ +install_sh = @install_sh@ +libdir = @libdir@ +libexecdir = @libexecdir@ +localedir = @localedir@ +localedir_c = @localedir_c@ +localedir_c_make = @localedir_c_make@ +localstatedir = @localstatedir@ +mandir = @mandir@ +mkdir_p = @mkdir_p@ +oldincludedir = @oldincludedir@ +pdfdir = @pdfdir@ +prefix = @prefix@ +program_transform_name = @program_transform_name@ +psdir = @psdir@ +runstatedir = @runstatedir@ +sbindir = @sbindir@ +sharedstatedir = @sharedstatedir@ +srcdir = @srcdir@ +sysconfdir = @sysconfdir@ +target_alias = @target_alias@ +top_build_prefix = @top_build_prefix@ +top_builddir = @top_builddir@ +top_srcdir = @top_srcdir@ +xz = @xz@ +nobase_include_HEADERS = \ + lzma.h \ + lzma/base.h \ + lzma/bcj.h \ + lzma/block.h \ + lzma/check.h \ + lzma/container.h \ + lzma/delta.h \ + lzma/filter.h \ + lzma/hardware.h \ + lzma/index.h \ + lzma/index_hash.h \ + lzma/lzma12.h \ + lzma/stream_flags.h \ + lzma/version.h \ + lzma/vli.h + +all: all-am + +.SUFFIXES: +$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps) + @for dep in $?; do \ + case '$(am__configure_deps)' in \ + *$$dep*) \ + ( cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh ) \ + && { if test -f $@; then exit 0; else break; fi; }; \ + exit 1;; \ + esac; \ + done; \ + echo ' cd $(top_srcdir) && $(AUTOMAKE) --foreign src/liblzma/api/Makefile'; \ + $(am__cd) $(top_srcdir) && \ + $(AUTOMAKE) --foreign src/liblzma/api/Makefile +Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status + @case '$?' in \ + *config.status*) \ + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh;; \ + *) \ + echo ' cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__maybe_remake_depfiles)'; \ + cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__maybe_remake_depfiles);; \ + esac; + +$(top_builddir)/config.status: $(top_srcdir)/configure $(CONFIG_STATUS_DEPENDENCIES) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh + +$(top_srcdir)/configure: $(am__configure_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(ACLOCAL_M4): $(am__aclocal_m4_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(am__aclocal_m4_deps): + +mostlyclean-libtool: + -rm -f *.lo + +clean-libtool: + -rm -rf .libs _libs +install-nobase_includeHEADERS: $(nobase_include_HEADERS) + @$(NORMAL_INSTALL) + @list='$(nobase_include_HEADERS)'; test -n "$(includedir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(includedir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(includedir)" || exit 1; \ + fi; \ + $(am__nobase_list) | while read dir files; do \ + xfiles=; for file in $$files; do \ + if test -f "$$file"; then xfiles="$$xfiles $$file"; \ + else xfiles="$$xfiles $(srcdir)/$$file"; fi; done; \ + test -z "$$xfiles" || { \ + test "x$$dir" = x. || { \ + echo " $(MKDIR_P) '$(DESTDIR)$(includedir)/$$dir'"; \ + $(MKDIR_P) "$(DESTDIR)$(includedir)/$$dir"; }; \ + echo " $(INSTALL_HEADER) $$xfiles '$(DESTDIR)$(includedir)/$$dir'"; \ + $(INSTALL_HEADER) $$xfiles "$(DESTDIR)$(includedir)/$$dir" || exit $$?; }; \ + done + +uninstall-nobase_includeHEADERS: + @$(NORMAL_UNINSTALL) + @list='$(nobase_include_HEADERS)'; test -n "$(includedir)" || list=; \ + $(am__nobase_strip_setup); files=`$(am__nobase_strip)`; \ + dir='$(DESTDIR)$(includedir)'; $(am__uninstall_files_from_dir) + +ID: $(am__tagged_files) + $(am__define_uniq_tagged_files); mkid -fID $$unique +tags: tags-am +TAGS: tags + +tags-am: $(TAGS_DEPENDENCIES) $(am__tagged_files) + set x; \ + here=`pwd`; \ + $(am__define_uniq_tagged_files); \ + shift; \ + if test -z "$(ETAGS_ARGS)$$*$$unique"; then :; else \ + test -n "$$unique" || unique=$$empty_fix; \ + if test $$# -gt 0; then \ + $(ETAGS) $(ETAGSFLAGS) $(AM_ETAGSFLAGS) $(ETAGS_ARGS) \ + "$$@" $$unique; \ + else \ + $(ETAGS) $(ETAGSFLAGS) $(AM_ETAGSFLAGS) $(ETAGS_ARGS) \ + $$unique; \ + fi; \ + fi +ctags: ctags-am + +CTAGS: ctags +ctags-am: $(TAGS_DEPENDENCIES) $(am__tagged_files) + $(am__define_uniq_tagged_files); \ + test -z "$(CTAGS_ARGS)$$unique" \ + || $(CTAGS) $(CTAGSFLAGS) $(AM_CTAGSFLAGS) $(CTAGS_ARGS) \ + $$unique + +GTAGS: + here=`$(am__cd) $(top_builddir) && pwd` \ + && $(am__cd) $(top_srcdir) \ + && gtags -i $(GTAGS_ARGS) "$$here" +cscopelist: cscopelist-am + +cscopelist-am: $(am__tagged_files) + list='$(am__tagged_files)'; \ + case "$(srcdir)" in \ + [\\/]* | ?:[\\/]*) sdir="$(srcdir)" ;; \ + *) sdir=$(subdir)/$(srcdir) ;; \ + esac; \ + for i in $$list; do \ + if test -f "$$i"; then \ + echo "$(subdir)/$$i"; \ + else \ + echo "$$sdir/$$i"; \ + fi; \ + done >> $(top_builddir)/cscope.files + +distclean-tags: + -rm -f TAGS ID GTAGS GRTAGS GSYMS GPATH tags +distdir: $(BUILT_SOURCES) + $(MAKE) $(AM_MAKEFLAGS) distdir-am + +distdir-am: $(DISTFILES) + @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + list='$(DISTFILES)'; \ + dist_files=`for file in $$list; do echo $$file; done | \ + sed -e "s|^$$srcdirstrip/||;t" \ + -e "s|^$$topsrcdirstrip/|$(top_builddir)/|;t"`; \ + case $$dist_files in \ + */*) $(MKDIR_P) `echo "$$dist_files" | \ + sed '/\//!d;s|^|$(distdir)/|;s,/[^/]*$$,,' | \ + sort -u` ;; \ + esac; \ + for file in $$dist_files; do \ + if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \ + if test -d $$d/$$file; then \ + dir=`echo "/$$file" | sed -e 's,/[^/]*$$,,'`; \ + if test -d "$(distdir)/$$file"; then \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \ + cp -fpR $(srcdir)/$$file "$(distdir)$$dir" || exit 1; \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + cp -fpR $$d/$$file "$(distdir)$$dir" || exit 1; \ + else \ + test -f "$(distdir)/$$file" \ + || cp -p $$d/$$file "$(distdir)/$$file" \ + || exit 1; \ + fi; \ + done +check-am: all-am +check: check-am +@COND_DOXYGEN_FALSE@all-local: +all-am: Makefile $(HEADERS) all-local +installdirs: + for dir in "$(DESTDIR)$(includedir)"; do \ + test -z "$$dir" || $(MKDIR_P) "$$dir"; \ + done +install: install-am +install-exec: install-exec-am +install-data: install-data-am +uninstall: uninstall-am + +install-am: all-am + @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am + +installcheck: installcheck-am +install-strip: + if test -z '$(STRIP)'; then \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + install; \ + else \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + "INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'" install; \ + fi +mostlyclean-generic: + +clean-generic: + +distclean-generic: + -$(am__rm_f) $(CONFIG_CLEAN_FILES) + -test . = "$(srcdir)" || $(am__rm_f) $(CONFIG_CLEAN_VPATH_FILES) + +maintainer-clean-generic: + @echo "This command is intended for maintainers to use" + @echo "it deletes files that may require special tools to rebuild." +@COND_DOXYGEN_FALSE@clean-local: +@COND_DOXYGEN_FALSE@install-data-local: +@COND_DOXYGEN_FALSE@uninstall-local: +clean: clean-am + +clean-am: clean-generic clean-libtool clean-local mostlyclean-am + +distclean: distclean-am + -rm -f Makefile +distclean-am: clean-am distclean-generic distclean-tags + +dvi: dvi-am + +dvi-am: + +html: html-am + +html-am: + +info: info-am + +info-am: + +install-data-am: install-data-local install-nobase_includeHEADERS + +install-dvi: install-dvi-am + +install-dvi-am: + +install-exec-am: + +install-html: install-html-am + +install-html-am: + +install-info: install-info-am + +install-info-am: + +install-man: + +install-pdf: install-pdf-am + +install-pdf-am: + +install-ps: install-ps-am + +install-ps-am: + +installcheck-am: + +maintainer-clean: maintainer-clean-am + -rm -f Makefile +maintainer-clean-am: distclean-am maintainer-clean-generic + +mostlyclean: mostlyclean-am + +mostlyclean-am: mostlyclean-generic mostlyclean-libtool + +pdf: pdf-am + +pdf-am: + +ps: ps-am + +ps-am: + +uninstall-am: uninstall-local uninstall-nobase_includeHEADERS + +.MAKE: install-am install-strip + +.PHONY: CTAGS GTAGS TAGS all all-am all-local check check-am clean \ + clean-generic clean-libtool clean-local cscopelist-am ctags \ + ctags-am distclean distclean-generic distclean-libtool \ + distclean-tags distdir dvi dvi-am html html-am info info-am \ + install install-am install-data install-data-am \ + install-data-local install-dvi install-dvi-am install-exec \ + install-exec-am install-html install-html-am install-info \ + install-info-am install-man install-nobase_includeHEADERS \ + install-pdf install-pdf-am install-ps install-ps-am \ + install-strip installcheck installcheck-am installdirs \ + maintainer-clean maintainer-clean-generic mostlyclean \ + mostlyclean-generic mostlyclean-libtool pdf pdf-am ps ps-am \ + tags tags-am uninstall uninstall-am uninstall-local \ + uninstall-nobase_includeHEADERS + +.PRECIOUS: Makefile + + +@COND_DOXYGEN_TRUE@$(top_builddir)/doc/api/index.html: $(top_srcdir)/doxygen/update-doxygen $(top_srcdir)/doxygen/Doxyfile $(nobase_include_HEADERS) +@COND_DOXYGEN_TRUE@ $(MKDIR_P) "$(top_builddir)/doc" +@COND_DOXYGEN_TRUE@ "$(top_srcdir)/doxygen/update-doxygen" api \ +@COND_DOXYGEN_TRUE@ "$(top_srcdir)" "$(top_builddir)/doc" + +@COND_DOXYGEN_TRUE@all-local: $(top_builddir)/doc/api/index.html + +@COND_DOXYGEN_TRUE@install-data-local: +@COND_DOXYGEN_TRUE@ $(MKDIR_P) "$(DESTDIR)$(docdir)/api" +@COND_DOXYGEN_TRUE@ $(INSTALL_DATA) "$(top_builddir)"/doc/api/* "$(DESTDIR)$(docdir)/api" + +@COND_DOXYGEN_TRUE@uninstall-local: +@COND_DOXYGEN_TRUE@ rm -rf "$(DESTDIR)$(docdir)/api" + +@COND_DOXYGEN_TRUE@clean-local: +@COND_DOXYGEN_TRUE@ rm -rf "$(top_builddir)/doc/api" + +# Tell versions [3.59,3.63) of GNU make to not export all variables. +# Otherwise a system limit (for SysV at least) may be exceeded. +.NOEXPORT: + +# Tell GNU make to disable its built-in pattern rules. +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% diff --git a/src/liblzma/api/lzma.h b/src/liblzma/api/lzma.h new file mode 100644 index 0000000000..6ca6e503d8 --- /dev/null +++ b/src/liblzma/api/lzma.h @@ -0,0 +1,327 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file api/lzma.h + * \brief The public API of liblzma data compression library + * \mainpage + * + * liblzma is a general-purpose data compression library with a zlib-like API. + * The native file format is .xz, but also the old .lzma format and raw (no + * headers) streams are supported. Multiple compression algorithms (filters) + * are supported. Currently LZMA2 is the primary filter. + * + * liblzma is part of XZ Utils . XZ Utils + * includes a gzip-like command line tool named xz and some other tools. + * XZ Utils is developed and maintained by Lasse Collin. + * + * Major parts of liblzma are based on code written by Igor Pavlov, + * specifically the LZMA SDK . + * + * The SHA-256 implementation in liblzma is based on code written by + * Wei Dai in Crypto++ Library . + * + * liblzma is distributed under the BSD Zero Clause License (0BSD). + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H +#define LZMA_H + +/***************************** + * Required standard headers * + *****************************/ + +/* + * liblzma API headers need some standard types and macros. To allow + * including lzma.h without requiring the application to include other + * headers first, lzma.h includes the required standard headers unless + * they already seem to be included already or if LZMA_MANUAL_HEADERS + * has been defined. + * + * Here's what types and macros are needed and from which headers: + * - stddef.h: size_t, NULL + * - stdint.h: uint8_t, uint32_t, uint64_t, UINT32_C(n), uint64_C(n), + * UINT32_MAX, UINT64_MAX + * + * However, inttypes.h is a little more portable than stdint.h, although + * inttypes.h declares some unneeded things compared to plain stdint.h. + * + * The hacks below aren't perfect, specifically they assume that inttypes.h + * exists and that it typedefs at least uint8_t, uint32_t, and uint64_t, + * and that, in case of incomplete inttypes.h, unsigned int is 32-bit. + * If the application already takes care of setting up all the types and + * macros properly (for example by using gnulib's stdint.h or inttypes.h), + * we try to detect that the macros are already defined and don't include + * inttypes.h here again. However, you may define LZMA_MANUAL_HEADERS to + * force this file to never include any system headers. + * + * Some could argue that liblzma API should provide all the required types, + * for example lzma_uint64, LZMA_UINT64_C(n), and LZMA_UINT64_MAX. This was + * seen as an unnecessary mess, since most systems already provide all the + * necessary types and macros in the standard headers. + * + * Note that liblzma API still has lzma_bool, because using stdbool.h would + * break C89 and C++ programs on many systems. sizeof(bool) in C99 isn't + * necessarily the same as sizeof(bool) in C++. + */ + +#ifndef LZMA_MANUAL_HEADERS + /* + * I suppose this works portably also in C++. Note that in C++, + * we need to get size_t into the global namespace. + */ +# include + + /* + * Skip inttypes.h if we already have all the required macros. If we + * have the macros, we assume that we have the matching typedefs too. + */ +# if !defined(UINT32_C) || !defined(UINT64_C) \ + || !defined(UINT32_MAX) || !defined(UINT64_MAX) + /* + * MSVC versions older than 2013 have no C99 support, and + * thus they cannot be used to compile liblzma. Using an + * existing liblzma.dll with old MSVC can work though(*), + * but we need to define the required standard integer + * types here in a MSVC-specific way. + * + * (*) If you do this, the existing liblzma.dll probably uses + * a different runtime library than your MSVC-built + * application. Mixing runtimes is generally bad, but + * in this case it should work as long as you avoid + * the few rarely-needed liblzma functions that allocate + * memory and expect the caller to free it using free(). + */ +# if defined(_WIN32) && defined(_MSC_VER) && _MSC_VER < 1800 + typedef unsigned __int8 uint8_t; + typedef unsigned __int32 uint32_t; + typedef unsigned __int64 uint64_t; +# else + /* Use the standard inttypes.h. */ +# ifdef __cplusplus + /* + * C99 sections 7.18.2 and 7.18.4 specify + * that C++ implementations define the limit + * and constant macros only if specifically + * requested. Note that if you want the + * format macros (PRIu64 etc.) too, you need + * to define __STDC_FORMAT_MACROS before + * including lzma.h, since re-including + * inttypes.h with __STDC_FORMAT_MACROS + * defined doesn't necessarily work. + */ +# ifndef __STDC_LIMIT_MACROS +# define __STDC_LIMIT_MACROS 1 +# endif +# ifndef __STDC_CONSTANT_MACROS +# define __STDC_CONSTANT_MACROS 1 +# endif +# endif + +# include +# endif + + /* + * Some old systems have only the typedefs in inttypes.h, and + * lack all the macros. For those systems, we need a few more + * hacks. We assume that unsigned int is 32-bit and unsigned + * long is either 32-bit or 64-bit. If these hacks aren't + * enough, the application has to setup the types manually + * before including lzma.h. + */ +# ifndef UINT32_C +# if defined(_WIN32) && defined(_MSC_VER) +# define UINT32_C(n) n ## UI32 +# else +# define UINT32_C(n) n ## U +# endif +# endif + +# ifndef UINT64_C +# if defined(_WIN32) && defined(_MSC_VER) +# define UINT64_C(n) n ## UI64 +# else + /* Get ULONG_MAX. */ +# include +# if ULONG_MAX == 4294967295UL +# define UINT64_C(n) n ## ULL +# else +# define UINT64_C(n) n ## UL +# endif +# endif +# endif + +# ifndef UINT32_MAX +# define UINT32_MAX (UINT32_C(4294967295)) +# endif + +# ifndef UINT64_MAX +# define UINT64_MAX (UINT64_C(18446744073709551615)) +# endif +# endif +#endif /* ifdef LZMA_MANUAL_HEADERS */ + + +/****************** + * LZMA_API macro * + ******************/ + +/* + * Some systems require that the functions and function pointers are + * declared specially in the headers. LZMA_API_IMPORT is for importing + * symbols and LZMA_API_CALL is to specify the calling convention. + * + * By default it is assumed that the application will link dynamically + * against liblzma. #define LZMA_API_STATIC in your application if you + * want to link against static liblzma. If you don't care about portability + * to operating systems like Windows, or at least don't care about linking + * against static liblzma on them, don't worry about LZMA_API_STATIC. That + * is, most developers will never need to use LZMA_API_STATIC. + * + * The GCC variants are a special case on Windows (Cygwin and MinGW-w64). + * We rely on GCC doing the right thing with its auto-import feature, + * and thus don't use __declspec(dllimport). This way developers don't + * need to worry about LZMA_API_STATIC. Also the calling convention is + * omitted on Cygwin but not on MinGW-w64. + */ +#ifndef LZMA_API_IMPORT +# if !defined(LZMA_API_STATIC) && defined(_WIN32) && !defined(__GNUC__) +# define LZMA_API_IMPORT __declspec(dllimport) +# else +# define LZMA_API_IMPORT +# endif +#endif + +#ifndef LZMA_API_CALL +# if defined(_WIN32) && !defined(__CYGWIN__) +# define LZMA_API_CALL __cdecl +# else +# define LZMA_API_CALL +# endif +#endif + +#ifndef LZMA_API +# define LZMA_API(type) LZMA_API_IMPORT type LZMA_API_CALL +#endif + + +/*********** + * nothrow * + ***********/ + +/* + * None of the functions in liblzma may throw an exception. Even + * the functions that use callback functions won't throw exceptions, + * because liblzma would break if a callback function threw an exception. + */ +#ifndef lzma_nothrow +# if defined(__cplusplus) +# if __cplusplus >= 201103L || (defined(_MSVC_LANG) \ + && _MSVC_LANG >= 201103L) +# define lzma_nothrow noexcept +# else +# define lzma_nothrow throw() +# endif +# elif defined(__GNUC__) && (__GNUC__ > 3 \ + || (__GNUC__ == 3 && __GNUC_MINOR__ >= 3)) +# define lzma_nothrow __attribute__((__nothrow__)) +# else +# define lzma_nothrow +# endif +#endif + + +/******************** + * GNU C extensions * + ********************/ + +/* + * GNU C extensions are used conditionally in the public API. It doesn't + * break anything if these are sometimes enabled and sometimes not, only + * affects warnings and optimizations. + */ +#if defined(__GNUC__) && __GNUC__ >= 3 +# ifndef lzma_attribute +# define lzma_attribute(attr) __attribute__(attr) +# endif + + /* warn_unused_result was added in GCC 3.4. */ +# ifndef lzma_attr_warn_unused_result +# if __GNUC__ == 3 && __GNUC_MINOR__ < 4 +# define lzma_attr_warn_unused_result +# endif +# endif + +#else +# ifndef lzma_attribute +# define lzma_attribute(attr) +# endif +#endif + + +#ifndef lzma_attr_pure +# define lzma_attr_pure lzma_attribute((__pure__)) +#endif + +#ifndef lzma_attr_const +# define lzma_attr_const lzma_attribute((__const__)) +#endif + +#ifndef lzma_attr_warn_unused_result +# define lzma_attr_warn_unused_result \ + lzma_attribute((__warn_unused_result__)) +#endif + + +/************** + * Subheaders * + **************/ + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * Subheaders check that this is defined. It is to prevent including + * them directly from applications. + */ +#define LZMA_H_INTERNAL 1 + +/* Basic features */ +#include "lzma/version.h" +#include "lzma/base.h" +#include "lzma/vli.h" +#include "lzma/check.h" + +/* Filters */ +#include "lzma/filter.h" +#include "lzma/bcj.h" +#include "lzma/delta.h" +#include "lzma/lzma12.h" + +/* Container formats */ +#include "lzma/container.h" + +/* Advanced features */ +#include "lzma/stream_flags.h" +#include "lzma/block.h" +#include "lzma/index.h" +#include "lzma/index_hash.h" + +/* Hardware information */ +#include "lzma/hardware.h" + +/* + * All subheaders included. Undefine LZMA_H_INTERNAL to prevent applications + * re-including the subheaders. + */ +#undef LZMA_H_INTERNAL + +#ifdef __cplusplus +} +#endif + +#endif /* ifndef LZMA_H */ diff --git a/src/liblzma/api/lzma/base.h b/src/liblzma/api/lzma/base.h new file mode 100644 index 0000000000..590e1d22bb --- /dev/null +++ b/src/liblzma/api/lzma/base.h @@ -0,0 +1,747 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/base.h + * \brief Data types and functions used in many places in liblzma API + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Boolean + * + * This is here because C89 doesn't have stdbool.h. To set a value for + * variables having type lzma_bool, you can use + * - C99's 'true' and 'false' from stdbool.h; + * - C++'s internal 'true' and 'false'; or + * - integers one (true) and zero (false). + */ +typedef unsigned char lzma_bool; + + +/** + * \brief Type of reserved enumeration variable in structures + * + * To avoid breaking library ABI when new features are added, several + * structures contain extra variables that may be used in future. Since + * sizeof(enum) can be different than sizeof(int), and sizeof(enum) may + * even vary depending on the range of enumeration constants, we specify + * a separate type to be used for reserved enumeration variables. All + * enumeration constants in liblzma API will be non-negative and less + * than 128, which should guarantee that the ABI won't break even when + * new constants are added to existing enumerations. + */ +typedef enum { + LZMA_RESERVED_ENUM = 0 +} lzma_reserved_enum; + + +/** + * \brief Return values used by several functions in liblzma + * + * Check the descriptions of specific functions to find out which return + * values they can return. With some functions the return values may have + * more specific meanings than described here; those differences are + * described per-function basis. + */ +typedef enum { + LZMA_OK = 0, + /**< + * \brief Operation completed successfully + */ + + LZMA_STREAM_END = 1, + /**< + * \brief End of stream was reached + * + * In encoder, LZMA_SYNC_FLUSH, LZMA_FULL_FLUSH, or + * LZMA_FINISH was finished. In decoder, this indicates + * that all the data was successfully decoded. + * + * In all cases, when LZMA_STREAM_END is returned, the last + * output bytes should be picked from strm->next_out. + */ + + LZMA_NO_CHECK = 2, + /**< + * \brief Input stream has no integrity check + * + * This return value can be returned only if the + * LZMA_TELL_NO_CHECK flag was used when initializing + * the decoder. LZMA_NO_CHECK is just a warning, and + * the decoding can be continued normally. + * + * It is possible to call lzma_get_check() immediately after + * lzma_code has returned LZMA_NO_CHECK. The result will + * naturally be LZMA_CHECK_NONE, but the possibility to call + * lzma_get_check() may be convenient in some applications. + */ + + LZMA_UNSUPPORTED_CHECK = 3, + /**< + * \brief Cannot calculate the integrity check + * + * The usage of this return value is different in encoders + * and decoders. + * + * Encoders can return this value only from the initialization + * function. If initialization fails with this value, the + * encoding cannot be done, because there's no way to produce + * output with the correct integrity check. + * + * Decoders can return this value only from lzma_code() and + * only if the LZMA_TELL_UNSUPPORTED_CHECK flag was used when + * initializing the decoder. The decoding can still be + * continued normally even if the check type is unsupported, + * but naturally the check will not be validated, and possible + * errors may go undetected. + * + * With decoder, it is possible to call lzma_get_check() + * immediately after lzma_code() has returned + * LZMA_UNSUPPORTED_CHECK. This way it is possible to find + * out what the unsupported Check ID was. + */ + + LZMA_GET_CHECK = 4, + /**< + * \brief Integrity check type is now available + * + * This value can be returned only by the lzma_code() function + * and only if the decoder was initialized with the + * LZMA_TELL_ANY_CHECK flag. LZMA_GET_CHECK tells the + * application that it may now call lzma_get_check() to find + * out the Check ID. This can be used, for example, to + * implement a decoder that accepts only files that have + * strong enough integrity check. + */ + + LZMA_MEM_ERROR = 5, + /**< + * \brief Cannot allocate memory + * + * Memory allocation failed, or the size of the allocation + * would be greater than SIZE_MAX. + * + * Due to internal implementation reasons, the coding cannot + * be continued even if more memory were made available after + * LZMA_MEM_ERROR. + */ + + LZMA_MEMLIMIT_ERROR = 6, + /**< + * \brief Memory usage limit was reached + * + * Decoder would need more memory than allowed by the + * specified memory usage limit. To continue decoding, + * the memory usage limit has to be increased with + * lzma_memlimit_set(). + * + * liblzma 5.2.6 and earlier had a bug in single-threaded .xz + * decoder (lzma_stream_decoder()) which made it impossible + * to continue decoding after LZMA_MEMLIMIT_ERROR even if + * the limit was increased using lzma_memlimit_set(). + * Other decoders worked correctly. + */ + + LZMA_FORMAT_ERROR = 7, + /**< + * \brief File format not recognized + * + * The decoder did not recognize the input as supported file + * format. This error can occur, for example, when trying to + * decode .lzma format file with lzma_stream_decoder, + * because lzma_stream_decoder accepts only the .xz format. + */ + + LZMA_OPTIONS_ERROR = 8, + /**< + * \brief Invalid or unsupported options + * + * Invalid or unsupported options, for example + * - unsupported filter(s) or filter options; or + * - reserved bits set in headers (decoder only). + * + * Rebuilding liblzma with more features enabled, or + * upgrading to a newer version of liblzma may help. + */ + + LZMA_DATA_ERROR = 9, + /**< + * \brief Data is corrupt + * + * The usage of this return value is different in encoders + * and decoders. In both encoder and decoder, the coding + * cannot continue after this error. + * + * Encoders return this if size limits of the target file + * format would be exceeded. These limits are huge, thus + * getting this error from an encoder is mostly theoretical. + * For example, the maximum compressed and uncompressed + * size of a .xz Stream is roughly 8 EiB (2^63 bytes). + * + * Decoders return this error if the input data is corrupt. + * This can mean, for example, invalid CRC32 in headers + * or invalid check of uncompressed data. + */ + + LZMA_BUF_ERROR = 10, + /**< + * \brief No progress is possible + * + * This error code is returned when the coder cannot consume + * any new input and produce any new output. The most common + * reason for this error is that the input stream being + * decoded is truncated or corrupt. + * + * This error is not fatal. Coding can be continued normally + * by providing more input and/or more output space, if + * possible. + * + * Typically the first call to lzma_code() that can do no + * progress returns LZMA_OK instead of LZMA_BUF_ERROR. Only + * the second consecutive call doing no progress will return + * LZMA_BUF_ERROR. This is intentional. + * + * With zlib, Z_BUF_ERROR may be returned even if the + * application is doing nothing wrong, so apps will need + * to handle Z_BUF_ERROR specially. The above hack + * guarantees that liblzma never returns LZMA_BUF_ERROR + * to properly written applications unless the input file + * is truncated or corrupt. This should simplify the + * applications a little. + */ + + LZMA_PROG_ERROR = 11, + /**< + * \brief Programming error + * + * This indicates that the arguments given to the function are + * invalid or the internal state of the decoder is corrupt. + * - Function arguments are invalid or the structures + * pointed by the argument pointers are invalid + * e.g. if strm->next_out has been set to NULL and + * strm->avail_out > 0 when calling lzma_code(). + * - lzma_* functions have been called in wrong order + * e.g. lzma_code() was called right after lzma_end(). + * - If errors occur randomly, the reason might be flaky + * hardware. + * + * If you think that your code is correct, this error code + * can be a sign of a bug in liblzma. See the documentation + * how to report bugs. + */ + + LZMA_SEEK_NEEDED = 12, + /**< + * \brief Request to change the input file position + * + * Some coders can do random access in the input file. The + * initialization functions of these coders take the file size + * as an argument. No other coders can return LZMA_SEEK_NEEDED. + * + * When this value is returned, the application must seek to + * the file position given in lzma_stream.seek_pos. This value + * is guaranteed to never exceed the file size that was + * specified at the coder initialization. + * + * After seeking the application should read new input and + * pass it normally via lzma_stream.next_in and .avail_in. + */ + + /* + * These enumerations may be used internally by liblzma + * but they will never be returned to applications. + */ + LZMA_RET_INTERNAL1 = 101, + LZMA_RET_INTERNAL2 = 102, + LZMA_RET_INTERNAL3 = 103, + LZMA_RET_INTERNAL4 = 104, + LZMA_RET_INTERNAL5 = 105, + LZMA_RET_INTERNAL6 = 106, + LZMA_RET_INTERNAL7 = 107, + LZMA_RET_INTERNAL8 = 108 +} lzma_ret; + + +/** + * \brief The 'action' argument for lzma_code() + * + * After the first use of LZMA_SYNC_FLUSH, LZMA_FULL_FLUSH, LZMA_FULL_BARRIER, + * or LZMA_FINISH, the same 'action' must be used until lzma_code() returns + * LZMA_STREAM_END. Also, the amount of input (that is, strm->avail_in) must + * not be modified by the application until lzma_code() returns + * LZMA_STREAM_END. Changing the 'action' or modifying the amount of input + * will make lzma_code() return LZMA_PROG_ERROR. + */ +typedef enum { + LZMA_RUN = 0, + /**< + * \brief Continue coding + * + * Encoder: Encode as much input as possible. Some internal + * buffering will probably be done (depends on the filter + * chain in use), which causes latency: the input used won't + * usually be decodeable from the output of the same + * lzma_code() call. + * + * Decoder: Decode as much input as possible and produce as + * much output as possible. + */ + + LZMA_SYNC_FLUSH = 1, + /**< + * \brief Make all the input available at output + * + * Normally the encoder introduces some latency. + * LZMA_SYNC_FLUSH forces all the buffered data to be + * available at output without resetting the internal + * state of the encoder. This way it is possible to use + * compressed stream for example for communication over + * network. + * + * Only some filters support LZMA_SYNC_FLUSH. Trying to use + * LZMA_SYNC_FLUSH with filters that don't support it will + * make lzma_code() return LZMA_OPTIONS_ERROR. For example, + * LZMA1 doesn't support LZMA_SYNC_FLUSH but LZMA2 does. + * + * Using LZMA_SYNC_FLUSH very often can dramatically reduce + * the compression ratio. With some filters (for example, + * LZMA2), fine-tuning the compression options may help + * mitigate this problem significantly (for example, + * match finder with LZMA2). + * + * Decoders don't support LZMA_SYNC_FLUSH. + */ + + LZMA_FULL_FLUSH = 2, + /**< + * \brief Finish encoding of the current Block + * + * All the input data going to the current Block must have + * been given to the encoder (the last bytes can still be + * pending in *next_in). Call lzma_code() with LZMA_FULL_FLUSH + * until it returns LZMA_STREAM_END. Then continue normally + * with LZMA_RUN or finish the Stream with LZMA_FINISH. + * + * This action is currently supported only by Stream encoder + * and easy encoder (which uses Stream encoder). If there is + * no unfinished Block, no empty Block is created. + */ + + LZMA_FULL_BARRIER = 4, + /**< + * \brief Finish encoding of the current Block + * + * This is like LZMA_FULL_FLUSH except that this doesn't + * necessarily wait until all the input has been made + * available via the output buffer. That is, lzma_code() + * might return LZMA_STREAM_END as soon as all the input + * has been consumed (avail_in == 0). + * + * LZMA_FULL_BARRIER is useful with a threaded encoder if + * one wants to split the .xz Stream into Blocks at specific + * offsets but doesn't care if the output isn't flushed + * immediately. Using LZMA_FULL_BARRIER allows keeping + * the threads busy while LZMA_FULL_FLUSH would make + * lzma_code() wait until all the threads have finished + * until more data could be passed to the encoder. + * + * With a lzma_stream initialized with the single-threaded + * lzma_stream_encoder() or lzma_easy_encoder(), + * LZMA_FULL_BARRIER is an alias for LZMA_FULL_FLUSH. + */ + + LZMA_FINISH = 3 + /**< + * \brief Finish the coding operation + * + * All the input data must have been given to the encoder + * (the last bytes can still be pending in next_in). + * Call lzma_code() with LZMA_FINISH until it returns + * LZMA_STREAM_END. Once LZMA_FINISH has been used, + * the amount of input must no longer be changed by + * the application. + * + * When decoding, using LZMA_FINISH is optional unless the + * LZMA_CONCATENATED flag was used when the decoder was + * initialized. When LZMA_CONCATENATED was not used, the only + * effect of LZMA_FINISH is that the amount of input must not + * be changed just like in the encoder. + */ +} lzma_action; + + +/** + * \brief Custom functions for memory handling + * + * A pointer to lzma_allocator may be passed via lzma_stream structure + * to liblzma, and some advanced functions take a pointer to lzma_allocator + * as a separate function argument. The library will use the functions + * specified in lzma_allocator for memory handling instead of the default + * malloc() and free(). C++ users should note that the custom memory + * handling functions must not throw exceptions. + * + * Single-threaded mode only: liblzma doesn't make an internal copy of + * lzma_allocator. Thus, it is OK to change these function pointers in + * the middle of the coding process, but obviously it must be done + * carefully to make sure that the replacement 'free' can deallocate + * memory allocated by the earlier 'alloc' function(s). + * + * Multithreaded mode: liblzma might internally store pointers to the + * lzma_allocator given via the lzma_stream structure. The application + * must not change the allocator pointer in lzma_stream or the contents + * of the pointed lzma_allocator structure until lzma_end() has been used + * to free the memory associated with that lzma_stream. The allocation + * functions might be called simultaneously from multiple threads, and + * thus they must be thread safe. + */ +typedef struct { + /** + * \brief Pointer to a custom memory allocation function + * + * If you don't want a custom allocator, but still want + * custom free(), set this to NULL and liblzma will use + * the standard malloc(). + * + * \param opaque lzma_allocator.opaque (see below) + * \param nmemb Number of elements like in calloc(). liblzma + * will always set nmemb to 1, so it is safe to + * ignore nmemb in a custom allocator if you like. + * The nmemb argument exists only for + * compatibility with zlib and libbzip2. + * \param size Size of an element in bytes. + * liblzma never sets this to zero. + * + * \return Pointer to the beginning of a memory block of + * 'size' bytes, or NULL if allocation fails + * for some reason. When allocation fails, functions + * of liblzma return LZMA_MEM_ERROR. + * + * The allocator should not waste time zeroing the allocated buffers. + * This is not only about speed, but also memory usage, since the + * operating system kernel doesn't necessarily allocate the requested + * memory in physical memory until it is actually used. With small + * input files, liblzma may actually need only a fraction of the + * memory that it requested for allocation. + * + * \note LZMA_MEM_ERROR is also used when the size of the + * allocation would be greater than SIZE_MAX. Thus, + * don't assume that the custom allocator must have + * returned NULL if some function from liblzma + * returns LZMA_MEM_ERROR. + */ + void *(LZMA_API_CALL *alloc)(void *opaque, size_t nmemb, size_t size); + + /** + * \brief Pointer to a custom memory freeing function + * + * If you don't want a custom freeing function, but still + * want a custom allocator, set this to NULL and liblzma + * will use the standard free(). + * + * \param opaque lzma_allocator.opaque (see below) + * \param ptr Pointer returned by lzma_allocator.alloc(), + * or when it is set to NULL, a pointer returned + * by the standard malloc(). + */ + void (LZMA_API_CALL *free)(void *opaque, void *ptr); + + /** + * \brief Pointer passed to .alloc() and .free() + * + * opaque is passed as the first argument to lzma_allocator.alloc() + * and lzma_allocator.free(). This intended to ease implementing + * custom memory allocation functions for use with liblzma. + * + * If you don't need this, you should set this to NULL. + */ + void *opaque; + +} lzma_allocator; + + +/** + * \brief Internal data structure + * + * The contents of this structure is not visible outside the library. + */ +typedef struct lzma_internal_s lzma_internal; + + +/** + * \brief Passing data to and from liblzma + * + * The lzma_stream structure is used for + * - passing pointers to input and output buffers to liblzma; + * - defining custom memory handler functions; and + * - holding a pointer to coder-specific internal data structures. + * + * Typical usage: + * + * - After allocating lzma_stream (on stack or with malloc()), it must be + * initialized to LZMA_STREAM_INIT (see LZMA_STREAM_INIT for details). + * + * - Initialize a coder to the lzma_stream, for example by using + * lzma_easy_encoder() or lzma_auto_decoder(). Some notes: + * - In contrast to zlib, strm->next_in and strm->next_out are + * ignored by all initialization functions, thus it is safe + * to not initialize them yet. + * - The initialization functions always set strm->total_in and + * strm->total_out to zero. + * - If the initialization function fails, no memory is left allocated + * that would require freeing with lzma_end() even if some memory was + * associated with the lzma_stream structure when the initialization + * function was called. + * + * - Use lzma_code() to do the actual work. + * + * - Once the coding has been finished, the existing lzma_stream can be + * reused. It is OK to reuse lzma_stream with different initialization + * function without calling lzma_end() first. Old allocations are + * automatically freed. + * + * - Finally, use lzma_end() to free the allocated memory. lzma_end() never + * frees the lzma_stream structure itself. + * + * Application may modify the values of total_in and total_out as it wants. + * They are updated by liblzma to match the amount of data read and + * written but aren't used for anything else except as a possible return + * values from lzma_get_progress(). + */ +typedef struct { + const uint8_t *next_in; /**< Pointer to the next input byte. */ + size_t avail_in; /**< Number of available input bytes in next_in. */ + uint64_t total_in; /**< Total number of bytes read by liblzma. */ + + uint8_t *next_out; /**< Pointer to the next output position. */ + size_t avail_out; /**< Amount of free space in next_out. */ + uint64_t total_out; /**< Total number of bytes written by liblzma. */ + + /** + * \brief Custom memory allocation functions + * + * In most cases this is NULL which makes liblzma use + * the standard malloc() and free(). + * + * \note In 5.0.x this is not a const pointer. + */ + const lzma_allocator *allocator; + + /** Internal state is not visible to applications. */ + lzma_internal *internal; + + /* + * Reserved space to allow possible future extensions without + * breaking the ABI. Excluding the initialization of this structure, + * you should not touch these, because the names of these variables + * may change. + */ + + /** \private Reserved member. */ + void *reserved_ptr1; + + /** \private Reserved member. */ + void *reserved_ptr2; + + /** \private Reserved member. */ + void *reserved_ptr3; + + /** \private Reserved member. */ + void *reserved_ptr4; + + /** + * \brief New seek input position for LZMA_SEEK_NEEDED + * + * When lzma_code() returns LZMA_SEEK_NEEDED, the new input position + * needed by liblzma will be available seek_pos. The value is + * guaranteed to not exceed the file size that was specified when + * this lzma_stream was initialized. + * + * In all other situations the value of this variable is undefined. + */ + uint64_t seek_pos; + + /** \private Reserved member. */ + uint64_t reserved_int2; + + /** \private Reserved member. */ + size_t reserved_int3; + + /** \private Reserved member. */ + size_t reserved_int4; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum1; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum2; + +} lzma_stream; + + +/** + * \brief Initialization for lzma_stream + * + * When you declare an instance of lzma_stream, you can immediately + * initialize it so that initialization functions know that no memory + * has been allocated yet: + * + * lzma_stream strm = LZMA_STREAM_INIT; + * + * If you need to initialize a dynamically allocated lzma_stream, you can use + * memset(strm_pointer, 0, sizeof(lzma_stream)). Strictly speaking, this + * violates the C standard since NULL may have different internal + * representation than zero, but it should be portable enough in practice. + * Anyway, for maximum portability, you can use something like this: + * + * lzma_stream tmp = LZMA_STREAM_INIT; + * *strm = tmp; + */ +#define LZMA_STREAM_INIT \ + { NULL, 0, 0, NULL, 0, 0, NULL, NULL, \ + NULL, NULL, NULL, NULL, 0, 0, 0, 0, \ + LZMA_RESERVED_ENUM, LZMA_RESERVED_ENUM } + + +/** + * \brief Encode or decode data + * + * Once the lzma_stream has been successfully initialized (e.g. with + * lzma_stream_encoder()), the actual encoding or decoding is done + * using this function. The application has to update strm->next_in, + * strm->avail_in, strm->next_out, and strm->avail_out to pass input + * to and get output from liblzma. + * + * See the description of the coder-specific initialization function to find + * out what 'action' values are supported by the coder. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param action Action for this function to take. Must be a valid + * lzma_action enum value. + * + * \return Any valid lzma_ret. See the lzma_ret enum description for more + * information. + */ +extern LZMA_API(lzma_ret) lzma_code(lzma_stream *strm, lzma_action action) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Free memory allocated for the coder data structures + * + * After lzma_end(strm), strm->internal is guaranteed to be NULL. No other + * members of the lzma_stream structure are touched. + * + * \note zlib indicates an error if application end()s unfinished + * stream structure. liblzma doesn't do this, and assumes that + * application knows what it is doing. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + */ +extern LZMA_API(void) lzma_end(lzma_stream *strm) lzma_nothrow; + + +/** + * \brief Get progress information + * + * In single-threaded mode, applications can get progress information from + * strm->total_in and strm->total_out. In multi-threaded mode this is less + * useful because a significant amount of both input and output data gets + * buffered internally by liblzma. This makes total_in and total_out give + * misleading information and also makes the progress indicator updates + * non-smooth. + * + * This function gives realistic progress information also in multi-threaded + * mode by taking into account the progress made by each thread. In + * single-threaded mode *progress_in and *progress_out are set to + * strm->total_in and strm->total_out, respectively. + * + * \param strm Pointer to lzma_stream that is at least + * initialized with LZMA_STREAM_INIT. + * \param[out] progress_in Pointer to the number of input bytes processed. + * \param[out] progress_out Pointer to the number of output bytes processed. + */ +extern LZMA_API(void) lzma_get_progress(lzma_stream *strm, + uint64_t *progress_in, uint64_t *progress_out) lzma_nothrow; + + +/** + * \brief Get the memory usage of decoder filter chain + * + * This function is currently supported only when *strm has been initialized + * with a function that takes a memlimit argument. With other functions, you + * should use e.g. lzma_raw_encoder_memusage() or lzma_raw_decoder_memusage() + * to estimate the memory requirements. + * + * This function is useful e.g. after LZMA_MEMLIMIT_ERROR to find out how big + * the memory usage limit should have been to decode the input. Note that + * this may give misleading information if decoding .xz Streams that have + * multiple Blocks, because each Block can have different memory requirements. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * + * \return How much memory is currently allocated for the filter + * decoders. If no filter chain is currently allocated, + * some non-zero value is still returned, which is less than + * or equal to what any filter chain would indicate as its + * memory requirement. + * + * If this function isn't supported by *strm or some other error + * occurs, zero is returned. + */ +extern LZMA_API(uint64_t) lzma_memusage(const lzma_stream *strm) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the current memory usage limit + * + * This function is supported only when *strm has been initialized with + * a function that takes a memlimit argument. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * + * \return On success, the current memory usage limit is returned + * (always non-zero). On error, zero is returned. + */ +extern LZMA_API(uint64_t) lzma_memlimit_get(const lzma_stream *strm) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Set the memory usage limit + * + * This function is supported only when *strm has been initialized with + * a function that takes a memlimit argument. + * + * liblzma 5.2.3 and earlier has a bug where memlimit value of 0 causes + * this function to do nothing (leaving the limit unchanged) and still + * return LZMA_OK. Later versions treat 0 as if 1 had been specified (so + * lzma_memlimit_get() will return 1 even if you specify 0 here). + * + * liblzma 5.2.6 and earlier had a bug in single-threaded .xz decoder + * (lzma_stream_decoder()) which made it impossible to continue decoding + * after LZMA_MEMLIMIT_ERROR even if the limit was increased using + * lzma_memlimit_set(). Other decoders worked correctly. + * + * \return Possible lzma_ret values: + * - LZMA_OK: New memory usage limit successfully set. + * - LZMA_MEMLIMIT_ERROR: The new limit is too small. + * The limit was not changed. + * - LZMA_PROG_ERROR: Invalid arguments, e.g. *strm doesn't + * support memory usage limit. + */ +extern LZMA_API(lzma_ret) lzma_memlimit_set( + lzma_stream *strm, uint64_t memlimit) lzma_nothrow; diff --git a/src/liblzma/api/lzma/bcj.h b/src/liblzma/api/lzma/bcj.h new file mode 100644 index 0000000000..fb737cbba4 --- /dev/null +++ b/src/liblzma/api/lzma/bcj.h @@ -0,0 +1,195 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/bcj.h + * \brief Branch/Call/Jump conversion filters + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/* Filter IDs for lzma_filter.id */ + +/** + * \brief Filter for x86 binaries + */ +#define LZMA_FILTER_X86 LZMA_VLI_C(0x04) + +/** + * \brief Filter for Big endian PowerPC binaries + */ +#define LZMA_FILTER_POWERPC LZMA_VLI_C(0x05) + +/** + * \brief Filter for IA-64 (Itanium) binaries + */ +#define LZMA_FILTER_IA64 LZMA_VLI_C(0x06) + +/** + * \brief Filter for ARM binaries + */ +#define LZMA_FILTER_ARM LZMA_VLI_C(0x07) + +/** + * \brief Filter for ARM-Thumb binaries + */ +#define LZMA_FILTER_ARMTHUMB LZMA_VLI_C(0x08) + +/** + * \brief Filter for SPARC binaries + */ +#define LZMA_FILTER_SPARC LZMA_VLI_C(0x09) + +/** + * \brief Filter for ARM64 binaries + */ +#define LZMA_FILTER_ARM64 LZMA_VLI_C(0x0A) + +/** + * \brief Filter for RISC-V binaries + */ +#define LZMA_FILTER_RISCV LZMA_VLI_C(0x0B) + + +/** + * \brief Options for BCJ filters + * + * The BCJ filters never change the size of the data. Specifying options + * for them is optional: if pointer to options is NULL, default value is + * used. You probably never need to specify options to BCJ filters, so just + * set the options pointer to NULL and be happy. + * + * If options with non-default values have been specified when encoding, + * the same options must also be specified when decoding. + * + * \note At the moment, none of the BCJ filters support + * LZMA_SYNC_FLUSH. If LZMA_SYNC_FLUSH is specified, + * LZMA_OPTIONS_ERROR will be returned. If there is need, + * partial support for LZMA_SYNC_FLUSH can be added in future. + * Partial means that flushing would be possible only at + * offsets that are multiple of 2, 4, or 16 depending on + * the filter, except x86 which cannot be made to support + * LZMA_SYNC_FLUSH predictably. + */ +typedef struct { + /** + * \brief Start offset for conversions + * + * This setting is useful only when the same filter is used + * _separately_ for multiple sections of the same executable file, + * and the sections contain cross-section branch/call/jump + * instructions. In that case it is beneficial to set the start + * offset of the non-first sections so that the relative addresses + * of the cross-section branch/call/jump instructions will use the + * same absolute addresses as in the first section. + * + * When the pointer to options is NULL, the default value (zero) + * is used. + */ + uint32_t start_offset; + +} lzma_options_bcj; + + +/** + * \brief Raw ARM64 BCJ encoder + * + * This is for special use cases only. + * + * \param start_offset The lowest 32 bits of the offset in the + * executable being filtered. For the ARM64 + * filter, this must be a multiple of four. + * For the very best results, this should also + * be in sync with 4096-byte page boundaries + * in the executable due to how ARM64's ADRP + * instruction works. + * \param buf Buffer to be filtered in place + * \param size Size of the buffer + * + * \return Number of bytes that were processed in `buf`. This is at most + * `size`. With the ARM64 filter, the return value is always + * a multiple of 4, and at most 3 bytes are left unfiltered. + * + * \since 5.7.1alpha + */ +extern LZMA_API(size_t) lzma_bcj_arm64_encode( + uint32_t start_offset, uint8_t *buf, size_t size) lzma_nothrow; + +/** + * \brief Raw ARM64 BCJ decoder + * + * See lzma_bcj_arm64_encode(). + * + * \since 5.7.1alpha + */ +extern LZMA_API(size_t) lzma_bcj_arm64_decode( + uint32_t start_offset, uint8_t *buf, size_t size) lzma_nothrow; + + +/** + * \brief Raw RISC-V BCJ encoder + * + * This is for special use cases only. + * + * \param start_offset The lowest 32 bits of the offset in the + * executable being filtered. For the RISC-V + * filter, this must be a multiple of 2. + * \param buf Buffer to be filtered in place + * \param size Size of the buffer + * + * \return Number of bytes that were processed in `buf`. This is at most + * `size`. With the RISC-V filter, the return value is always + * a multiple of 2, and at most 7 bytes are left unfiltered. + * + * \since 5.7.1alpha + */ +extern LZMA_API(size_t) lzma_bcj_riscv_encode( + uint32_t start_offset, uint8_t *buf, size_t size) lzma_nothrow; + +/** + * \brief Raw RISC-V BCJ decoder + * + * See lzma_bcj_riscv_encode(). + * + * \since 5.7.1alpha + */ +extern LZMA_API(size_t) lzma_bcj_riscv_decode( + uint32_t start_offset, uint8_t *buf, size_t size) lzma_nothrow; + + +/** + * \brief Raw x86 BCJ encoder + * + * This is for special use cases only. + * + * \param start_offset The lowest 32 bits of the offset in the + * executable being filtered. For the x86 + * filter, all values are valid. + * \param buf Buffer to be filtered in place + * \param size Size of the buffer + * + * \return Number of bytes that were processed in `buf`. This is at most + * `size`. For the x86 filter, the return value is always + * a multiple of 1, and at most 4 bytes are left unfiltered. + * + * \since 5.7.1alpha + */ +extern LZMA_API(size_t) lzma_bcj_x86_encode( + uint32_t start_offset, uint8_t *buf, size_t size) lzma_nothrow; + +/** + * \brief Raw x86 BCJ decoder + * + * See lzma_bcj_x86_encode(). + * + * \since 5.7.1alpha + */ +extern LZMA_API(size_t) lzma_bcj_x86_decode( + uint32_t start_offset, uint8_t *buf, size_t size) lzma_nothrow; diff --git a/src/liblzma/api/lzma/block.h b/src/liblzma/api/lzma/block.h new file mode 100644 index 0000000000..05b77e59aa --- /dev/null +++ b/src/liblzma/api/lzma/block.h @@ -0,0 +1,694 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/block.h + * \brief .xz Block handling + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Options for the Block and Block Header encoders and decoders + * + * Different Block handling functions use different parts of this structure. + * Some read some members, other functions write, and some do both. Only the + * members listed for reading need to be initialized when the specified + * functions are called. The members marked for writing will be assigned + * new values at some point either by calling the given function or by + * later calls to lzma_code(). + */ +typedef struct { + /** + * \brief Block format version + * + * To prevent API and ABI breakages when new features are needed, + * a version number is used to indicate which members in this + * structure are in use: + * - liblzma >= 5.0.0: version = 0 is supported. + * - liblzma >= 5.1.4beta: Support for version = 1 was added, + * which adds the ignore_check member. + * + * If version is greater than one, most Block related functions + * will return LZMA_OPTIONS_ERROR (lzma_block_header_decode() works + * with any version value). + * + * Read by: + * - lzma_block_header_size() + * - lzma_block_header_encode() + * - lzma_block_header_decode() + * - lzma_block_compressed_size() + * - lzma_block_unpadded_size() + * - lzma_block_total_size() + * - lzma_block_encoder() + * - lzma_block_decoder() + * - lzma_block_buffer_encode() + * - lzma_block_uncomp_encode() + * - lzma_block_buffer_decode() + * + * Written by: + * - lzma_block_header_decode() + */ + uint32_t version; + + /** + * \brief Size of the Block Header field in bytes + * + * This is always a multiple of four. + * + * Read by: + * - lzma_block_header_encode() + * - lzma_block_header_decode() + * - lzma_block_compressed_size() + * - lzma_block_unpadded_size() + * - lzma_block_total_size() + * - lzma_block_decoder() + * - lzma_block_buffer_decode() + * + * Written by: + * - lzma_block_header_size() + * - lzma_block_buffer_encode() + * - lzma_block_uncomp_encode() + */ + uint32_t header_size; +# define LZMA_BLOCK_HEADER_SIZE_MIN 8 +# define LZMA_BLOCK_HEADER_SIZE_MAX 1024 + + /** + * \brief Type of integrity Check + * + * The Check ID is not stored into the Block Header, thus its value + * must be provided also when decoding. + * + * Read by: + * - lzma_block_header_encode() + * - lzma_block_header_decode() + * - lzma_block_compressed_size() + * - lzma_block_unpadded_size() + * - lzma_block_total_size() + * - lzma_block_encoder() + * - lzma_block_decoder() + * - lzma_block_buffer_encode() + * - lzma_block_buffer_decode() + */ + lzma_check check; + + /** + * \brief Size of the Compressed Data in bytes + * + * Encoding: If this is not LZMA_VLI_UNKNOWN, Block Header encoder + * will store this value to the Block Header. Block encoder doesn't + * care about this value, but will set it once the encoding has been + * finished. + * + * Decoding: If this is not LZMA_VLI_UNKNOWN, Block decoder will + * verify that the size of the Compressed Data field matches + * compressed_size. + * + * Usually you don't know this value when encoding in streamed mode, + * and thus cannot write this field into the Block Header. + * + * In non-streamed mode you can reserve space for this field before + * encoding the actual Block. After encoding the data, finish the + * Block by encoding the Block Header. Steps in detail: + * + * - Set compressed_size to some big enough value. If you don't know + * better, use LZMA_VLI_MAX, but remember that bigger values take + * more space in Block Header. + * + * - Call lzma_block_header_size() to see how much space you need to + * reserve for the Block Header. + * + * - Encode the Block using lzma_block_encoder() and lzma_code(). + * It sets compressed_size to the correct value. + * + * - Use lzma_block_header_encode() to encode the Block Header. + * Because space was reserved in the first step, you don't need + * to call lzma_block_header_size() anymore, because due to + * reserving, header_size has to be big enough. If it is "too big", + * lzma_block_header_encode() will add enough Header Padding to + * make Block Header to match the size specified by header_size. + * + * Read by: + * - lzma_block_header_size() + * - lzma_block_header_encode() + * - lzma_block_compressed_size() + * - lzma_block_unpadded_size() + * - lzma_block_total_size() + * - lzma_block_decoder() + * - lzma_block_buffer_decode() + * + * Written by: + * - lzma_block_header_decode() + * - lzma_block_compressed_size() + * - lzma_block_encoder() + * - lzma_block_decoder() + * - lzma_block_buffer_encode() + * - lzma_block_uncomp_encode() + * - lzma_block_buffer_decode() + */ + lzma_vli compressed_size; + + /** + * \brief Uncompressed Size in bytes + * + * This is handled very similarly to compressed_size above. + * + * uncompressed_size is needed by fewer functions than + * compressed_size. This is because uncompressed_size isn't + * needed to validate that Block stays within proper limits. + * + * Read by: + * - lzma_block_header_size() + * - lzma_block_header_encode() + * - lzma_block_decoder() + * - lzma_block_buffer_decode() + * + * Written by: + * - lzma_block_header_decode() + * - lzma_block_encoder() + * - lzma_block_decoder() + * - lzma_block_buffer_encode() + * - lzma_block_uncomp_encode() + * - lzma_block_buffer_decode() + */ + lzma_vli uncompressed_size; + + /** + * \brief Array of filters + * + * There can be 1-4 filters. The end of the array is marked with + * .id = LZMA_VLI_UNKNOWN. + * + * Read by: + * - lzma_block_header_size() + * - lzma_block_header_encode() + * - lzma_block_encoder() + * - lzma_block_decoder() + * - lzma_block_buffer_encode() + * - lzma_block_buffer_decode() + * + * Written by: + * - lzma_block_header_decode(): Note that this does NOT free() + * the old filter options structures. All unused filters[] will + * have .id == LZMA_VLI_UNKNOWN and .options == NULL. If + * decoding fails, all filters[] are guaranteed to be + * LZMA_VLI_UNKNOWN and NULL. + * + * \note Because of the array is terminated with + * .id = LZMA_VLI_UNKNOWN, the actual array must + * have LZMA_FILTERS_MAX + 1 members or the Block + * Header decoder will overflow the buffer. + */ + lzma_filter *filters; + + /** + * \brief Raw value stored in the Check field + * + * After successful coding, the first lzma_check_size(check) bytes + * of this array contain the raw value stored in the Check field. + * + * Note that CRC32 and CRC64 are stored in little endian byte order. + * Take it into account if you display the Check values to the user. + * + * Written by: + * - lzma_block_encoder() + * - lzma_block_decoder() + * - lzma_block_buffer_encode() + * - lzma_block_uncomp_encode() + * - lzma_block_buffer_decode() + */ + uint8_t raw_check[LZMA_CHECK_SIZE_MAX]; + + /* + * Reserved space to allow possible future extensions without + * breaking the ABI. You should not touch these, because the names + * of these variables may change. These are and will never be used + * with the currently supported options, so it is safe to leave these + * uninitialized. + */ + + /** \private Reserved member. */ + void *reserved_ptr1; + + /** \private Reserved member. */ + void *reserved_ptr2; + + /** \private Reserved member. */ + void *reserved_ptr3; + + /** \private Reserved member. */ + uint32_t reserved_int1; + + /** \private Reserved member. */ + uint32_t reserved_int2; + + /** \private Reserved member. */ + lzma_vli reserved_int3; + + /** \private Reserved member. */ + lzma_vli reserved_int4; + + /** \private Reserved member. */ + lzma_vli reserved_int5; + + /** \private Reserved member. */ + lzma_vli reserved_int6; + + /** \private Reserved member. */ + lzma_vli reserved_int7; + + /** \private Reserved member. */ + lzma_vli reserved_int8; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum1; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum2; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum3; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum4; + + /** + * \brief A flag to Block decoder to not verify the Check field + * + * This member is supported by liblzma >= 5.1.4beta if .version >= 1. + * + * If this is set to true, the integrity check won't be calculated + * and verified. Unless you know what you are doing, you should + * leave this to false. (A reason to set this to true is when the + * file integrity is verified externally anyway and you want to + * speed up the decompression, which matters mostly when using + * SHA-256 as the integrity check.) + * + * If .version >= 1, read by: + * - lzma_block_decoder() + * - lzma_block_buffer_decode() + * + * Written by (.version is ignored): + * - lzma_block_header_decode() always sets this to false + */ + lzma_bool ignore_check; + + /** \private Reserved member. */ + lzma_bool reserved_bool2; + + /** \private Reserved member. */ + lzma_bool reserved_bool3; + + /** \private Reserved member. */ + lzma_bool reserved_bool4; + + /** \private Reserved member. */ + lzma_bool reserved_bool5; + + /** \private Reserved member. */ + lzma_bool reserved_bool6; + + /** \private Reserved member. */ + lzma_bool reserved_bool7; + + /** \private Reserved member. */ + lzma_bool reserved_bool8; + +} lzma_block; + + +/** + * \brief Decode the Block Header Size field + * + * To decode Block Header using lzma_block_header_decode(), the size of the + * Block Header has to be known and stored into lzma_block.header_size. + * The size can be calculated from the first byte of a Block using this macro. + * Note that if the first byte is 0x00, it indicates beginning of Index; use + * this macro only when the byte is not 0x00. + * + * There is no encoding macro because lzma_block_header_size() and + * lzma_block_header_encode() should be used. + */ +#define lzma_block_header_size_decode(b) (((uint32_t)(b) + 1) * 4) + + +/** + * \brief Calculate Block Header Size + * + * Calculate the minimum size needed for the Block Header field using the + * settings specified in the lzma_block structure. Note that it is OK to + * increase the calculated header_size value as long as it is a multiple of + * four and doesn't exceed LZMA_BLOCK_HEADER_SIZE_MAX. Increasing header_size + * just means that lzma_block_header_encode() will add Header Padding. + * + * \note This doesn't check that all the options are valid i.e. this + * may return LZMA_OK even if lzma_block_header_encode() or + * lzma_block_encoder() would fail. If you want to validate the + * filter chain, consider using lzma_memlimit_encoder() which as + * a side-effect validates the filter chain. + * + * \param block Block options + * + * \return Possible lzma_ret values: + * - LZMA_OK: Size calculated successfully and stored to + * block->header_size. + * - LZMA_OPTIONS_ERROR: Unsupported version, filters or + * filter options. + * - LZMA_PROG_ERROR: Invalid values like compressed_size == 0. + */ +extern LZMA_API(lzma_ret) lzma_block_header_size(lzma_block *block) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Encode Block Header + * + * The caller must have calculated the size of the Block Header already with + * lzma_block_header_size(). If a value larger than the one calculated by + * lzma_block_header_size() is used, the Block Header will be padded to the + * specified size. + * + * \param block Block options to be encoded. + * \param[out] out Beginning of the output buffer. This must be + * at least block->header_size bytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. block->header_size + * bytes were written to output buffer. + * - LZMA_OPTIONS_ERROR: Invalid or unsupported options. + * - LZMA_PROG_ERROR: Invalid arguments, for example + * block->header_size is invalid or block->filters is NULL. + */ +extern LZMA_API(lzma_ret) lzma_block_header_encode( + const lzma_block *block, uint8_t *out) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode Block Header + * + * block->version should (usually) be set to the highest value supported + * by the application. If the application sets block->version to a value + * higher than supported by the current liblzma version, this function will + * downgrade block->version to the highest value supported by it. Thus one + * should check the value of block->version after calling this function if + * block->version was set to a non-zero value and the application doesn't + * otherwise know that the liblzma version being used is new enough to + * support the specified block->version. + * + * The size of the Block Header must have already been decoded with + * lzma_block_header_size_decode() macro and stored to block->header_size. + * + * The integrity check type from Stream Header must have been stored + * to block->check. + * + * block->filters must have been allocated, but they don't need to be + * initialized (possible existing filter options are not freed). + * + * \param[out] block Destination for Block options + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() (and also free() + * if an error occurs). + * \param in Beginning of the input buffer. This must be + * at least block->header_size bytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. block->header_size + * bytes were read from the input buffer. + * - LZMA_OPTIONS_ERROR: The Block Header specifies some + * unsupported options such as unsupported filters. This can + * happen also if block->version was set to a too low value + * compared to what would be required to properly represent + * the information stored in the Block Header. + * - LZMA_DATA_ERROR: Block Header is corrupt, for example, + * the CRC32 doesn't match. + * - LZMA_PROG_ERROR: Invalid arguments, for example + * block->header_size is invalid or block->filters is NULL. + */ +extern LZMA_API(lzma_ret) lzma_block_header_decode(lzma_block *block, + const lzma_allocator *allocator, const uint8_t *in) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Validate and set Compressed Size according to Unpadded Size + * + * Block Header stores Compressed Size, but Index has Unpadded Size. If the + * application has already parsed the Index and is now decoding Blocks, + * it can calculate Compressed Size from Unpadded Size. This function does + * exactly that with error checking: + * + * - Compressed Size calculated from Unpadded Size must be positive integer, + * that is, Unpadded Size must be big enough that after Block Header and + * Check fields there's still at least one byte for Compressed Size. + * + * - If Compressed Size was present in Block Header, the new value + * calculated from Unpadded Size is compared against the value + * from Block Header. + * + * \note This function must be called _after_ decoding the Block Header + * field so that it can properly validate Compressed Size if it + * was present in Block Header. + * + * \param block Block options: block->header_size must + * already be set with lzma_block_header_size(). + * \param unpadded_size Unpadded Size from the Index field in bytes + * + * \return Possible lzma_ret values: + * - LZMA_OK: block->compressed_size was set successfully. + * - LZMA_DATA_ERROR: unpadded_size is too small compared to + * block->header_size and lzma_check_size(block->check). + * - LZMA_PROG_ERROR: Some values are invalid. For example, + * block->header_size must be a multiple of four and + * between 8 and 1024 inclusive. + */ +extern LZMA_API(lzma_ret) lzma_block_compressed_size( + lzma_block *block, lzma_vli unpadded_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Calculate Unpadded Size + * + * The Index field stores Unpadded Size and Uncompressed Size. The latter + * can be taken directly from the lzma_block structure after coding a Block, + * but Unpadded Size needs to be calculated from Block Header Size, + * Compressed Size, and size of the Check field. This is where this function + * is needed. + * + * \param block Block options: block->header_size must already be + * set with lzma_block_header_size(). + * + * \return Unpadded Size on success, or zero on error. + */ +extern LZMA_API(lzma_vli) lzma_block_unpadded_size(const lzma_block *block) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Calculate the total encoded size of a Block + * + * This is equivalent to lzma_block_unpadded_size() except that the returned + * value includes the size of the Block Padding field. + * + * \param block Block options: block->header_size must already be + * set with lzma_block_header_size(). + * + * \return On success, total encoded size of the Block. On error, + * zero is returned. + */ +extern LZMA_API(lzma_vli) lzma_block_total_size(const lzma_block *block) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Initialize .xz Block encoder + * + * Valid actions for lzma_code() are LZMA_RUN, LZMA_SYNC_FLUSH (only if the + * filter chain supports it), and LZMA_FINISH. + * + * The Block encoder encodes the Block Data, Block Padding, and Check value. + * It does NOT encode the Block Header which can be encoded with + * lzma_block_header_encode(). + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param block Block options: block->version, block->check, + * and block->filters must have been initialized. + * + * \return Possible lzma_ret values: + * - LZMA_OK: All good, continue with lzma_code(). + * - LZMA_MEM_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_UNSUPPORTED_CHECK: block->check specifies a Check ID + * that is not supported by this build of liblzma. Initializing + * the encoder failed. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_block_encoder( + lzma_stream *strm, lzma_block *block) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize .xz Block decoder + * + * Valid actions for lzma_code() are LZMA_RUN and LZMA_FINISH. Using + * LZMA_FINISH is not required. It is supported only for convenience. + * + * The Block decoder decodes the Block Data, Block Padding, and Check value. + * It does NOT decode the Block Header which can be decoded with + * lzma_block_header_decode(). + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param block Block options + * + * \return Possible lzma_ret values: + * - LZMA_OK: All good, continue with lzma_code(). + * - LZMA_PROG_ERROR + * - LZMA_MEM_ERROR + */ +extern LZMA_API(lzma_ret) lzma_block_decoder( + lzma_stream *strm, lzma_block *block) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Calculate maximum output size for single-call Block encoding + * + * This is equivalent to lzma_stream_buffer_bound() but for .xz Blocks. + * See the documentation of lzma_stream_buffer_bound(). + * + * \param uncompressed_size Size of the data to be encoded with the + * single-call Block encoder. + * + * \return Maximum output size in bytes for single-call Block encoding. + */ +extern LZMA_API(size_t) lzma_block_buffer_bound(size_t uncompressed_size) + lzma_nothrow; + + +/** + * \brief Single-call .xz Block encoder + * + * In contrast to the multi-call encoder initialized with + * lzma_block_encoder(), this function encodes also the Block Header. This + * is required to make it possible to write appropriate Block Header also + * in case the data isn't compressible, and different filter chain has to be + * used to encode the data in uncompressed form using uncompressed chunks + * of the LZMA2 filter. + * + * When the data isn't compressible, header_size, compressed_size, and + * uncompressed_size are set just like when the data was compressible, but + * it is possible that header_size is too small to hold the filter chain + * specified in block->filters, because that isn't necessarily the filter + * chain that was actually used to encode the data. lzma_block_unpadded_size() + * still works normally, because it doesn't read the filters array. + * + * \param block Block options: block->version, block->check, + * and block->filters must have been initialized. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_size Size of the input buffer + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_BUF_ERROR: Not enough output buffer space. + * - LZMA_UNSUPPORTED_CHECK + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_block_buffer_encode( + lzma_block *block, const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Single-call uncompressed .xz Block encoder + * + * This is like lzma_block_buffer_encode() except this doesn't try to + * compress the data and instead encodes the data using LZMA2 uncompressed + * chunks. The required output buffer size can be determined with + * lzma_block_buffer_bound(). + * + * Since the data won't be compressed, this function ignores block->filters. + * This function doesn't take lzma_allocator because this function doesn't + * allocate any memory from the heap. + * + * \param block Block options: block->version, block->check, + * and block->filters must have been initialized. + * \param in Beginning of the input buffer + * \param in_size Size of the input buffer + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_BUF_ERROR: Not enough output buffer space. + * - LZMA_UNSUPPORTED_CHECK + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_block_uncomp_encode(lzma_block *block, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Single-call .xz Block decoder + * + * This is single-call equivalent of lzma_block_decoder(), and requires that + * the caller has already decoded Block Header and checked its memory usage. + * + * \param block Block options + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_pos The next byte will be read from in[*in_pos]. + * *in_pos is updated only if decoding succeeds. + * \param in_size Size of the input buffer; the first byte that + * won't be read is in[in_size]. + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. + * - LZMA_OPTIONS_ERROR + * - LZMA_DATA_ERROR + * - LZMA_MEM_ERROR + * - LZMA_BUF_ERROR: Output buffer was too small. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_block_buffer_decode( + lzma_block *block, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow; diff --git a/src/liblzma/api/lzma/check.h b/src/liblzma/api/lzma/check.h new file mode 100644 index 0000000000..e7a50ed3a3 --- /dev/null +++ b/src/liblzma/api/lzma/check.h @@ -0,0 +1,163 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/check.h + * \brief Integrity checks + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Type of the integrity check (Check ID) + * + * The .xz format supports multiple types of checks that are calculated + * from the uncompressed data. They vary in both speed and ability to + * detect errors. + */ +typedef enum { + LZMA_CHECK_NONE = 0, + /**< + * No Check is calculated. + * + * Size of the Check field: 0 bytes + */ + + LZMA_CHECK_CRC32 = 1, + /**< + * CRC32 using the polynomial from the IEEE 802.3 standard + * + * Size of the Check field: 4 bytes + */ + + LZMA_CHECK_CRC64 = 4, + /**< + * CRC64 using the polynomial from the ECMA-182 standard + * + * Size of the Check field: 8 bytes + */ + + LZMA_CHECK_SHA256 = 10 + /**< + * SHA-256 + * + * Size of the Check field: 32 bytes + */ +} lzma_check; + + +/** + * \brief Maximum valid Check ID + * + * The .xz file format specification specifies 16 Check IDs (0-15). Some + * of them are only reserved, that is, no actual Check algorithm has been + * assigned. When decoding, liblzma still accepts unknown Check IDs for + * future compatibility. If a valid but unsupported Check ID is detected, + * liblzma can indicate a warning; see the flags LZMA_TELL_NO_CHECK, + * LZMA_TELL_UNSUPPORTED_CHECK, and LZMA_TELL_ANY_CHECK in container.h. + */ +#define LZMA_CHECK_ID_MAX 15 + + +/** + * \brief Test if the given Check ID is supported + * + * LZMA_CHECK_NONE and LZMA_CHECK_CRC32 are always supported (even if + * liblzma is built with limited features). + * + * \note It is safe to call this with a value that is not in the + * range [0, 15]; in that case the return value is always false. + * + * \param check Check ID + * + * \return lzma_bool: + * - true if Check ID is supported by this liblzma build. + * - false otherwise. + */ +extern LZMA_API(lzma_bool) lzma_check_is_supported(lzma_check check) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Get the size of the Check field with the given Check ID + * + * Although not all Check IDs have a check algorithm associated, the size of + * every Check is already frozen. This function returns the size (in bytes) of + * the Check field with the specified Check ID. The values are: + * { 0, 4, 4, 4, 8, 8, 8, 16, 16, 16, 32, 32, 32, 64, 64, 64 } + * + * \param check Check ID + * + * \return Size of the Check field in bytes. If the argument is not in + * the range [0, 15], UINT32_MAX is returned. + */ +extern LZMA_API(uint32_t) lzma_check_size(lzma_check check) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Maximum size of a Check field + */ +#define LZMA_CHECK_SIZE_MAX 64 + + +/** + * \brief Calculate CRC32 + * + * Calculate CRC32 using the polynomial from the IEEE 802.3 standard. + * + * \param buf Pointer to the input buffer + * \param size Size of the input buffer + * \param crc Previously returned CRC value. This is used to + * calculate the CRC of a big buffer in smaller chunks. + * Set to zero when starting a new calculation. + * + * \return Updated CRC value, which can be passed to this function + * again to continue CRC calculation. + */ +extern LZMA_API(uint32_t) lzma_crc32( + const uint8_t *buf, size_t size, uint32_t crc) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Calculate CRC64 + * + * Calculate CRC64 using the polynomial from the ECMA-182 standard. + * + * This function is used similarly to lzma_crc32(). + * + * \param buf Pointer to the input buffer + * \param size Size of the input buffer + * \param crc Previously returned CRC value. This is used to + * calculate the CRC of a big buffer in smaller chunks. + * Set to zero when starting a new calculation. + * + * \return Updated CRC value, which can be passed to this function + * again to continue CRC calculation. + */ +extern LZMA_API(uint64_t) lzma_crc64( + const uint8_t *buf, size_t size, uint64_t crc) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the type of the integrity check + * + * This function can be called only immediately after lzma_code() has + * returned LZMA_NO_CHECK, LZMA_UNSUPPORTED_CHECK, or LZMA_GET_CHECK. + * Calling this function in any other situation has undefined behavior. + * + * \param strm Pointer to lzma_stream meeting the above conditions. + * + * \return Check ID in the lzma_stream, or undefined if called improperly. + */ +extern LZMA_API(lzma_check) lzma_get_check(const lzma_stream *strm) + lzma_nothrow; diff --git a/src/liblzma/api/lzma/container.h b/src/liblzma/api/lzma/container.h new file mode 100644 index 0000000000..dbd414cbf8 --- /dev/null +++ b/src/liblzma/api/lzma/container.h @@ -0,0 +1,995 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/container.h + * \brief File formats + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/************ + * Encoding * + ************/ + +/** + * \brief Default compression preset + * + * It's not straightforward to recommend a default preset, because in some + * cases keeping the resource usage relatively low is more important that + * getting the maximum compression ratio. + */ +#define LZMA_PRESET_DEFAULT UINT32_C(6) + + +/** + * \brief Mask for preset level + * + * This is useful only if you need to extract the level from the preset + * variable. That should be rare. + */ +#define LZMA_PRESET_LEVEL_MASK UINT32_C(0x1F) + + +/* + * Preset flags + * + * Currently only one flag is defined. + */ + +/** + * \brief Extreme compression preset + * + * This flag modifies the preset to make the encoding significantly slower + * while improving the compression ratio only marginally. This is useful + * when you don't mind spending time to get as small result as possible. + * + * This flag doesn't affect the memory usage requirements of the decoder (at + * least not significantly). The memory usage of the encoder may be increased + * a little but only at the lowest preset levels (0-3). + */ +#define LZMA_PRESET_EXTREME (UINT32_C(1) << 31) + + +/** + * \brief Multithreading options + */ +typedef struct { + /** + * \brief Flags + * + * Set this to zero if no flags are wanted. + * + * Encoder: No flags are currently supported. + * + * Decoder: Bitwise-or of zero or more of the decoder flags: + * - LZMA_TELL_NO_CHECK + * - LZMA_TELL_UNSUPPORTED_CHECK + * - LZMA_TELL_ANY_CHECK + * - LZMA_IGNORE_CHECK + * - LZMA_CONCATENATED + * - LZMA_FAIL_FAST + */ + uint32_t flags; + + /** + * \brief Number of worker threads to use + */ + uint32_t threads; + + /** + * \brief Encoder only: Maximum uncompressed size of a Block + * + * The encoder will start a new .xz Block every block_size bytes. + * Using LZMA_FULL_FLUSH or LZMA_FULL_BARRIER with lzma_code() + * the caller may tell liblzma to start a new Block earlier. + * + * With LZMA2, a recommended block size is 2-4 times the LZMA2 + * dictionary size. With very small dictionaries, it is recommended + * to use at least 1 MiB block size for good compression ratio, even + * if this is more than four times the dictionary size. Note that + * these are only recommendations for typical use cases; feel free + * to use other values. Just keep in mind that using a block size + * less than the LZMA2 dictionary size is waste of RAM. + * + * Set this to 0 to let liblzma choose the block size depending + * on the compression options. For LZMA2 it will be 3*dict_size + * or 1 MiB, whichever is more. + * + * For each thread, about 3 * block_size bytes of memory will be + * allocated. This may change in later liblzma versions. If so, + * the memory usage will probably be reduced, not increased. + */ + uint64_t block_size; + + /** + * \brief Timeout to allow lzma_code() to return early + * + * Multithreading can make liblzma consume input and produce + * output in a very bursty way: it may first read a lot of input + * to fill internal buffers, then no input or output occurs for + * a while. + * + * In single-threaded mode, lzma_code() won't return until it has + * either consumed all the input or filled the output buffer. If + * this is done in multithreaded mode, it may cause a call + * lzma_code() to take even tens of seconds, which isn't acceptable + * in all applications. + * + * To avoid very long blocking times in lzma_code(), a timeout + * (in milliseconds) may be set here. If lzma_code() would block + * longer than this number of milliseconds, it will return with + * LZMA_OK. Reasonable values are 100 ms or more. The xz command + * line tool uses 300 ms. + * + * If long blocking times are acceptable, set timeout to a special + * value of 0. This will disable the timeout mechanism and will make + * lzma_code() block until all the input is consumed or the output + * buffer has been filled. + * + * \note Even with a timeout, lzma_code() might sometimes take + * a long time to return. No timing guarantees are made. + */ + uint32_t timeout; + + /** + * \brief Encoder only: Compression preset + * + * The preset is set just like with lzma_easy_encoder(). + * The preset is ignored if filters below is non-NULL. + */ + uint32_t preset; + + /** + * \brief Encoder only: Filter chain (alternative to a preset) + * + * If this is NULL, the preset above is used. Otherwise the preset + * is ignored and the filter chain specified here is used. + */ + const lzma_filter *filters; + + /** + * \brief Encoder only: Integrity check type + * + * See check.h for available checks. The xz command line tool + * defaults to LZMA_CHECK_CRC64, which is a good choice if you + * are unsure. + */ + lzma_check check; + + /* + * Reserved space to allow possible future extensions without + * breaking the ABI. You should not touch these, because the names + * of these variables may change. These are and will never be used + * with the currently supported options, so it is safe to leave these + * uninitialized. + */ + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum1; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum2; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum3; + + /** \private Reserved member. */ + uint32_t reserved_int1; + + /** \private Reserved member. */ + uint32_t reserved_int2; + + /** \private Reserved member. */ + uint32_t reserved_int3; + + /** \private Reserved member. */ + uint32_t reserved_int4; + + /** + * \brief Memory usage limit to reduce the number of threads + * + * Encoder: Ignored. + * + * Decoder: + * + * If the number of threads has been set so high that more than + * memlimit_threading bytes of memory would be needed, the number + * of threads will be reduced so that the memory usage will not exceed + * memlimit_threading bytes. However, if memlimit_threading cannot + * be met even in single-threaded mode, then decoding will continue + * in single-threaded mode and memlimit_threading may be exceeded + * even by a large amount. That is, memlimit_threading will never make + * lzma_code() return LZMA_MEMLIMIT_ERROR. To truly cap the memory + * usage, see memlimit_stop below. + * + * Setting memlimit_threading to UINT64_MAX or a similar huge value + * means that liblzma is allowed to keep the whole compressed file + * and the whole uncompressed file in memory in addition to the memory + * needed by the decompressor data structures used by each thread! + * In other words, a reasonable value limit must be set here or it + * will cause problems sooner or later. If you have no idea what + * a reasonable value could be, try lzma_physmem() / 4 as a starting + * point. Setting this limit will never prevent decompression of + * a file; this will only reduce the number of threads. + * + * If memlimit_threading is greater than memlimit_stop, then the value + * of memlimit_stop will be used for both. + */ + uint64_t memlimit_threading; + + /** + * \brief Memory usage limit that should never be exceeded + * + * Encoder: Ignored. + * + * Decoder: If decompressing will need more than this amount of + * memory even in the single-threaded mode, then lzma_code() will + * return LZMA_MEMLIMIT_ERROR. + */ + uint64_t memlimit_stop; + + /** \private Reserved member. */ + uint64_t reserved_int7; + + /** \private Reserved member. */ + uint64_t reserved_int8; + + /** \private Reserved member. */ + void *reserved_ptr1; + + /** \private Reserved member. */ + void *reserved_ptr2; + + /** \private Reserved member. */ + void *reserved_ptr3; + + /** \private Reserved member. */ + void *reserved_ptr4; + +} lzma_mt; + + +/** + * \brief Calculate approximate memory usage of easy encoder + * + * This function is a wrapper for lzma_raw_encoder_memusage(). + * + * \param preset Compression preset (level and possible flags) + * + * \return Number of bytes of memory required for the given + * preset when encoding or UINT64_MAX on error. + */ +extern LZMA_API(uint64_t) lzma_easy_encoder_memusage(uint32_t preset) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Calculate approximate decoder memory usage of a preset + * + * This function is a wrapper for lzma_raw_decoder_memusage(). + * + * \param preset Compression preset (level and possible flags) + * + * \return Number of bytes of memory required to decompress a file + * that was compressed using the given preset or UINT64_MAX + * on error. + */ +extern LZMA_API(uint64_t) lzma_easy_decoder_memusage(uint32_t preset) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Initialize .xz Stream encoder using a preset number + * + * This function is intended for those who just want to use the basic features + * of liblzma (that is, most developers out there). + * + * If initialization fails (return value is not LZMA_OK), all the memory + * allocated for *strm by liblzma is always freed. Thus, there is no need + * to call lzma_end() after failed initialization. + * + * If initialization succeeds, use lzma_code() to do the actual encoding. + * Valid values for 'action' (the second argument of lzma_code()) are + * LZMA_RUN, LZMA_SYNC_FLUSH, LZMA_FULL_FLUSH, and LZMA_FINISH. In future, + * there may be compression levels or flags that don't support LZMA_SYNC_FLUSH. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param preset Compression preset to use. A preset consist of level + * number and zero or more flags. Usually flags aren't + * used, so preset is simply a number [0, 9] which match + * the options -0 ... -9 of the xz command line tool. + * Additional flags can be set using bitwise-or with + * the preset level number, e.g. 6 | LZMA_PRESET_EXTREME. + * \param check Integrity check type to use. See check.h for available + * checks. The xz command line tool defaults to + * LZMA_CHECK_CRC64, which is a good choice if you are + * unsure. LZMA_CHECK_CRC32 is good too as long as the + * uncompressed file is not many gigabytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization succeeded. Use lzma_code() to + * encode your data. + * - LZMA_MEM_ERROR: Memory allocation failed. + * - LZMA_OPTIONS_ERROR: The given compression preset is not + * supported by this build of liblzma. + * - LZMA_UNSUPPORTED_CHECK: The given check type is not + * supported by this liblzma build. + * - LZMA_PROG_ERROR: One or more of the parameters have values + * that will never be valid. For example, strm == NULL. + */ +extern LZMA_API(lzma_ret) lzma_easy_encoder( + lzma_stream *strm, uint32_t preset, lzma_check check) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Single-call .xz Stream encoding using a preset number + * + * The maximum required output buffer size can be calculated with + * lzma_stream_buffer_bound(). + * + * \param preset Compression preset to use. See the description + * in lzma_easy_encoder(). + * \param check Type of the integrity check to calculate from + * uncompressed data. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_size Size of the input buffer + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_BUF_ERROR: Not enough output buffer space. + * - LZMA_UNSUPPORTED_CHECK + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_easy_buffer_encode( + uint32_t preset, lzma_check check, + const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) lzma_nothrow; + + +/** + * \brief Initialize .xz Stream encoder using a custom filter chain + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. See filters.h for more + * information. + * \param check Type of the integrity check to calculate from + * uncompressed data. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR + * - LZMA_UNSUPPORTED_CHECK + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_encoder(lzma_stream *strm, + const lzma_filter *filters, lzma_check check) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Calculate approximate memory usage of multithreaded .xz encoder + * + * Since doing the encoding in threaded mode doesn't affect the memory + * requirements of single-threaded decompressor, you can use + * lzma_easy_decoder_memusage(options->preset) or + * lzma_raw_decoder_memusage(options->filters) to calculate + * the decompressor memory requirements. + * + * \param options Compression options + * + * \return Number of bytes of memory required for encoding with the + * given options. If an error occurs, for example due to + * unsupported preset or filter chain, UINT64_MAX is returned. + */ +extern LZMA_API(uint64_t) lzma_stream_encoder_mt_memusage( + const lzma_mt *options) lzma_nothrow lzma_attr_pure; + + +/** + * \brief Initialize multithreaded .xz Stream encoder + * + * This provides the functionality of lzma_easy_encoder() and + * lzma_stream_encoder() as a single function for multithreaded use. + * + * The supported actions for lzma_code() are LZMA_RUN, LZMA_FULL_FLUSH, + * LZMA_FULL_BARRIER, and LZMA_FINISH. Support for LZMA_SYNC_FLUSH might be + * added in the future. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param options Pointer to multithreaded compression options + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_UNSUPPORTED_CHECK + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_encoder_mt( + lzma_stream *strm, const lzma_mt *options) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Calculate recommended Block size for multithreaded .xz encoder + * + * This calculates a recommended Block size for multithreaded encoding given + * a filter chain. This is used internally by lzma_stream_encoder_mt() to + * determine the Block size if the block_size member is not set to the + * special value of 0 in the lzma_mt options struct. + * + * If one wishes to change the filters between Blocks, this function is + * helpful to set the block_size member of the lzma_mt struct before calling + * lzma_stream_encoder_mt(). Since the block_size member represents the + * maximum possible Block size for the multithreaded .xz encoder, one can + * use this function to find the maximum recommended Block size based on + * all planned filter chains. Otherwise, the multithreaded encoder will + * base its maximum Block size on the first filter chain used (if the + * block_size member is not set), which may unnecessarily limit the Block + * size for a later filter chain. + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Recommended Block size in bytes, or UINT64_MAX if + * an error occurred. + */ +extern LZMA_API(uint64_t) lzma_mt_block_size(const lzma_filter *filters) + lzma_nothrow; + + +/** + * \brief Initialize .lzma encoder (legacy file format) + * + * The .lzma format is sometimes called the LZMA_Alone format, which is the + * reason for the name of this function. The .lzma format supports only the + * LZMA1 filter. There is no support for integrity checks like CRC32. + * + * Use this function if and only if you need to create files readable by + * legacy LZMA tools such as LZMA Utils 4.32.x. Moving to the .xz format + * is strongly recommended. + * + * The valid action values for lzma_code() are LZMA_RUN and LZMA_FINISH. + * No kind of flushing is supported, because the file format doesn't make + * it possible. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param options Pointer to encoder options + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_alone_encoder( + lzma_stream *strm, const lzma_options_lzma *options) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Calculate output buffer size for single-call Stream encoder + * + * When trying to compress incompressible data, the encoded size will be + * slightly bigger than the input data. This function calculates how much + * output buffer space is required to be sure that lzma_stream_buffer_encode() + * doesn't return LZMA_BUF_ERROR. + * + * The calculated value is not exact, but it is guaranteed to be big enough. + * The actual maximum output space required may be slightly smaller (up to + * about 100 bytes). This should not be a problem in practice. + * + * If the calculated maximum size doesn't fit into size_t or would make the + * Stream grow past LZMA_VLI_MAX (which should never happen in practice), + * zero is returned to indicate the error. + * + * \note The limit calculated by this function applies only to + * single-call encoding. Multi-call encoding may (and probably + * will) have larger maximum expansion when encoding + * incompressible data. Currently there is no function to + * calculate the maximum expansion of multi-call encoding. + * + * \param uncompressed_size Size in bytes of the uncompressed + * input data + * + * \return Maximum number of bytes needed to store the compressed data. + */ +extern LZMA_API(size_t) lzma_stream_buffer_bound(size_t uncompressed_size) + lzma_nothrow; + + +/** + * \brief Single-call .xz Stream encoder + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. See filters.h for more + * information. + * \param check Type of the integrity check to calculate from + * uncompressed data. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_size Size of the input buffer + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_BUF_ERROR: Not enough output buffer space. + * - LZMA_UNSUPPORTED_CHECK + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_buffer_encode( + lzma_filter *filters, lzma_check check, + const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief MicroLZMA encoder + * + * The MicroLZMA format is a raw LZMA stream whose first byte (always 0x00) + * has been replaced with bitwise-negation of the LZMA properties (lc/lp/pb). + * This encoding ensures that the first byte of MicroLZMA stream is never + * 0x00. There is no end of payload marker and thus the uncompressed size + * must be stored separately. For the best error detection the dictionary + * size should be stored separately as well but alternatively one may use + * the uncompressed size as the dictionary size when decoding. + * + * With the MicroLZMA encoder, lzma_code() behaves slightly unusually. + * The action argument must be LZMA_FINISH and the return value will never be + * LZMA_OK. Thus the encoding is always done with a single lzma_code() after + * the initialization. The benefit of the combination of initialization + * function and lzma_code() is that memory allocations can be reused for + * better performance. + * + * lzma_code() will try to encode as much input as is possible to fit into + * the given output buffer. If not all input can be encoded, the stream will + * be finished without encoding all the input. The caller must check both + * input and output buffer usage after lzma_code() (total_in and total_out + * in lzma_stream can be convenient). Often lzma_code() can fill the output + * buffer completely if there is a lot of input, but sometimes a few bytes + * may remain unused because the next LZMA symbol would require more space. + * + * lzma_stream.avail_out must be at least 6. Otherwise LZMA_PROG_ERROR + * will be returned. + * + * The LZMA dictionary should be reasonably low to speed up the encoder + * re-initialization. A good value is bigger than the resulting + * uncompressed size of most of the output chunks. For example, if output + * size is 4 KiB, dictionary size of 32 KiB or 64 KiB is good. If the + * data compresses extremely well, even 128 KiB may be useful. + * + * The MicroLZMA format and this encoder variant were made with the EROFS + * file system in mind. This format may be convenient in other embedded + * uses too where many small streams are needed. XZ Embedded includes a + * decoder for this format. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param options Pointer to encoder options + * + * \return Possible lzma_ret values: + * - LZMA_STREAM_END: All good. Check the amounts of input used + * and output produced. Store the amount of input used + * (uncompressed size) as it needs to be known to decompress + * the data. + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR: In addition to the generic reasons for this + * error code, this may also be returned if there isn't enough + * output space (6 bytes) to create a valid MicroLZMA stream. + */ +extern LZMA_API(lzma_ret) lzma_microlzma_encoder( + lzma_stream *strm, const lzma_options_lzma *options) + lzma_nothrow; + + +/************ + * Decoding * + ************/ + +/** + * This flag makes lzma_code() return LZMA_NO_CHECK if the input stream + * being decoded has no integrity check. Note that when used with + * lzma_auto_decoder(), all .lzma files will trigger LZMA_NO_CHECK + * if LZMA_TELL_NO_CHECK is used. + */ +#define LZMA_TELL_NO_CHECK UINT32_C(0x01) + + +/** + * This flag makes lzma_code() return LZMA_UNSUPPORTED_CHECK if the input + * stream has an integrity check, but the type of the integrity check is not + * supported by this liblzma version or build. Such files can still be + * decoded, but the integrity check cannot be verified. + */ +#define LZMA_TELL_UNSUPPORTED_CHECK UINT32_C(0x02) + + +/** + * This flag makes lzma_code() return LZMA_GET_CHECK as soon as the type + * of the integrity check is known. The type can then be got with + * lzma_get_check(). + */ +#define LZMA_TELL_ANY_CHECK UINT32_C(0x04) + + +/** + * This flag makes lzma_code() not calculate and verify the integrity check + * of the compressed data in .xz files. This means that invalid integrity + * check values won't be detected and LZMA_DATA_ERROR won't be returned in + * such cases. + * + * This flag only affects the checks of the compressed data itself; the CRC32 + * values in the .xz headers will still be verified normally. + * + * Don't use this flag unless you know what you are doing. Possible reasons + * to use this flag: + * + * - Trying to recover data from a corrupt .xz file. + * + * - Speeding up decompression, which matters mostly with SHA-256 + * or with files that have compressed extremely well. It's recommended + * to not use this flag for this purpose unless the file integrity is + * verified externally in some other way. + * + * Support for this flag was added in liblzma 5.1.4beta. + */ +#define LZMA_IGNORE_CHECK UINT32_C(0x10) + + +/** + * This flag enables decoding of concatenated files with file formats that + * allow concatenating compressed files as is. From the formats currently + * supported by liblzma, only the .xz and .lz formats allow concatenated + * files. Concatenated files are not allowed with the legacy .lzma format. + * + * This flag also affects the usage of the 'action' argument for lzma_code(). + * When LZMA_CONCATENATED is used, lzma_code() won't return LZMA_STREAM_END + * unless LZMA_FINISH is used as 'action'. Thus, the application has to set + * LZMA_FINISH in the same way as it does when encoding. + * + * If LZMA_CONCATENATED is not used, the decoders still accept LZMA_FINISH + * as 'action' for lzma_code(), but the usage of LZMA_FINISH isn't required. + */ +#define LZMA_CONCATENATED UINT32_C(0x08) + + +/** + * This flag makes the threaded decoder report errors (like LZMA_DATA_ERROR) + * as soon as they are detected. This saves time when the application has no + * interest in a partially decompressed truncated or corrupt file. Note that + * due to timing randomness, if the same truncated or corrupt input is + * decompressed multiple times with this flag, a different amount of output + * may be produced by different runs, and even the error code might vary. + * + * When using LZMA_FAIL_FAST, it is recommended to use LZMA_FINISH to tell + * the decoder when no more input will be coming because it can help fast + * detection and reporting of truncated files. Note that in this situation + * truncated files might be diagnosed with LZMA_DATA_ERROR instead of + * LZMA_OK or LZMA_BUF_ERROR! + * + * Without this flag the threaded decoder will provide as much output as + * possible at first and then report the pending error. This default behavior + * matches the single-threaded decoder and provides repeatable behavior + * with truncated or corrupt input. There are a few special cases where the + * behavior can still differ like memory allocation failures (LZMA_MEM_ERROR). + * + * Single-threaded decoders currently ignore this flag. + * + * Support for this flag was added in liblzma 5.3.3alpha. Note that in older + * versions this flag isn't supported (LZMA_OPTIONS_ERROR) even by functions + * that ignore this flag in newer liblzma versions. + */ +#define LZMA_FAIL_FAST UINT32_C(0x20) + + +/** + * \brief Initialize .xz Stream decoder + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param memlimit Memory usage limit as bytes. Use UINT64_MAX + * to effectively disable the limiter. liblzma + * 5.2.3 and earlier don't allow 0 here and return + * LZMA_PROG_ERROR; later versions treat 0 as if 1 + * had been specified. + * \param flags Bitwise-or of zero or more of the decoder flags: + * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, + * LZMA_TELL_ANY_CHECK, LZMA_IGNORE_CHECK, + * LZMA_CONCATENATED, LZMA_FAIL_FAST + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR: Cannot allocate memory. + * - LZMA_OPTIONS_ERROR: Unsupported flags + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_decoder( + lzma_stream *strm, uint64_t memlimit, uint32_t flags) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize multithreaded .xz Stream decoder + * + * The decoder can decode multiple Blocks in parallel. This requires that each + * Block Header contains the Compressed Size and Uncompressed size fields + * which are added by the multi-threaded encoder, see lzma_stream_encoder_mt(). + * + * A Stream with one Block will only utilize one thread. A Stream with multiple + * Blocks but without size information in Block Headers will be processed in + * single-threaded mode in the same way as done by lzma_stream_decoder(). + * Concatenated Streams are processed one Stream at a time; no inter-Stream + * parallelization is done. + * + * This function behaves like lzma_stream_decoder() when options->threads == 1 + * and options->memlimit_threading <= 1. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param options Pointer to multithreaded compression options + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR: Cannot allocate memory. + * - LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. + * - LZMA_OPTIONS_ERROR: Unsupported flags. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_decoder_mt( + lzma_stream *strm, const lzma_mt *options) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode .xz, .lzma, and .lz (lzip) files with autodetection + * + * This decoder autodetects between the .xz, .lzma, and .lz file formats, + * and calls lzma_stream_decoder(), lzma_alone_decoder(), or + * lzma_lzip_decoder() once the type of the input file has been detected. + * + * Support for .lz was added in 5.4.0. + * + * If the flag LZMA_CONCATENATED is used and the input is a .lzma file: + * For historical reasons concatenated .lzma files aren't supported. + * If there is trailing data after one .lzma stream, lzma_code() will + * return LZMA_DATA_ERROR. (lzma_alone_decoder() doesn't have such a check + * as it doesn't support any decoder flags. It will return LZMA_STREAM_END + * after one .lzma stream.) + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param memlimit Memory usage limit as bytes. Use UINT64_MAX + * to effectively disable the limiter. liblzma + * 5.2.3 and earlier don't allow 0 here and return + * LZMA_PROG_ERROR; later versions treat 0 as if 1 + * had been specified. + * \param flags Bitwise-or of zero or more of the decoder flags: + * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, + * LZMA_TELL_ANY_CHECK, LZMA_IGNORE_CHECK, + * LZMA_CONCATENATED, LZMA_FAIL_FAST + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR: Cannot allocate memory. + * - LZMA_OPTIONS_ERROR: Unsupported flags + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_auto_decoder( + lzma_stream *strm, uint64_t memlimit, uint32_t flags) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize .lzma decoder (legacy file format) + * + * Valid 'action' arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. + * There is no need to use LZMA_FINISH, but it's allowed because it may + * simplify certain types of applications. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param memlimit Memory usage limit as bytes. Use UINT64_MAX + * to effectively disable the limiter. liblzma + * 5.2.3 and earlier don't allow 0 here and return + * LZMA_PROG_ERROR; later versions treat 0 as if 1 + * had been specified. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_alone_decoder( + lzma_stream *strm, uint64_t memlimit) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize .lz (lzip) decoder (a foreign file format) + * + * This decoder supports the .lz format version 0 and the unextended .lz + * format version 1: + * + * - Files in the format version 0 were produced by lzip 1.3 and older. + * Such files aren't common but may be found from file archives + * as a few source packages were released in this format. People + * might have old personal files in this format too. Decompression + * support for the format version 0 was removed in lzip 1.18. + * + * - lzip 1.3 added decompression support for .lz format version 1 files. + * Compression support was added in lzip 1.4. In lzip 1.6 the .lz format + * version 1 was extended to support the Sync Flush marker. This extension + * is not supported by liblzma. lzma_code() will return LZMA_DATA_ERROR + * at the location of the Sync Flush marker. In practice files with + * the Sync Flush marker are very rare and thus liblzma can decompress + * almost all .lz files. + * + * Just like with lzma_stream_decoder() for .xz files, LZMA_CONCATENATED + * should be used when decompressing normal standalone .lz files. + * + * The .lz format allows putting non-.lz data at the end of a file after at + * least one valid .lz member. That is, one can append custom data at the end + * of a .lz file and the decoder is required to ignore it. In liblzma this + * is relevant only when LZMA_CONCATENATED is used. In that case lzma_code() + * will return LZMA_STREAM_END and leave lzma_stream.next_in pointing to + * the first byte of the non-.lz data. An exception to this is if the first + * 1-3 bytes of the non-.lz data are identical to the .lz magic bytes + * (0x4C, 0x5A, 0x49, 0x50; "LZIP" in US-ASCII). In such a case the 1-3 bytes + * will have been ignored by lzma_code(). If one wishes to locate the non-.lz + * data reliably, one must ensure that the first byte isn't 0x4C. Actually + * one should ensure that none of the first four bytes of trailing data are + * equal to the magic bytes because lzip >= 1.20 requires it by default. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param memlimit Memory usage limit as bytes. Use UINT64_MAX + * to effectively disable the limiter. + * \param flags Bitwise-or of flags, or zero for no flags. + * All decoder flags listed above are supported + * although only LZMA_CONCATENATED and (in very rare + * cases) LZMA_IGNORE_CHECK are actually useful. + * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, + * and LZMA_FAIL_FAST do nothing. LZMA_TELL_ANY_CHECK + * is supported for consistency only as CRC32 is + * always used in the .lz format. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR: Cannot allocate memory. + * - LZMA_OPTIONS_ERROR: Unsupported flags + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_lzip_decoder( + lzma_stream *strm, uint64_t memlimit, uint32_t flags) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Single-call .xz Stream decoder + * + * \param memlimit Pointer to how much memory the decoder is allowed + * to allocate. The value pointed by this pointer is + * modified if and only if LZMA_MEMLIMIT_ERROR is + * returned. + * \param flags Bitwise-or of zero or more of the decoder flags: + * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, + * LZMA_IGNORE_CHECK, LZMA_CONCATENATED, + * LZMA_FAIL_FAST. Note that LZMA_TELL_ANY_CHECK + * is not allowed and will return LZMA_PROG_ERROR. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_pos The next byte will be read from in[*in_pos]. + * *in_pos is updated only if decoding succeeds. + * \param in_size Size of the input buffer; the first byte that + * won't be read is in[in_size]. + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if decoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. + * - LZMA_FORMAT_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_DATA_ERROR + * - LZMA_NO_CHECK: This can be returned only if using + * the LZMA_TELL_NO_CHECK flag. + * - LZMA_UNSUPPORTED_CHECK: This can be returned only if using + * the LZMA_TELL_UNSUPPORTED_CHECK flag. + * - LZMA_MEM_ERROR + * - LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. + * The minimum required memlimit value was stored to *memlimit. + * - LZMA_BUF_ERROR: Output buffer was too small. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_buffer_decode( + uint64_t *memlimit, uint32_t flags, + const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief MicroLZMA decoder + * + * See lzma_microlzma_encoder() for more information. + * + * The lzma_code() usage with this decoder is completely normal. The + * special behavior of lzma_code() applies to lzma_microlzma_encoder() only. + * + * \param strm Pointer to lzma_stream that is at least initialized + * with LZMA_STREAM_INIT. + * \param comp_size Compressed size of the MicroLZMA stream. + * The caller must somehow know this exactly. + * \param uncomp_size Uncompressed size of the MicroLZMA stream. + * If the exact uncompressed size isn't known, this + * can be set to a value that is at most as big as + * the exact uncompressed size would be, but then the + * next argument uncomp_size_is_exact must be false. + * \param uncomp_size_is_exact + * If true, uncomp_size must be exactly correct. + * This will improve error detection at the end of + * the stream. If the exact uncompressed size isn't + * known, this must be false. uncomp_size must still + * be at most as big as the exact uncompressed size + * is. Setting this to false when the exact size is + * known will work but error detection at the end of + * the stream will be weaker. + * \param dict_size LZMA dictionary size that was used when + * compressing the data. It is OK to use a bigger + * value too but liblzma will then allocate more + * memory than would actually be required and error + * detection will be slightly worse. (Note that with + * the implementation in XZ Embedded it doesn't + * affect the memory usage if one specifies bigger + * dictionary than actually required.) + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_microlzma_decoder( + lzma_stream *strm, uint64_t comp_size, + uint64_t uncomp_size, lzma_bool uncomp_size_is_exact, + uint32_t dict_size) lzma_nothrow; diff --git a/src/liblzma/api/lzma/delta.h b/src/liblzma/api/lzma/delta.h new file mode 100644 index 0000000000..5ebacef815 --- /dev/null +++ b/src/liblzma/api/lzma/delta.h @@ -0,0 +1,95 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/delta.h + * \brief Delta filter + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Filter ID + * + * Filter ID of the Delta filter. This is used as lzma_filter.id. + */ +#define LZMA_FILTER_DELTA LZMA_VLI_C(0x03) + + +/** + * \brief Type of the delta calculation + * + * Currently only byte-wise delta is supported. Other possible types could + * be, for example, delta of 16/32/64-bit little/big endian integers, but + * these are not currently planned since byte-wise delta is almost as good. + */ +typedef enum { + LZMA_DELTA_TYPE_BYTE +} lzma_delta_type; + + +/** + * \brief Options for the Delta filter + * + * These options are needed by both encoder and decoder. + */ +typedef struct { + /** For now, this must always be LZMA_DELTA_TYPE_BYTE. */ + lzma_delta_type type; + + /** + * \brief Delta distance + * + * With the only currently supported type, LZMA_DELTA_TYPE_BYTE, + * the distance is as bytes. + * + * Examples: + * - 16-bit stereo audio: distance = 4 bytes + * - 24-bit RGB image data: distance = 3 bytes + */ + uint32_t dist; + + /** + * \brief Minimum value for lzma_options_delta.dist. + */ +# define LZMA_DELTA_DIST_MIN 1 + + /** + * \brief Maximum value for lzma_options_delta.dist. + */ +# define LZMA_DELTA_DIST_MAX 256 + + /* + * Reserved space to allow possible future extensions without + * breaking the ABI. You should not touch these, because the names + * of these variables may change. These are and will never be used + * when type is LZMA_DELTA_TYPE_BYTE, so it is safe to leave these + * uninitialized. + */ + + /** \private Reserved member. */ + uint32_t reserved_int1; + + /** \private Reserved member. */ + uint32_t reserved_int2; + + /** \private Reserved member. */ + uint32_t reserved_int3; + + /** \private Reserved member. */ + uint32_t reserved_int4; + + /** \private Reserved member. */ + void *reserved_ptr1; + + /** \private Reserved member. */ + void *reserved_ptr2; + +} lzma_options_delta; diff --git a/src/liblzma/api/lzma/filter.h b/src/liblzma/api/lzma/filter.h new file mode 100644 index 0000000000..e86809c4e3 --- /dev/null +++ b/src/liblzma/api/lzma/filter.h @@ -0,0 +1,769 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/filter.h + * \brief Common filter related types and functions + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Maximum number of filters in a chain + * + * A filter chain can have 1-4 filters, of which three are allowed to change + * the size of the data. Usually only one or two filters are needed. + */ +#define LZMA_FILTERS_MAX 4 + + +/** + * \brief Filter options + * + * This structure is used to pass a Filter ID and a pointer to the filter's + * options to liblzma. A few functions work with a single lzma_filter + * structure, while most functions expect a filter chain. + * + * A filter chain is indicated with an array of lzma_filter structures. + * The array is terminated with .id = LZMA_VLI_UNKNOWN. Thus, the filter + * array must have LZMA_FILTERS_MAX + 1 elements (that is, five) to + * be able to hold any arbitrary filter chain. This is important when + * using lzma_block_header_decode() from block.h, because a filter array + * that is too small would make liblzma write past the end of the array. + */ +typedef struct { + /** + * \brief Filter ID + * + * Use constants whose name begin with 'LZMA_FILTER_' to specify + * different filters. In an array of lzma_filter structures, use + * LZMA_VLI_UNKNOWN to indicate end of filters. + * + * \note This is not an enum, because on some systems enums + * cannot be 64-bit. + */ + lzma_vli id; + + /** + * \brief Pointer to filter-specific options structure + * + * If the filter doesn't need options, set this to NULL. If id is + * set to LZMA_VLI_UNKNOWN, options is ignored, and thus + * doesn't need be initialized. + */ + void *options; + +} lzma_filter; + + +/** + * \brief Test if the given Filter ID is supported for encoding + * + * \param id Filter ID + * + * \return lzma_bool: + * - true if the Filter ID is supported for encoding by this + * liblzma build. + * - false otherwise. + */ +extern LZMA_API(lzma_bool) lzma_filter_encoder_is_supported(lzma_vli id) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Test if the given Filter ID is supported for decoding + * + * \param id Filter ID + * + * \return lzma_bool: + * - true if the Filter ID is supported for decoding by this + * liblzma build. + * - false otherwise. + */ +extern LZMA_API(lzma_bool) lzma_filter_decoder_is_supported(lzma_vli id) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Copy the filters array + * + * Copy the Filter IDs and filter-specific options from src to dest. + * Up to LZMA_FILTERS_MAX filters are copied, plus the terminating + * .id == LZMA_VLI_UNKNOWN. Thus, dest should have at least + * LZMA_FILTERS_MAX + 1 elements space unless the caller knows that + * src is smaller than that. + * + * Unless the filter-specific options is NULL, the Filter ID has to be + * supported by liblzma, because liblzma needs to know the size of every + * filter-specific options structure. The filter-specific options are not + * validated. If options is NULL, any unsupported Filter IDs are copied + * without returning an error. + * + * Old filter-specific options in dest are not freed, so dest doesn't + * need to be initialized by the caller in any way. + * + * If an error occurs, memory possibly already allocated by this function + * is always freed. liblzma versions older than 5.2.7 may modify the dest + * array and leave its contents in an undefined state if an error occurs. + * liblzma 5.2.7 and newer only modify the dest array when returning LZMA_OK. + * + * \param src Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * \param[out] dest Destination filter array + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_OPTIONS_ERROR: Unsupported Filter ID and its options + * is not NULL. + * - LZMA_PROG_ERROR: src or dest is NULL. + */ +extern LZMA_API(lzma_ret) lzma_filters_copy( + const lzma_filter *src, lzma_filter *dest, + const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Free the options in the array of lzma_filter structures + * + * This frees the filter chain options. The filters array itself is not freed. + * + * The filters array must have at most LZMA_FILTERS_MAX + 1 elements + * including the terminating element which must have .id = LZMA_VLI_UNKNOWN. + * For all elements before the terminating element: + * - options will be freed using the given lzma_allocator or, + * if allocator is NULL, using free(). + * - options will be set to NULL. + * - id will be set to LZMA_VLI_UNKNOWN. + * + * If filters is NULL, this does nothing. Again, this never frees the + * filters array itself. + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + */ +extern LZMA_API(void) lzma_filters_free( + lzma_filter *filters, const lzma_allocator *allocator) + lzma_nothrow; + + +/** + * \brief Calculate approximate memory requirements for raw encoder + * + * This function can be used to calculate the memory requirements for + * Block and Stream encoders too because Block and Stream encoders don't + * need significantly more memory than raw encoder. + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Number of bytes of memory required for the given + * filter chain when encoding or UINT64_MAX on error. + */ +extern LZMA_API(uint64_t) lzma_raw_encoder_memusage(const lzma_filter *filters) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Calculate approximate memory requirements for raw decoder + * + * This function can be used to calculate the memory requirements for + * Block and Stream decoders too because Block and Stream decoders don't + * need significantly more memory than raw decoder. + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Number of bytes of memory required for the given + * filter chain when decoding or UINT64_MAX on error. + */ +extern LZMA_API(uint64_t) lzma_raw_decoder_memusage(const lzma_filter *filters) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Initialize raw encoder + * + * This function may be useful when implementing custom file formats. + * + * The 'action' with lzma_code() can be LZMA_RUN, LZMA_SYNC_FLUSH (if the + * filter chain supports it), or LZMA_FINISH. + * + * \param strm Pointer to lzma_stream that is at least + * initialized with LZMA_STREAM_INIT. + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_raw_encoder( + lzma_stream *strm, const lzma_filter *filters) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize raw decoder + * + * The initialization of raw decoder goes similarly to raw encoder. + * + * The 'action' with lzma_code() can be LZMA_RUN or LZMA_FINISH. Using + * LZMA_FINISH is not required, it is supported just for convenience. + * + * \param strm Pointer to lzma_stream that is at least + * initialized with LZMA_STREAM_INIT. + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_raw_decoder( + lzma_stream *strm, const lzma_filter *filters) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Update the filter chain in the encoder + * + * This function may be called after lzma_code() has returned LZMA_STREAM_END + * when LZMA_FULL_BARRIER, LZMA_FULL_FLUSH, or LZMA_SYNC_FLUSH was used: + * + * - After LZMA_FULL_BARRIER or LZMA_FULL_FLUSH: Single-threaded .xz Stream + * encoder (lzma_stream_encoder()) and (since liblzma 5.4.0) multi-threaded + * Stream encoder (lzma_stream_encoder_mt()) allow setting a new filter + * chain to be used for the next Block(s). + * + * - After LZMA_SYNC_FLUSH: Raw encoder (lzma_raw_encoder()), + * Block encoder (lzma_block_encoder()), and single-threaded .xz Stream + * encoder (lzma_stream_encoder()) allow changing certain filter-specific + * options in the middle of encoding. The actual filters in the chain + * (Filter IDs) must not be changed! Currently only the lc, lp, and pb + * options of LZMA2 (not LZMA1) can be changed this way. + * + * - In the future some filters might allow changing some of their options + * without any barrier or flushing but currently such filters don't exist. + * + * This function may also be called when no data has been compressed yet + * although this is rarely useful. In that case, this function will behave + * as if LZMA_FULL_FLUSH (Stream encoders) or LZMA_SYNC_FLUSH (Raw or Block + * encoder) had been used right before calling this function. + * + * \param strm Pointer to lzma_stream that is at least + * initialized with LZMA_STREAM_INIT. + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_MEMLIMIT_ERROR + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_filters_update( + lzma_stream *strm, const lzma_filter *filters) lzma_nothrow; + + +/** + * \brief Single-call raw encoder + * + * \note There is no function to calculate how big output buffer + * would surely be big enough. (lzma_stream_buffer_bound() + * works only for lzma_stream_buffer_encode(); raw encoder + * won't necessarily meet that bound.) + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_size Size of the input buffer + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_BUF_ERROR: Not enough output buffer space. + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_raw_buffer_encode( + const lzma_filter *filters, const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, uint8_t *out, + size_t *out_pos, size_t out_size) lzma_nothrow; + + +/** + * \brief Single-call raw decoder + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_pos The next byte will be read from in[*in_pos]. + * *in_pos is updated only if decoding succeeds. + * \param in_size Size of the input buffer; the first byte that + * won't be read is in[in_size]. + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. + * - LZMA_BUF_ERROR: Not enough output buffer space. + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_raw_buffer_decode( + const lzma_filter *filters, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) lzma_nothrow; + + +/** + * \brief Get the size of the Filter Properties field + * + * This function may be useful when implementing custom file formats + * using the raw encoder and decoder. + * + * \note This function validates the Filter ID, but does not + * necessarily validate the options. Thus, it is possible + * that this returns LZMA_OK while the following call to + * lzma_properties_encode() returns LZMA_OPTIONS_ERROR. + * + * \param[out] size Pointer to uint32_t to hold the size of the properties + * \param filter Filter ID and options (the size of the properties may + * vary depending on the options) + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_OPTIONS_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_properties_size( + uint32_t *size, const lzma_filter *filter) lzma_nothrow; + + +/** + * \brief Encode the Filter Properties field + * + * \note Even this function won't validate more options than actually + * necessary. Thus, it is possible that encoding the properties + * succeeds but using the same options to initialize the encoder + * will fail. + * + * \note If lzma_properties_size() indicated that the size + * of the Filter Properties field is zero, calling + * lzma_properties_encode() is not required, but it + * won't do any harm either. + * + * \param filter Filter ID and options + * \param[out] props Buffer to hold the encoded options. The size of + * the buffer must have been already determined with + * lzma_properties_size(). + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_properties_encode( + const lzma_filter *filter, uint8_t *props) lzma_nothrow; + + +/** + * \brief Decode the Filter Properties field + * + * \param filter filter->id must have been set to the correct + * Filter ID. filter->options doesn't need to be + * initialized (it's not freed by this function). The + * decoded options will be stored in filter->options; + * it's application's responsibility to free it when + * appropriate. filter->options is set to NULL if + * there are no properties or if an error occurs. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * and in case of an error, also free(). + * \param props Input buffer containing the properties. + * \param props_size Size of the properties. This must be the exact + * size; giving too much or too little input will + * return LZMA_OPTIONS_ERROR. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + */ +extern LZMA_API(lzma_ret) lzma_properties_decode( + lzma_filter *filter, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size) lzma_nothrow; + + +/** + * \brief Calculate encoded size of a Filter Flags field + * + * Knowing the size of Filter Flags is useful to know when allocating + * memory to hold the encoded Filter Flags. + * + * \note If you need to calculate size of List of Filter Flags, + * you need to loop over every lzma_filter entry. + * + * \param[out] size Pointer to integer to hold the calculated size + * \param filter Filter ID and associated options whose encoded + * size is to be calculated + * + * \return Possible lzma_ret values: + * - LZMA_OK: *size set successfully. Note that this doesn't + * guarantee that filter->options is valid, thus + * lzma_filter_flags_encode() may still fail. + * - LZMA_OPTIONS_ERROR: Unknown Filter ID or unsupported options. + * - LZMA_PROG_ERROR: Invalid options + */ +extern LZMA_API(lzma_ret) lzma_filter_flags_size( + uint32_t *size, const lzma_filter *filter) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Encode Filter Flags into given buffer + * + * In contrast to some functions, this doesn't allocate the needed buffer. + * This is due to how this function is used internally by liblzma. + * + * \param filter Filter ID and options to be encoded + * \param[out] out Beginning of the output buffer + * \param[out] out_pos out[*out_pos] is the next write position. This + * is updated by the encoder. + * \param out_size out[out_size] is the first byte to not write. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_OPTIONS_ERROR: Invalid or unsupported options. + * - LZMA_PROG_ERROR: Invalid options or not enough output + * buffer space (you should have checked it with + * lzma_filter_flags_size()). + */ +extern LZMA_API(lzma_ret) lzma_filter_flags_encode(const lzma_filter *filter, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode Filter Flags from given buffer + * + * The decoded result is stored into *filter. The old value of + * filter->options is not free()d. If anything other than LZMA_OK + * is returned, filter->options is set to NULL. + * + * \param[out] filter Destination filter. The decoded Filter ID will + * be stored in filter->id. If options are needed + * they will be allocated and the pointer will be + * stored in filter->options. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param[out] in_pos The next byte will be read from in[*in_pos]. + * *in_pos is updated only if decoding succeeds. + * \param in_size Size of the input buffer; the first byte that + * won't be read is in[in_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_filter_flags_decode( + lzma_filter *filter, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/*********** + * Strings * + ***********/ + +/** + * \brief Allow or show all filters + * + * By default only the filters supported in the .xz format are accept by + * lzma_str_to_filters() or shown by lzma_str_list_filters(). + */ +#define LZMA_STR_ALL_FILTERS UINT32_C(0x01) + + +/** + * \brief Do not validate the filter chain in lzma_str_to_filters() + * + * By default lzma_str_to_filters() can return an error if the filter chain + * as a whole isn't usable in the .xz format or in the raw encoder or decoder. + * With this flag, this validation is skipped. This flag doesn't affect the + * handling of the individual filter options. To allow non-.xz filters also + * LZMA_STR_ALL_FILTERS is needed. + */ +#define LZMA_STR_NO_VALIDATION UINT32_C(0x02) + + +/** + * \brief Stringify encoder options + * + * Show the filter-specific options that the encoder will use. + * This may be useful for verbose diagnostic messages. + * + * Note that if options were decoded from .xz headers then the encoder options + * may be undefined. This flag shouldn't be used in such a situation. + */ +#define LZMA_STR_ENCODER UINT32_C(0x10) + + +/** + * \brief Stringify decoder options + * + * Show the filter-specific options that the decoder will use. + * This may be useful for showing what filter options were decoded + * from file headers. + */ +#define LZMA_STR_DECODER UINT32_C(0x20) + + +/** + * \brief Produce xz-compatible getopt_long() syntax + * + * That is, "delta:dist=2 lzma2:dict=4MiB,pb=1,lp=1" becomes + * "--delta=dist=2 --lzma2=dict=4MiB,pb=1,lp=1". + * + * This syntax is compatible with xz 5.0.0 as long as the filters and + * their options are supported too. + */ +#define LZMA_STR_GETOPT_LONG UINT32_C(0x40) + + +/** + * \brief Use two dashes "--" instead of a space to separate filters + * + * That is, "delta:dist=2 lzma2:pb=1,lp=1" becomes + * "delta:dist=2--lzma2:pb=1,lp=1". This looks slightly odd but this + * kind of strings should be usable on the command line without quoting. + * However, it is possible that future versions with new filter options + * might produce strings that require shell quoting anyway as the exact + * set of possible characters isn't frozen for now. + * + * It is guaranteed that the single quote (') will never be used in + * filter chain strings (even if LZMA_STR_NO_SPACES isn't used). + */ +#define LZMA_STR_NO_SPACES UINT32_C(0x80) + + +/** + * \brief Convert a string to a filter chain + * + * This tries to make it easier to write applications that allow users + * to set custom compression options. This only handles the filter + * configuration (including presets) but not the number of threads, + * block size, check type, or memory limits. + * + * The input string can be either a preset or a filter chain. Presets + * begin with a digit 0-9 and may be followed by zero or more flags + * which are lower-case letters. Currently only "e" is supported, matching + * LZMA_PRESET_EXTREME. For partial xz command line syntax compatibility, + * a preset string may start with a single dash "-". + * + * A filter chain consists of one or more "filtername:opt1=value1,opt2=value2" + * strings separated by one or more spaces. Leading and trailing spaces are + * ignored. All names and values must be lower-case. Extra commas in the + * option list are ignored. The order of filters is significant: when + * encoding, the uncompressed input data goes to the leftmost filter first. + * Normally "lzma2" is the last filter in the chain. + * + * If one wishes to avoid spaces, for example, to avoid shell quoting, + * it is possible to use two dashes "--" instead of spaces to separate + * the filters. + * + * For xz command line compatibility, each filter may be prefixed with + * two dashes "--" and the colon ":" separating the filter name from + * the options may be replaced with an equals sign "=". + * + * By default, only filters that can be used in the .xz format are accepted. + * To allow all filters (LZMA1) use the flag LZMA_STR_ALL_FILTERS. + * + * By default, very basic validation is done for the filter chain as a whole, + * for example, that LZMA2 is only used as the last filter in the chain. + * The validation isn't perfect though and it's possible that this function + * succeeds but using the filter chain for encoding or decoding will still + * result in LZMA_OPTIONS_ERROR. To disable this validation, use the flag + * LZMA_STR_NO_VALIDATION. + * + * The available filter names and their options are available via + * lzma_str_list_filters(). See the xz man page for the description + * of filter names and options. + * + * For command line applications, below is an example how an error message + * can be displayed. Note the use of an empty string for the field width. + * If "^" was used there it would create an off-by-one error except at + * the very beginning of the line. + * + * \code{.c} + * const char *str = ...; // From user + * lzma_filter filters[LZMA_FILTERS_MAX + 1]; + * int pos; + * const char *msg = lzma_str_to_filters(str, &pos, filters, 0, NULL); + * if (msg != NULL) { + * printf("%s: Error in XZ compression options:\n", argv[0]); + * printf("%s: %s\n", argv[0], str); + * printf("%s: %*s^\n", argv[0], errpos, ""); + * printf("%s: %s\n", argv[0], msg); + * } + * \endcode + * + * \param str User-supplied string describing a preset or + * a filter chain. If a default value is needed and + * you don't know what would be good, use "6" since + * that is the default preset in xz too. + * \param[out] error_pos If this isn't NULL, this value will be set on + * both success and on all errors. This tells the + * location of the error in the string. This is + * an int to make it straightforward to use this + * as printf() field width. The value is guaranteed + * to be in the range [0, INT_MAX] even if strlen(str) + * somehow was greater than INT_MAX. + * \param[out] filters An array of lzma_filter structures. There must + * be LZMA_FILTERS_MAX + 1 (that is, five) elements + * in the array. The old contents are ignored so it + * doesn't need to be initialized. This array is + * modified only if this function returns NULL. + * Once the allocated filter options are no longer + * needed, lzma_filters_free() can be used to free the + * options (it doesn't free the filters array itself). + * \param flags Bitwise-or of zero or more of the flags + * LZMA_STR_ALL_FILTERS and LZMA_STR_NO_VALIDATION. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return On success, NULL is returned. On error, a statically-allocated + * error message is returned which together with the error_pos + * should give some idea what is wrong. + */ +extern LZMA_API(const char *) lzma_str_to_filters( + const char *str, int *error_pos, lzma_filter *filters, + uint32_t flags, const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Convert a filter chain to a string + * + * Use cases: + * + * - Verbose output showing the full encoder options to the user + * (use LZMA_STR_ENCODER in flags) + * + * - Showing the filters and options that are required to decode a file + * (use LZMA_STR_DECODER in flags) + * + * - Showing the filter names without any options in informational messages + * where the technical details aren't important (no flags). In this case + * the .options in the filters array are ignored and may be NULL even if + * a filter has a mandatory options structure. + * + * Note that even if the filter chain was specified using a preset, + * the resulting filter chain isn't reversed to a preset. So if you + * specify "6" to lzma_str_to_filters() then lzma_str_from_filters() + * will produce a string containing "lzma2". + * + * \param[out] str On success *str will be set to point to an + * allocated string describing the given filter + * chain. Old value is ignored. On error *str is + * always set to NULL. + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * \param flags Bitwise-or of zero or more of the flags + * LZMA_STR_ENCODER, LZMA_STR_DECODER, + * LZMA_STR_GETOPT_LONG, and LZMA_STR_NO_SPACES. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_OPTIONS_ERROR: Empty filter chain + * (filters[0].id == LZMA_VLI_UNKNOWN) or the filter chain + * includes a Filter ID that is not supported by this function. + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_str_from_filters( + char **str, const lzma_filter *filters, uint32_t flags, + const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief List available filters and/or their options (for help message) + * + * If a filter_id is given then only one line is created which contains the + * filter name. If LZMA_STR_ENCODER or LZMA_STR_DECODER is used then the + * options read by the encoder or decoder are printed on the same line. + * + * If filter_id is LZMA_VLI_UNKNOWN then all supported .xz-compatible filters + * are listed: + * + * - If neither LZMA_STR_ENCODER nor LZMA_STR_DECODER is used then + * the supported filter names are listed on a single line separated + * by spaces. + * + * - If LZMA_STR_ENCODER or LZMA_STR_DECODER is used then filters and + * the supported options are listed one filter per line. There won't + * be a newline after the last filter. + * + * - If LZMA_STR_ALL_FILTERS is used then the list will include also + * those filters that cannot be used in the .xz format (LZMA1). + * + * \param str On success *str will be set to point to an + * allocated string listing the filters and options. + * Old value is ignored. On error *str is always set + * to NULL. + * \param filter_id Filter ID or LZMA_VLI_UNKNOWN. + * \param flags Bitwise-or of zero or more of the flags + * LZMA_STR_ALL_FILTERS, LZMA_STR_ENCODER, + * LZMA_STR_DECODER, and LZMA_STR_GETOPT_LONG. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_OPTIONS_ERROR: Unsupported filter_id or flags + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_str_list_filters( + char **str, lzma_vli filter_id, uint32_t flags, + const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; diff --git a/src/liblzma/api/lzma/hardware.h b/src/liblzma/api/lzma/hardware.h new file mode 100644 index 0000000000..7a1a84fccc --- /dev/null +++ b/src/liblzma/api/lzma/hardware.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/hardware.h + * \brief Hardware information + * \note Never include this file directly. Use instead. + * + * Since liblzma can consume a lot of system resources, it also provides + * ways to limit the resource usage. Applications linking against liblzma + * need to do the actual decisions how much resources to let liblzma to use. + * To ease making these decisions, liblzma provides functions to find out + * the relevant capabilities of the underlying hardware. Currently there + * is only a function to find out the amount of RAM, but in the future there + * will be also a function to detect how many concurrent threads the system + * can run. + * + * \note On some operating systems, these function may temporarily + * load a shared library or open file descriptor(s) to find out + * the requested hardware information. Unless the application + * assumes that specific file descriptors are not touched by + * other threads, this should have no effect on thread safety. + * Possible operations involving file descriptors will restart + * the syscalls if they return EINTR. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Get the total amount of physical memory (RAM) in bytes + * + * This function may be useful when determining a reasonable memory + * usage limit for decompressing or how much memory it is OK to use + * for compressing. + * + * \return On success, the total amount of physical memory in bytes + * is returned. If the amount of RAM cannot be determined, + * zero is returned. This can happen if an error occurs + * or if there is no code in liblzma to detect the amount + * of RAM on the specific operating system. + */ +extern LZMA_API(uint64_t) lzma_physmem(void) lzma_nothrow; + + +/** + * \brief Get the number of processor cores or threads + * + * This function may be useful when determining how many threads to use. + * If the hardware supports more than one thread per CPU core, the number + * of hardware threads is returned if that information is available. + * + * \return On success, the number of available CPU threads or cores is + * returned. If this information isn't available or an error + * occurs, zero is returned. + */ +extern LZMA_API(uint32_t) lzma_cputhreads(void) lzma_nothrow; diff --git a/src/liblzma/api/lzma/index.h b/src/liblzma/api/lzma/index.h new file mode 100644 index 0000000000..b17025e3d9 --- /dev/null +++ b/src/liblzma/api/lzma/index.h @@ -0,0 +1,882 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/index.h + * \brief Handling of .xz Index and related information + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Opaque data type to hold the Index(es) and other information + * + * lzma_index often holds just one .xz Index and possibly the Stream Flags + * of the same Stream and size of the Stream Padding field. However, + * multiple lzma_indexes can be concatenated with lzma_index_cat() and then + * there may be information about multiple Streams in the same lzma_index. + * + * Notes about thread safety: Only one thread may modify lzma_index at + * a time. All functions that take non-const pointer to lzma_index + * modify it. As long as no thread is modifying the lzma_index, getting + * information from the same lzma_index can be done from multiple threads + * at the same time with functions that take a const pointer to + * lzma_index or use lzma_index_iter. The same iterator must be used + * only by one thread at a time, of course, but there can be as many + * iterators for the same lzma_index as needed. + */ +typedef struct lzma_index_s lzma_index; + + +/** + * \brief Iterator to get information about Blocks and Streams + */ +typedef struct { + struct { + /** + * \brief Pointer to Stream Flags + * + * This is NULL if Stream Flags have not been set for + * this Stream with lzma_index_stream_flags(). + */ + const lzma_stream_flags *flags; + + /** \private Reserved member. */ + const void *reserved_ptr1; + + /** \private Reserved member. */ + const void *reserved_ptr2; + + /** \private Reserved member. */ + const void *reserved_ptr3; + + /** + * \brief Stream number in the lzma_index + * + * The first Stream is 1. + */ + lzma_vli number; + + /** + * \brief Number of Blocks in the Stream + * + * If this is zero, the block structure below has + * undefined values. + */ + lzma_vli block_count; + + /** + * \brief Compressed start offset of this Stream + * + * The offset is relative to the beginning of the lzma_index + * (i.e. usually the beginning of the .xz file). + */ + lzma_vli compressed_offset; + + /** + * \brief Uncompressed start offset of this Stream + * + * The offset is relative to the beginning of the lzma_index + * (i.e. usually the beginning of the .xz file). + */ + lzma_vli uncompressed_offset; + + /** + * \brief Compressed size of this Stream + * + * This includes all headers except the possible + * Stream Padding after this Stream. + */ + lzma_vli compressed_size; + + /** + * \brief Uncompressed size of this Stream + */ + lzma_vli uncompressed_size; + + /** + * \brief Size of Stream Padding after this Stream + * + * If it hasn't been set with lzma_index_stream_padding(), + * this defaults to zero. Stream Padding is always + * a multiple of four bytes. + */ + lzma_vli padding; + + + /** \private Reserved member. */ + lzma_vli reserved_vli1; + + /** \private Reserved member. */ + lzma_vli reserved_vli2; + + /** \private Reserved member. */ + lzma_vli reserved_vli3; + + /** \private Reserved member. */ + lzma_vli reserved_vli4; + } stream; + + struct { + /** + * \brief Block number in the file + * + * The first Block is 1. + */ + lzma_vli number_in_file; + + /** + * \brief Compressed start offset of this Block + * + * This offset is relative to the beginning of the + * lzma_index (i.e. usually the beginning of the .xz file). + * Normally this is where you should seek in the .xz file + * to start decompressing this Block. + */ + lzma_vli compressed_file_offset; + + /** + * \brief Uncompressed start offset of this Block + * + * This offset is relative to the beginning of the lzma_index + * (i.e. usually the beginning of the .xz file). + * + * When doing random-access reading, it is possible that + * the target offset is not exactly at Block boundary. One + * will need to compare the target offset against + * uncompressed_file_offset or uncompressed_stream_offset, + * and possibly decode and throw away some amount of data + * before reaching the target offset. + */ + lzma_vli uncompressed_file_offset; + + /** + * \brief Block number in this Stream + * + * The first Block is 1. + */ + lzma_vli number_in_stream; + + /** + * \brief Compressed start offset of this Block + * + * This offset is relative to the beginning of the Stream + * containing this Block. + */ + lzma_vli compressed_stream_offset; + + /** + * \brief Uncompressed start offset of this Block + * + * This offset is relative to the beginning of the Stream + * containing this Block. + */ + lzma_vli uncompressed_stream_offset; + + /** + * \brief Uncompressed size of this Block + * + * You should pass this to the Block decoder if you will + * decode this Block. It will allow the Block decoder to + * validate the uncompressed size. + */ + lzma_vli uncompressed_size; + + /** + * \brief Unpadded size of this Block + * + * You should pass this to the Block decoder if you will + * decode this Block. It will allow the Block decoder to + * validate the unpadded size. + */ + lzma_vli unpadded_size; + + /** + * \brief Total compressed size + * + * This includes all headers and padding in this Block. + * This is useful if you need to know how many bytes + * the Block decoder will actually read. + */ + lzma_vli total_size; + + /** \private Reserved member. */ + lzma_vli reserved_vli1; + + /** \private Reserved member. */ + lzma_vli reserved_vli2; + + /** \private Reserved member. */ + lzma_vli reserved_vli3; + + /** \private Reserved member. */ + lzma_vli reserved_vli4; + + /** \private Reserved member. */ + const void *reserved_ptr1; + + /** \private Reserved member. */ + const void *reserved_ptr2; + + /** \private Reserved member. */ + const void *reserved_ptr3; + + /** \private Reserved member. */ + const void *reserved_ptr4; + } block; + + /** + * \private Internal data + * + * Internal data which is used to store the state of the iterator. + * The exact format may vary between liblzma versions, so don't + * touch these in any way. + */ + union { + /** \private Internal member. */ + const void *p; + + /** \private Internal member. */ + size_t s; + + /** \private Internal member. */ + lzma_vli v; + } internal[6]; +} lzma_index_iter; + + +/** + * \brief Operation mode for lzma_index_iter_next() + */ +typedef enum { + LZMA_INDEX_ITER_ANY = 0, + /**< + * \brief Get the next Block or Stream + * + * Go to the next Block if the current Stream has at least + * one Block left. Otherwise go to the next Stream even if + * it has no Blocks. If the Stream has no Blocks + * (lzma_index_iter.stream.block_count == 0), + * lzma_index_iter.block will have undefined values. + */ + + LZMA_INDEX_ITER_STREAM = 1, + /**< + * \brief Get the next Stream + * + * Go to the next Stream even if the current Stream has + * unread Blocks left. If the next Stream has at least one + * Block, the iterator will point to the first Block. + * If there are no Blocks, lzma_index_iter.block will have + * undefined values. + */ + + LZMA_INDEX_ITER_BLOCK = 2, + /**< + * \brief Get the next Block + * + * Go to the next Block if the current Stream has at least + * one Block left. If the current Stream has no Blocks left, + * the next Stream with at least one Block is located and + * the iterator will be made to point to the first Block of + * that Stream. + */ + + LZMA_INDEX_ITER_NONEMPTY_BLOCK = 3 + /**< + * \brief Get the next non-empty Block + * + * This is like LZMA_INDEX_ITER_BLOCK except that it will + * skip Blocks whose Uncompressed Size is zero. + */ + +} lzma_index_iter_mode; + + +/** + * \brief Mask for return value from lzma_index_checks() for check none + * + * \note This and the other CHECK_MASK macros were added in 5.5.1alpha. + */ +#define LZMA_INDEX_CHECK_MASK_NONE (UINT32_C(1) << LZMA_CHECK_NONE) + +/** + * \brief Mask for return value from lzma_index_checks() for check CRC32 + */ +#define LZMA_INDEX_CHECK_MASK_CRC32 (UINT32_C(1) << LZMA_CHECK_CRC32) + +/** + * \brief Mask for return value from lzma_index_checks() for check CRC64 + */ +#define LZMA_INDEX_CHECK_MASK_CRC64 (UINT32_C(1) << LZMA_CHECK_CRC64) + +/** + * \brief Mask for return value from lzma_index_checks() for check SHA256 + */ +#define LZMA_INDEX_CHECK_MASK_SHA256 (UINT32_C(1) << LZMA_CHECK_SHA256) + +/** + * \brief Calculate memory usage of lzma_index + * + * On disk, the size of the Index field depends on both the number of Records + * stored and the size of the Records (due to variable-length integer + * encoding). When the Index is kept in lzma_index structure, the memory usage + * depends only on the number of Records/Blocks stored in the Index(es), and + * in case of concatenated lzma_indexes, the number of Streams. The size in + * RAM is almost always significantly bigger than in the encoded form on disk. + * + * This function calculates an approximate amount of memory needed to hold + * the given number of Streams and Blocks in lzma_index structure. This + * value may vary between CPU architectures and also between liblzma versions + * if the internal implementation is modified. + * + * \param streams Number of Streams + * \param blocks Number of Blocks + * + * \return Approximate memory in bytes needed in a lzma_index structure. + */ +extern LZMA_API(uint64_t) lzma_index_memusage( + lzma_vli streams, lzma_vli blocks) lzma_nothrow; + + +/** + * \brief Calculate the memory usage of an existing lzma_index + * + * This is a shorthand for lzma_index_memusage(lzma_index_stream_count(i), + * lzma_index_block_count(i)). + * + * \param i Pointer to lzma_index structure + * + * \return Approximate memory in bytes used by the lzma_index structure. + */ +extern LZMA_API(uint64_t) lzma_index_memused(const lzma_index *i) + lzma_nothrow; + + +/** + * \brief Allocate and initialize a new lzma_index structure + * + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return On success, a pointer to an empty initialized lzma_index is + * returned. If allocation fails, NULL is returned. + */ +extern LZMA_API(lzma_index *) lzma_index_init(const lzma_allocator *allocator) + lzma_nothrow; + + +/** + * \brief Deallocate lzma_index + * + * If i is NULL, this does nothing. + * + * \param i Pointer to lzma_index structure to deallocate + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + */ +extern LZMA_API(void) lzma_index_end( + lzma_index *i, const lzma_allocator *allocator) lzma_nothrow; + + +/** + * \brief Add a new Block to lzma_index + * + * \param i Pointer to a lzma_index structure + * \param allocator lzma_allocator for custom allocator + * functions. Set to NULL to use malloc() + * and free(). + * \param unpadded_size Unpadded Size of a Block. This can be + * calculated with lzma_block_unpadded_size() + * after encoding or decoding the Block. + * \param uncompressed_size Uncompressed Size of a Block. This can be + * taken directly from lzma_block structure + * after encoding or decoding the Block. + * + * Appending a new Block does not invalidate iterators. For example, + * if an iterator was pointing to the end of the lzma_index, after + * lzma_index_append() it is possible to read the next Block with + * an existing iterator. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_DATA_ERROR: Compressed or uncompressed size of the + * Stream or size of the Index field would grow too big. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_append( + lzma_index *i, const lzma_allocator *allocator, + lzma_vli unpadded_size, lzma_vli uncompressed_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Set the Stream Flags + * + * Set the Stream Flags of the last (and typically the only) Stream + * in lzma_index. This can be useful when reading information from the + * lzma_index, because to decode Blocks, knowing the integrity check type + * is needed. + * + * \param i Pointer to lzma_index structure + * \param stream_flags Pointer to lzma_stream_flags structure. This + * is copied into the internal preallocated + * structure, so the caller doesn't need to keep + * the flags' data available after calling this + * function. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_OPTIONS_ERROR: Unsupported stream_flags->version. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_stream_flags( + lzma_index *i, const lzma_stream_flags *stream_flags) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Get the types of integrity Checks + * + * If lzma_index_stream_flags() is used to set the Stream Flags for + * every Stream, lzma_index_checks() can be used to get a bitmask to + * indicate which Check types have been used. It can be useful e.g. if + * showing the Check types to the user. + * + * The bitmask is 1 << check_id, e.g. CRC32 is 1 << 1 and SHA-256 is 1 << 10. + * These masks are defined for convenience as LZMA_INDEX_CHECK_MASK_XXX + * + * \param i Pointer to lzma_index structure + * + * \return Bitmask indicating which Check types are used in the lzma_index + */ +extern LZMA_API(uint32_t) lzma_index_checks(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Set the amount of Stream Padding + * + * Set the amount of Stream Padding of the last (and typically the only) + * Stream in the lzma_index. This is needed when planning to do random-access + * reading within multiple concatenated Streams. + * + * By default, the amount of Stream Padding is assumed to be zero bytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_DATA_ERROR: The file size would grow too big. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_stream_padding( + lzma_index *i, lzma_vli stream_padding) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Get the number of Streams + * + * \param i Pointer to lzma_index structure + * + * \return Number of Streams in the lzma_index + */ +extern LZMA_API(lzma_vli) lzma_index_stream_count(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the number of Blocks + * + * This returns the total number of Blocks in lzma_index. To get number + * of Blocks in individual Streams, use lzma_index_iter. + * + * \param i Pointer to lzma_index structure + * + * \return Number of blocks in the lzma_index + */ +extern LZMA_API(lzma_vli) lzma_index_block_count(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the size of the Index field as bytes + * + * This is needed to verify the Backward Size field in the Stream Footer. + * + * \param i Pointer to lzma_index structure + * + * \return Size in bytes of the Index + */ +extern LZMA_API(lzma_vli) lzma_index_size(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the total size of the Stream + * + * If multiple lzma_indexes have been combined, this works as if the Blocks + * were in a single Stream. This is useful if you are going to combine + * Blocks from multiple Streams into a single new Stream. + * + * \param i Pointer to lzma_index structure + * + * \return Size in bytes of the Stream (if all Blocks are combined + * into one Stream). + */ +extern LZMA_API(lzma_vli) lzma_index_stream_size(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the total size of the Blocks + * + * This doesn't include the Stream Header, Stream Footer, Stream Padding, + * or Index fields. + * + * \param i Pointer to lzma_index structure + * + * \return Size in bytes of all Blocks in the Stream(s) + */ +extern LZMA_API(lzma_vli) lzma_index_total_size(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the total size of the file + * + * When no lzma_indexes have been combined with lzma_index_cat() and there is + * no Stream Padding, this function is identical to lzma_index_stream_size(). + * If multiple lzma_indexes have been combined, this includes also the headers + * of each separate Stream and the possible Stream Padding fields. + * + * \param i Pointer to lzma_index structure + * + * \return Total size of the .xz file in bytes + */ +extern LZMA_API(lzma_vli) lzma_index_file_size(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Get the uncompressed size of the file + * + * \param i Pointer to lzma_index structure + * + * \return Size in bytes of the uncompressed data in the file + */ +extern LZMA_API(lzma_vli) lzma_index_uncompressed_size(const lzma_index *i) + lzma_nothrow lzma_attr_pure; + + +/** + * \brief Initialize an iterator + * + * This function associates the iterator with the given lzma_index, and calls + * lzma_index_iter_rewind() on the iterator. + * + * This function doesn't allocate any memory, thus there is no + * lzma_index_iter_end(). The iterator is valid as long as the + * associated lzma_index is valid, that is, until lzma_index_end() or + * using it as source in lzma_index_cat(). Specifically, lzma_index doesn't + * become invalid if new Blocks are added to it with lzma_index_append() or + * if it is used as the destination in lzma_index_cat(). + * + * It is safe to make copies of an initialized lzma_index_iter, for example, + * to easily restart reading at some particular position. + * + * \param iter Pointer to a lzma_index_iter structure + * \param i lzma_index to which the iterator will be associated + */ +extern LZMA_API(void) lzma_index_iter_init( + lzma_index_iter *iter, const lzma_index *i) lzma_nothrow; + + +/** + * \brief Rewind the iterator + * + * Rewind the iterator so that next call to lzma_index_iter_next() will + * return the first Block or Stream. + * + * \param iter Pointer to a lzma_index_iter structure + */ +extern LZMA_API(void) lzma_index_iter_rewind(lzma_index_iter *iter) + lzma_nothrow; + + +/** + * \brief Get the next Block or Stream + * + * \param iter Iterator initialized with lzma_index_iter_init() + * \param mode Specify what kind of information the caller wants + * to get. See lzma_index_iter_mode for details. + * + * \return lzma_bool: + * - true if no Block or Stream matching the mode is found. + * *iter is not updated (failure). + * - false if the next Block or Stream matching the mode was + * found. *iter is updated (success). + */ +extern LZMA_API(lzma_bool) lzma_index_iter_next( + lzma_index_iter *iter, lzma_index_iter_mode mode) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Locate a Block + * + * If it is possible to seek in the .xz file, it is possible to parse + * the Index field(s) and use lzma_index_iter_locate() to do random-access + * reading with granularity of Block size. + * + * If the target is smaller than the uncompressed size of the Stream (can be + * checked with lzma_index_uncompressed_size()): + * - Information about the Stream and Block containing the requested + * uncompressed offset is stored into *iter. + * - Internal state of the iterator is adjusted so that + * lzma_index_iter_next() can be used to read subsequent Blocks or Streams. + * + * If the target is greater than the uncompressed size of the Stream, *iter + * is not modified. + * + * \param iter Iterator that was earlier initialized with + * lzma_index_iter_init(). + * \param target Uncompressed target offset which the caller would + * like to locate from the Stream + * + * \return lzma_bool: + * - true if the target is greater than or equal to the + * uncompressed size of the Stream (failure) + * - false if the target is smaller than the uncompressed size + * of the Stream (success) + */ +extern LZMA_API(lzma_bool) lzma_index_iter_locate( + lzma_index_iter *iter, lzma_vli target) lzma_nothrow; + + +/** + * \brief Concatenate lzma_indexes + * + * Concatenating lzma_indexes is useful when doing random-access reading in + * multi-Stream .xz file, or when combining multiple Streams into single + * Stream. + * + * \param[out] dest lzma_index after which src is appended + * \param src lzma_index to be appended after dest. If this + * function succeeds, the memory allocated for src + * is freed or moved to be part of dest, and all + * iterators pointing to src will become invalid. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return Possible lzma_ret values: + * - LZMA_OK: lzma_indexes were concatenated successfully. + * src is now a dangling pointer. + * - LZMA_DATA_ERROR: *dest would grow too big. + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_cat(lzma_index *dest, lzma_index *src, + const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Duplicate lzma_index + * + * \param i Pointer to lzma_index structure to be duplicated + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return A copy of the lzma_index, or NULL if memory allocation failed. + */ +extern LZMA_API(lzma_index *) lzma_index_dup( + const lzma_index *i, const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize .xz Index encoder + * + * \param strm Pointer to properly prepared lzma_stream + * \param i Pointer to lzma_index which should be encoded. + * + * The valid 'action' values for lzma_code() are LZMA_RUN and LZMA_FINISH. + * It is enough to use only one of them (you can choose freely). + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization succeeded, continue with lzma_code(). + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_encoder( + lzma_stream *strm, const lzma_index *i) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Initialize .xz Index decoder + * + * \param strm Pointer to properly prepared lzma_stream + * \param[out] i The decoded Index will be made available via + * this pointer. Initially this function will + * set *i to NULL (the old value is ignored). If + * decoding succeeds (lzma_code() returns + * LZMA_STREAM_END), *i will be set to point + * to a new lzma_index, which the application + * has to later free with lzma_index_end(). + * \param memlimit How much memory the resulting lzma_index is + * allowed to require. liblzma 5.2.3 and earlier + * don't allow 0 here and return LZMA_PROG_ERROR; + * later versions treat 0 as if 1 had been specified. + * + * Valid 'action' arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. + * There is no need to use LZMA_FINISH, but it's allowed because it may + * simplify certain types of applications. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Initialization succeeded, continue with lzma_code(). + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + * + * \note liblzma 5.2.3 and older list also LZMA_MEMLIMIT_ERROR here + * but that error code has never been possible from this + * initialization function. + */ +extern LZMA_API(lzma_ret) lzma_index_decoder( + lzma_stream *strm, lzma_index **i, uint64_t memlimit) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Single-call .xz Index encoder + * + * \note This function doesn't take allocator argument since all + * the internal data is allocated on stack. + * + * \param i lzma_index to be encoded + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * *out_pos is updated only if encoding succeeds. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_BUF_ERROR: Output buffer is too small. Use + * lzma_index_size() to find out how much output + * space is needed. + * - LZMA_PROG_ERROR + * + */ +extern LZMA_API(lzma_ret) lzma_index_buffer_encode(const lzma_index *i, + uint8_t *out, size_t *out_pos, size_t out_size) lzma_nothrow; + + +/** + * \brief Single-call .xz Index decoder + * + * \param[out] i If decoding succeeds, *i will point to a new + * lzma_index, which the application has to + * later free with lzma_index_end(). If an error + * occurs, *i will be NULL. The old value of *i + * is always ignored and thus doesn't need to be + * initialized by the caller. + * \param[out] memlimit Pointer to how much memory the resulting + * lzma_index is allowed to require. The value + * pointed by this pointer is modified if and only + * if LZMA_MEMLIMIT_ERROR is returned. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * \param in Beginning of the input buffer + * \param in_pos The next byte will be read from in[*in_pos]. + * *in_pos is updated only if decoding succeeds. + * \param in_size Size of the input buffer; the first byte that + * won't be read is in[in_size]. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. + * - LZMA_MEM_ERROR + * - LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. + * The minimum required memlimit value was stored to *memlimit. + * - LZMA_DATA_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_buffer_decode(lzma_index **i, + uint64_t *memlimit, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size) + lzma_nothrow; + + +/** + * \brief Initialize a .xz file information decoder + * + * This decoder decodes the Stream Header, Stream Footer, Index, and + * Stream Padding field(s) from the input .xz file and stores the resulting + * combined index in *dest_index. This information can be used to get the + * uncompressed file size with lzma_index_uncompressed_size(*dest_index) or, + * for example, to implement random access reading by locating the Blocks + * in the Streams. + * + * To get the required information from the .xz file, lzma_code() may ask + * the application to seek in the input file by returning LZMA_SEEK_NEEDED + * and having the target file position specified in lzma_stream.seek_pos. + * The number of seeks required depends on the input file and how big buffers + * the application provides. When possible, the decoder will seek backward + * and forward in the given buffer to avoid useless seek requests. Thus, if + * the application provides the whole file at once, no external seeking will + * be required (that is, lzma_code() won't return LZMA_SEEK_NEEDED). + * + * The value in lzma_stream.total_in can be used to estimate how much data + * liblzma had to read to get the file information. However, due to seeking + * and the way total_in is updated, the value of total_in will be somewhat + * inaccurate (a little too big). Thus, total_in is a good estimate but don't + * expect to see the same exact value for the same file if you change the + * input buffer size or switch to a different liblzma version. + * + * Valid 'action' arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. + * You only need to use LZMA_RUN; LZMA_FINISH is only supported because it + * might be convenient for some applications. If you use LZMA_FINISH and if + * lzma_code() asks the application to seek, remember to reset 'action' back + * to LZMA_RUN unless you hit the end of the file again. + * + * Possible return values from lzma_code(): + * - LZMA_OK: All OK so far, more input needed + * - LZMA_SEEK_NEEDED: Provide more input starting from the absolute + * file position strm->seek_pos + * - LZMA_STREAM_END: Decoding was successful, *dest_index has been set + * - LZMA_FORMAT_ERROR: The input file is not in the .xz format (the + * expected magic bytes were not found from the beginning of the file) + * - LZMA_OPTIONS_ERROR: File looks valid but contains headers that aren't + * supported by this version of liblzma + * - LZMA_DATA_ERROR: File is corrupt + * - LZMA_BUF_ERROR + * - LZMA_MEM_ERROR + * - LZMA_MEMLIMIT_ERROR + * - LZMA_PROG_ERROR + * + * \param strm Pointer to a properly prepared lzma_stream + * \param[out] dest_index Pointer to a pointer where the decoder will put + * the decoded lzma_index. The old value + * of *dest_index is ignored (not freed). + * \param memlimit How much memory the resulting lzma_index is + * allowed to require. Use UINT64_MAX to + * effectively disable the limiter. + * \param file_size Size of the input .xz file + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_file_info_decoder( + lzma_stream *strm, lzma_index **dest_index, + uint64_t memlimit, uint64_t file_size) + lzma_nothrow; diff --git a/src/liblzma/api/lzma/index_hash.h b/src/liblzma/api/lzma/index_hash.h new file mode 100644 index 0000000000..68f9024eb3 --- /dev/null +++ b/src/liblzma/api/lzma/index_hash.h @@ -0,0 +1,123 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/index_hash.h + * \brief Validate Index by using a hash function + * \note Never include this file directly. Use instead. + * + * Hashing makes it possible to use constant amount of memory to validate + * Index of arbitrary size. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + +/** + * \brief Opaque data type to hold the Index hash + */ +typedef struct lzma_index_hash_s lzma_index_hash; + + +/** + * \brief Allocate and initialize a new lzma_index_hash structure + * + * If index_hash is NULL, this function allocates and initializes a new + * lzma_index_hash structure and returns a pointer to it. If allocation + * fails, NULL is returned. + * + * If index_hash is non-NULL, this function reinitializes the lzma_index_hash + * structure and returns the same pointer. In this case, return value cannot + * be NULL or a different pointer than the index_hash that was given as + * an argument. + * + * \param index_hash Pointer to a lzma_index_hash structure or NULL. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return Initialized lzma_index_hash structure on success or + * NULL on failure. + */ +extern LZMA_API(lzma_index_hash *) lzma_index_hash_init( + lzma_index_hash *index_hash, const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Deallocate lzma_index_hash structure + * + * \param index_hash Pointer to a lzma_index_hash structure to free. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + */ +extern LZMA_API(void) lzma_index_hash_end( + lzma_index_hash *index_hash, const lzma_allocator *allocator) + lzma_nothrow; + + +/** + * \brief Add a new Record to an Index hash + * + * \param index_hash Pointer to a lzma_index_hash structure + * \param unpadded_size Unpadded Size of a Block + * \param uncompressed_size Uncompressed Size of a Block + * + * \return Possible lzma_ret values: + * - LZMA_OK + * - LZMA_DATA_ERROR: Compressed or uncompressed size of the + * Stream or size of the Index field would grow too big. + * - LZMA_PROG_ERROR: Invalid arguments or this function is being + * used when lzma_index_hash_decode() has already been used. + */ +extern LZMA_API(lzma_ret) lzma_index_hash_append(lzma_index_hash *index_hash, + lzma_vli unpadded_size, lzma_vli uncompressed_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode and validate the Index field + * + * After telling the sizes of all Blocks with lzma_index_hash_append(), + * the actual Index field is decoded with this function. Specifically, + * once decoding of the Index field has been started, no more Records + * can be added using lzma_index_hash_append(). + * + * This function doesn't use lzma_stream structure to pass the input data. + * Instead, the input buffer is specified using three arguments. This is + * because it matches better the internal APIs of liblzma. + * + * \param index_hash Pointer to a lzma_index_hash structure + * \param in Pointer to the beginning of the input buffer + * \param[out] in_pos in[*in_pos] is the next byte to process + * \param in_size in[in_size] is the first byte not to process + * + * \return Possible lzma_ret values: + * - LZMA_OK: So far good, but more input is needed. + * - LZMA_STREAM_END: Index decoded successfully and it matches + * the Records given with lzma_index_hash_append(). + * - LZMA_DATA_ERROR: Index is corrupt or doesn't match the + * information given with lzma_index_hash_append(). + * - LZMA_BUF_ERROR: Cannot progress because *in_pos >= in_size. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_index_hash_decode(lzma_index_hash *index_hash, + const uint8_t *in, size_t *in_pos, size_t in_size) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Get the size of the Index field as bytes + * + * This is needed to verify the Backward Size field in the Stream Footer. + * + * \param index_hash Pointer to a lzma_index_hash structure + * + * \return Size of the Index field in bytes. + */ +extern LZMA_API(lzma_vli) lzma_index_hash_size( + const lzma_index_hash *index_hash) + lzma_nothrow lzma_attr_pure; diff --git a/src/liblzma/api/lzma/lzma12.h b/src/liblzma/api/lzma/lzma12.h new file mode 100644 index 0000000000..fec3e0dadb --- /dev/null +++ b/src/liblzma/api/lzma/lzma12.h @@ -0,0 +1,568 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/lzma12.h + * \brief LZMA1 and LZMA2 filters + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief LZMA1 Filter ID (for raw encoder/decoder only, not in .xz) + * + * LZMA1 is the very same thing as what was called just LZMA in LZMA Utils, + * 7-Zip, and LZMA SDK. It's called LZMA1 here to prevent developers from + * accidentally using LZMA when they actually want LZMA2. + */ +#define LZMA_FILTER_LZMA1 LZMA_VLI_C(0x4000000000000001) + +/** + * \brief LZMA1 Filter ID with extended options (for raw encoder/decoder) + * + * This is like LZMA_FILTER_LZMA1 but with this ID a few extra options + * are supported in the lzma_options_lzma structure: + * + * - A flag to tell the encoder if the end of payload marker (EOPM) alias + * end of stream (EOS) marker must be written at the end of the stream. + * In contrast, LZMA_FILTER_LZMA1 always writes the end marker. + * + * - Decoder needs to be told the uncompressed size of the stream + * or that it is unknown (using the special value UINT64_MAX). + * If the size is known, a flag can be set to allow the presence of + * the end marker anyway. In contrast, LZMA_FILTER_LZMA1 always + * behaves as if the uncompressed size was unknown. + * + * This allows handling file formats where LZMA1 streams are used but where + * the end marker isn't allowed or where it might not (always) be present. + * This extended LZMA1 functionality is provided as a Filter ID for raw + * encoder and decoder instead of adding new encoder and decoder initialization + * functions because this way it is possible to also use extra filters, + * for example, LZMA_FILTER_X86 in a filter chain with LZMA_FILTER_LZMA1EXT, + * which might be needed to handle some file formats. + */ +#define LZMA_FILTER_LZMA1EXT LZMA_VLI_C(0x4000000000000002) + +/** + * \brief LZMA2 Filter ID + * + * Usually you want this instead of LZMA1. Compared to LZMA1, LZMA2 adds + * support for LZMA_SYNC_FLUSH, uncompressed chunks (smaller expansion + * when trying to compress incompressible data), possibility to change + * lc/lp/pb in the middle of encoding, and some other internal improvements. + */ +#define LZMA_FILTER_LZMA2 LZMA_VLI_C(0x21) + + +/** + * \brief Match finders + * + * Match finder has major effect on both speed and compression ratio. + * Usually hash chains are faster than binary trees. + * + * If you will use LZMA_SYNC_FLUSH often, the hash chains may be a better + * choice, because binary trees get much higher compression ratio penalty + * with LZMA_SYNC_FLUSH. + * + * The memory usage formulas are only rough estimates, which are closest to + * reality when dict_size is a power of two. The formulas are more complex + * in reality, and can also change a little between liblzma versions. Use + * lzma_raw_encoder_memusage() to get more accurate estimate of memory usage. + */ +typedef enum { + LZMA_MF_HC3 = 0x03, + /**< + * \brief Hash Chain with 2- and 3-byte hashing + * + * Minimum nice_len: 3 + * + * Memory usage: + * - dict_size <= 16 MiB: dict_size * 7.5 + * - dict_size > 16 MiB: dict_size * 5.5 + 64 MiB + */ + + LZMA_MF_HC4 = 0x04, + /**< + * \brief Hash Chain with 2-, 3-, and 4-byte hashing + * + * Minimum nice_len: 4 + * + * Memory usage: + * - dict_size <= 32 MiB: dict_size * 7.5 + * - dict_size > 32 MiB: dict_size * 6.5 + */ + + LZMA_MF_BT2 = 0x12, + /**< + * \brief Binary Tree with 2-byte hashing + * + * Minimum nice_len: 2 + * + * Memory usage: dict_size * 9.5 + */ + + LZMA_MF_BT3 = 0x13, + /**< + * \brief Binary Tree with 2- and 3-byte hashing + * + * Minimum nice_len: 3 + * + * Memory usage: + * - dict_size <= 16 MiB: dict_size * 11.5 + * - dict_size > 16 MiB: dict_size * 9.5 + 64 MiB + */ + + LZMA_MF_BT4 = 0x14 + /**< + * \brief Binary Tree with 2-, 3-, and 4-byte hashing + * + * Minimum nice_len: 4 + * + * Memory usage: + * - dict_size <= 32 MiB: dict_size * 11.5 + * - dict_size > 32 MiB: dict_size * 10.5 + */ +} lzma_match_finder; + + +/** + * \brief Test if given match finder is supported + * + * It is safe to call this with a value that isn't listed in + * lzma_match_finder enumeration; the return value will be false. + * + * There is no way to list which match finders are available in this + * particular liblzma version and build. It would be useless, because + * a new match finder, which the application developer wasn't aware, + * could require giving additional options to the encoder that the older + * match finders don't need. + * + * \param match_finder Match finder ID + * + * \return lzma_bool: + * - true if the match finder is supported by this liblzma build. + * - false otherwise. + */ +extern LZMA_API(lzma_bool) lzma_mf_is_supported(lzma_match_finder match_finder) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Compression modes + * + * This selects the function used to analyze the data produced by the match + * finder. + */ +typedef enum { + LZMA_MODE_FAST = 1, + /**< + * \brief Fast compression + * + * Fast mode is usually at its best when combined with + * a hash chain match finder. + */ + + LZMA_MODE_NORMAL = 2 + /**< + * \brief Normal compression + * + * This is usually notably slower than fast mode. Use this + * together with binary tree match finders to expose the + * full potential of the LZMA1 or LZMA2 encoder. + */ +} lzma_mode; + + +/** + * \brief Test if given compression mode is supported + * + * It is safe to call this with a value that isn't listed in lzma_mode + * enumeration; the return value will be false. + * + * There is no way to list which modes are available in this particular + * liblzma version and build. It would be useless, because a new compression + * mode, which the application developer wasn't aware, could require giving + * additional options to the encoder that the older modes don't need. + * + * \param mode Mode ID. + * + * \return lzma_bool: + * - true if the compression mode is supported by this liblzma + * build. + * - false otherwise. + */ +extern LZMA_API(lzma_bool) lzma_mode_is_supported(lzma_mode mode) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Options specific to the LZMA1 and LZMA2 filters + * + * Since LZMA1 and LZMA2 share most of the code, it's simplest to share + * the options structure too. For encoding, all but the reserved variables + * need to be initialized unless specifically mentioned otherwise. + * lzma_lzma_preset() can be used to get a good starting point. + * + * For raw decoding, both LZMA1 and LZMA2 need dict_size, preset_dict, and + * preset_dict_size (if preset_dict != NULL). LZMA1 needs also lc, lp, and pb. + */ +typedef struct { + /** + * \brief Dictionary size in bytes + * + * Dictionary size indicates how many bytes of the recently processed + * uncompressed data is kept in memory. One method to reduce size of + * the uncompressed data is to store distance-length pairs, which + * indicate what data to repeat from the dictionary buffer. Thus, + * the bigger the dictionary, the better the compression ratio + * usually is. + * + * Maximum size of the dictionary depends on multiple things: + * - Memory usage limit + * - Available address space (not a problem on 64-bit systems) + * - Selected match finder (encoder only) + * + * Currently the maximum dictionary size for encoding is 1.5 GiB + * (i.e. (UINT32_C(1) << 30) + (UINT32_C(1) << 29)) even on 64-bit + * systems for certain match finder implementation reasons. In the + * future, there may be match finders that support bigger + * dictionaries. + * + * Decoder already supports dictionaries up to 4 GiB - 1 B (i.e. + * UINT32_MAX), so increasing the maximum dictionary size of the + * encoder won't cause problems for old decoders. + * + * Because extremely small dictionaries sizes would have unneeded + * overhead in the decoder, the minimum dictionary size is 4096 bytes. + * + * \note When decoding, too big dictionary does no other harm + * than wasting memory. + */ + uint32_t dict_size; +# define LZMA_DICT_SIZE_MIN UINT32_C(4096) +# define LZMA_DICT_SIZE_DEFAULT (UINT32_C(1) << 23) + + /** + * \brief Pointer to an initial dictionary + * + * It is possible to initialize the LZ77 history window using + * a preset dictionary. It is useful when compressing many + * similar, relatively small chunks of data independently from + * each other. The preset dictionary should contain typical + * strings that occur in the files being compressed. The most + * probable strings should be near the end of the preset dictionary. + * + * This feature should be used only in special situations. For + * now, it works correctly only with raw encoding and decoding. + * Currently none of the container formats supported by + * liblzma allow preset dictionary when decoding, thus if + * you create a .xz or .lzma file with preset dictionary, it + * cannot be decoded with the regular decoder functions. In the + * future, the .xz format will likely get support for preset + * dictionary though. + */ + const uint8_t *preset_dict; + + /** + * \brief Size of the preset dictionary + * + * Specifies the size of the preset dictionary. If the size is + * bigger than dict_size, only the last dict_size bytes are + * processed. + * + * This variable is read only when preset_dict is not NULL. + * If preset_dict is not NULL but preset_dict_size is zero, + * no preset dictionary is used (identical to only setting + * preset_dict to NULL). + */ + uint32_t preset_dict_size; + + /** + * \brief Number of literal context bits + * + * How many of the highest bits of the previous uncompressed + * eight-bit byte (also known as 'literal') are taken into + * account when predicting the bits of the next literal. + * + * E.g. in typical English text, an upper-case letter is + * often followed by a lower-case letter, and a lower-case + * letter is usually followed by another lower-case letter. + * In the US-ASCII character set, the highest three bits are 010 + * for upper-case letters and 011 for lower-case letters. + * When lc is at least 3, the literal coding can take advantage of + * this property in the uncompressed data. + * + * There is a limit that applies to literal context bits and literal + * position bits together: lc + lp <= 4. Without this limit the + * decoding could become very slow, which could have security related + * results in some cases like email servers doing virus scanning. + * This limit also simplifies the internal implementation in liblzma. + * + * There may be LZMA1 streams that have lc + lp > 4 (maximum possible + * lc would be 8). It is not possible to decode such streams with + * liblzma. + */ + uint32_t lc; +# define LZMA_LCLP_MIN 0 +# define LZMA_LCLP_MAX 4 +# define LZMA_LC_DEFAULT 3 + + /** + * \brief Number of literal position bits + * + * lp affects what kind of alignment in the uncompressed data is + * assumed when encoding literals. A literal is a single 8-bit byte. + * See pb below for more information about alignment. + */ + uint32_t lp; +# define LZMA_LP_DEFAULT 0 + + /** + * \brief Number of position bits + * + * pb affects what kind of alignment in the uncompressed data is + * assumed in general. The default means four-byte alignment + * (2^ pb =2^2=4), which is often a good choice when there's + * no better guess. + * + * When the alignment is known, setting pb accordingly may reduce + * the file size a little. E.g. with text files having one-byte + * alignment (US-ASCII, ISO-8859-*, UTF-8), setting pb=0 can + * improve compression slightly. For UTF-16 text, pb=1 is a good + * choice. If the alignment is an odd number like 3 bytes, pb=0 + * might be the best choice. + * + * Even though the assumed alignment can be adjusted with pb and + * lp, LZMA1 and LZMA2 still slightly favor 16-byte alignment. + * It might be worth taking into account when designing file formats + * that are likely to be often compressed with LZMA1 or LZMA2. + */ + uint32_t pb; +# define LZMA_PB_MIN 0 +# define LZMA_PB_MAX 4 +# define LZMA_PB_DEFAULT 2 + + /** Compression mode */ + lzma_mode mode; + + /** + * \brief Nice length of a match + * + * This determines how many bytes the encoder compares from the match + * candidates when looking for the best match. Once a match of at + * least nice_len bytes long is found, the encoder stops looking for + * better candidates and encodes the match. (Naturally, if the found + * match is actually longer than nice_len, the actual length is + * encoded; it's not truncated to nice_len.) + * + * Bigger values usually increase the compression ratio and + * compression time. For most files, 32 to 128 is a good value, + * which gives very good compression ratio at good speed. + * + * The exact minimum value depends on the match finder. The maximum + * is 273, which is the maximum length of a match that LZMA1 and + * LZMA2 can encode. + */ + uint32_t nice_len; + + /** Match finder ID */ + lzma_match_finder mf; + + /** + * \brief Maximum search depth in the match finder + * + * For every input byte, match finder searches through the hash chain + * or binary tree in a loop, each iteration going one step deeper in + * the chain or tree. The searching stops if + * - a match of at least nice_len bytes long is found; + * - all match candidates from the hash chain or binary tree have + * been checked; or + * - maximum search depth is reached. + * + * Maximum search depth is needed to prevent the match finder from + * wasting too much time in case there are lots of short match + * candidates. On the other hand, stopping the search before all + * candidates have been checked can reduce compression ratio. + * + * Setting depth to zero tells liblzma to use an automatic default + * value, that depends on the selected match finder and nice_len. + * The default is in the range [4, 200] or so (it may vary between + * liblzma versions). + * + * Using a bigger depth value than the default can increase + * compression ratio in some cases. There is no strict maximum value, + * but high values (thousands or millions) should be used with care: + * the encoder could remain fast enough with typical input, but + * malicious input could cause the match finder to slow down + * dramatically, possibly creating a denial of service attack. + */ + uint32_t depth; + + /** + * \brief For LZMA_FILTER_LZMA1EXT: Extended flags + * + * This is used only with LZMA_FILTER_LZMA1EXT. + * + * Currently only one flag is supported, LZMA_LZMA1EXT_ALLOW_EOPM: + * + * - Encoder: If the flag is set, then end marker is written just + * like it is with LZMA_FILTER_LZMA1. Without this flag the + * end marker isn't written and the application has to store + * the uncompressed size somewhere outside the compressed stream. + * To decompress streams without the end marker, the application + * has to set the correct uncompressed size in ext_size_low and + * ext_size_high. + * + * - Decoder: If the uncompressed size in ext_size_low and + * ext_size_high is set to the special value UINT64_MAX + * (indicating unknown uncompressed size) then this flag is + * ignored and the end marker must always be present, that is, + * the behavior is identical to LZMA_FILTER_LZMA1. + * + * Otherwise, if this flag isn't set, then the input stream + * must not have the end marker; if the end marker is detected + * then it will result in LZMA_DATA_ERROR. This is useful when + * it is known that the stream must not have the end marker and + * strict validation is wanted. + * + * If this flag is set, then it is autodetected if the end marker + * is present after the specified number of uncompressed bytes + * has been decompressed (ext_size_low and ext_size_high). The + * end marker isn't allowed in any other position. This behavior + * is useful when uncompressed size is known but the end marker + * may or may not be present. This is the case, for example, + * in .7z files (valid .7z files that have the end marker in + * LZMA1 streams are rare but they do exist). + */ + uint32_t ext_flags; +# define LZMA_LZMA1EXT_ALLOW_EOPM UINT32_C(0x01) + + /** + * \brief For LZMA_FILTER_LZMA1EXT: Uncompressed size (low bits) + * + * The 64-bit uncompressed size is needed for decompression with + * LZMA_FILTER_LZMA1EXT. The size is ignored by the encoder. + * + * The special value UINT64_MAX indicates that the uncompressed size + * is unknown and that the end of payload marker (also known as + * end of stream marker) must be present to indicate the end of + * the LZMA1 stream. Any other value indicates the expected + * uncompressed size of the LZMA1 stream. (If LZMA1 was used together + * with filters that change the size of the data then the uncompressed + * size of the LZMA1 stream could be different than the final + * uncompressed size of the filtered stream.) + * + * ext_size_low holds the least significant 32 bits of the + * uncompressed size. The most significant 32 bits must be set + * in ext_size_high. The macro lzma_set_ext_size(opt_lzma, u64size) + * can be used to set these members. + * + * The 64-bit uncompressed size is split into two uint32_t variables + * because there were no reserved uint64_t members and using the + * same options structure for LZMA_FILTER_LZMA1, LZMA_FILTER_LZMA1EXT, + * and LZMA_FILTER_LZMA2 was otherwise more convenient than having + * a new options structure for LZMA_FILTER_LZMA1EXT. (Replacing two + * uint32_t members with one uint64_t changes the ABI on some systems + * as the alignment of this struct can increase from 4 bytes to 8.) + */ + uint32_t ext_size_low; + + /** + * \brief For LZMA_FILTER_LZMA1EXT: Uncompressed size (high bits) + * + * This holds the most significant 32 bits of the uncompressed size. + */ + uint32_t ext_size_high; + + /* + * Reserved space to allow possible future extensions without + * breaking the ABI. You should not touch these, because the names + * of these variables may change. These are and will never be used + * with the currently supported options, so it is safe to leave these + * uninitialized. + */ + + /** \private Reserved member. */ + uint32_t reserved_int4; + + /** \private Reserved member. */ + uint32_t reserved_int5; + + /** \private Reserved member. */ + uint32_t reserved_int6; + + /** \private Reserved member. */ + uint32_t reserved_int7; + + /** \private Reserved member. */ + uint32_t reserved_int8; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum1; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum2; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum3; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum4; + + /** \private Reserved member. */ + void *reserved_ptr1; + + /** \private Reserved member. */ + void *reserved_ptr2; + +} lzma_options_lzma; + + +/** + * \brief Macro to set the 64-bit uncompressed size in ext_size_* + * + * This might be convenient when decoding using LZMA_FILTER_LZMA1EXT. + * This isn't used with LZMA_FILTER_LZMA1 or LZMA_FILTER_LZMA2. + */ +#define lzma_set_ext_size(opt_lzma2, u64size) \ +do { \ + (opt_lzma2).ext_size_low = (uint32_t)(u64size); \ + (opt_lzma2).ext_size_high = (uint32_t)((uint64_t)(u64size) >> 32); \ +} while (0) + + +/** + * \brief Set a compression preset to lzma_options_lzma structure + * + * 0 is the fastest and 9 is the slowest. These match the switches -0 .. -9 + * of the xz command line tool. In addition, it is possible to bitwise-or + * flags to the preset. Currently only LZMA_PRESET_EXTREME is supported. + * The flags are defined in container.h, because the flags are used also + * with lzma_easy_encoder(). + * + * The preset levels are subject to changes between liblzma versions. + * + * This function is available only if LZMA1 or LZMA2 encoder has been enabled + * when building liblzma. + * + * If features (like certain match finders) have been disabled at build time, + * then the function may return success (false) even though the resulting + * LZMA1/LZMA2 options may not be usable for encoder initialization + * (LZMA_OPTIONS_ERROR). + * + * \param[out] options Pointer to LZMA1 or LZMA2 options to be filled + * \param preset Preset level bitwse-ORed with preset flags + * + * \return lzma_bool: + * - true if the preset is not supported (failure). + * - false otherwise (success). + */ +extern LZMA_API(lzma_bool) lzma_lzma_preset( + lzma_options_lzma *options, uint32_t preset) lzma_nothrow; diff --git a/src/liblzma/api/lzma/stream_flags.h b/src/liblzma/api/lzma/stream_flags.h new file mode 100644 index 0000000000..a33fe46837 --- /dev/null +++ b/src/liblzma/api/lzma/stream_flags.h @@ -0,0 +1,265 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/stream_flags.h + * \brief .xz Stream Header and Stream Footer encoder and decoder + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Size of Stream Header and Stream Footer + * + * Stream Header and Stream Footer have the same size and they are not + * going to change even if a newer version of the .xz file format is + * developed in future. + */ +#define LZMA_STREAM_HEADER_SIZE 12 + + +/** + * \brief Options for encoding/decoding Stream Header and Stream Footer + */ +typedef struct { + /** + * \brief Stream Flags format version + * + * To prevent API and ABI breakages if new features are needed in + * Stream Header or Stream Footer, a version number is used to + * indicate which members in this structure are in use. For now, + * version must always be zero. With non-zero version, the + * lzma_stream_header_encode() and lzma_stream_footer_encode() + * will return LZMA_OPTIONS_ERROR. + * + * lzma_stream_header_decode() and lzma_stream_footer_decode() + * will always set this to the lowest value that supports all the + * features indicated by the Stream Flags field. The application + * must check that the version number set by the decoding functions + * is supported by the application. Otherwise it is possible that + * the application will decode the Stream incorrectly. + */ + uint32_t version; + + /** + * \brief Backward Size + * + * Backward Size must be a multiple of four bytes. In this Stream + * format version, Backward Size is the size of the Index field. + * + * Backward Size isn't actually part of the Stream Flags field, but + * it is convenient to include in this structure anyway. Backward + * Size is present only in the Stream Footer. There is no need to + * initialize backward_size when encoding Stream Header. + * + * lzma_stream_header_decode() always sets backward_size to + * LZMA_VLI_UNKNOWN so that it is convenient to use + * lzma_stream_flags_compare() when both Stream Header and Stream + * Footer have been decoded. + */ + lzma_vli backward_size; + + /** + * \brief Minimum value for lzma_stream_flags.backward_size + */ +# define LZMA_BACKWARD_SIZE_MIN 4 + + /** + * \brief Maximum value for lzma_stream_flags.backward_size + */ +# define LZMA_BACKWARD_SIZE_MAX (LZMA_VLI_C(1) << 34) + + /** + * \brief Check ID + * + * This indicates the type of the integrity check calculated from + * uncompressed data. + */ + lzma_check check; + + /* + * Reserved space to allow possible future extensions without + * breaking the ABI. You should not touch these, because the + * names of these variables may change. + * + * (We will never be able to use all of these since Stream Flags + * is just two bytes plus Backward Size of four bytes. But it's + * nice to have the proper types when they are needed.) + */ + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum1; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum2; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum3; + + /** \private Reserved member. */ + lzma_reserved_enum reserved_enum4; + + /** \private Reserved member. */ + lzma_bool reserved_bool1; + + /** \private Reserved member. */ + lzma_bool reserved_bool2; + + /** \private Reserved member. */ + lzma_bool reserved_bool3; + + /** \private Reserved member. */ + lzma_bool reserved_bool4; + + /** \private Reserved member. */ + lzma_bool reserved_bool5; + + /** \private Reserved member. */ + lzma_bool reserved_bool6; + + /** \private Reserved member. */ + lzma_bool reserved_bool7; + + /** \private Reserved member. */ + lzma_bool reserved_bool8; + + /** \private Reserved member. */ + uint32_t reserved_int1; + + /** \private Reserved member. */ + uint32_t reserved_int2; + +} lzma_stream_flags; + + +/** + * \brief Encode Stream Header + * + * \param options Stream Header options to be encoded. + * options->backward_size is ignored and doesn't + * need to be initialized. + * \param[out] out Beginning of the output buffer of + * LZMA_STREAM_HEADER_SIZE bytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_OPTIONS_ERROR: options->version is not supported by + * this liblzma version. + * - LZMA_PROG_ERROR: Invalid options. + */ +extern LZMA_API(lzma_ret) lzma_stream_header_encode( + const lzma_stream_flags *options, uint8_t *out) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Encode Stream Footer + * + * \param options Stream Footer options to be encoded. + * \param[out] out Beginning of the output buffer of + * LZMA_STREAM_HEADER_SIZE bytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Encoding was successful. + * - LZMA_OPTIONS_ERROR: options->version is not supported by + * this liblzma version. + * - LZMA_PROG_ERROR: Invalid options. + */ +extern LZMA_API(lzma_ret) lzma_stream_footer_encode( + const lzma_stream_flags *options, uint8_t *out) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode Stream Header + * + * options->backward_size is always set to LZMA_VLI_UNKNOWN. This is to + * help comparing Stream Flags from Stream Header and Stream Footer with + * lzma_stream_flags_compare(). + * + * \note When decoding .xz files that contain multiple Streams, it may + * make sense to print "file format not recognized" only if + * decoding of the Stream Header of the \a first Stream gives + * LZMA_FORMAT_ERROR. If non-first Stream Header gives + * LZMA_FORMAT_ERROR, the message used for LZMA_DATA_ERROR is + * probably more appropriate. + * For example, the Stream decoder in liblzma uses + * LZMA_DATA_ERROR if LZMA_FORMAT_ERROR is returned by + * lzma_stream_header_decode() when decoding non-first Stream. + * + * \param[out] options Target for the decoded Stream Header options. + * \param in Beginning of the input buffer of + * LZMA_STREAM_HEADER_SIZE bytes. + * + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. + * - LZMA_FORMAT_ERROR: Magic bytes don't match, thus the given + * buffer cannot be Stream Header. + * - LZMA_DATA_ERROR: CRC32 doesn't match, thus the header + * is corrupt. + * - LZMA_OPTIONS_ERROR: Unsupported options are present + * in the header. + */ +extern LZMA_API(lzma_ret) lzma_stream_header_decode( + lzma_stream_flags *options, const uint8_t *in) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode Stream Footer + * + * \note If Stream Header was already decoded successfully, but + * decoding Stream Footer returns LZMA_FORMAT_ERROR, the + * application should probably report some other error message + * than "file format not recognized". The file likely + * is corrupt (possibly truncated). The Stream decoder in liblzma + * uses LZMA_DATA_ERROR in this situation. + * + * \param[out] options Target for the decoded Stream Footer options. + * \param in Beginning of the input buffer of + * LZMA_STREAM_HEADER_SIZE bytes. + * + * \return Possible lzma_ret values: + * - LZMA_OK: Decoding was successful. + * - LZMA_FORMAT_ERROR: Magic bytes don't match, thus the given + * buffer cannot be Stream Footer. + * - LZMA_DATA_ERROR: CRC32 doesn't match, thus the Stream Footer + * is corrupt. + * - LZMA_OPTIONS_ERROR: Unsupported options are present + * in Stream Footer. + */ +extern LZMA_API(lzma_ret) lzma_stream_footer_decode( + lzma_stream_flags *options, const uint8_t *in) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Compare two lzma_stream_flags structures + * + * backward_size values are compared only if both are not + * LZMA_VLI_UNKNOWN. + * + * \param a Pointer to lzma_stream_flags structure + * \param b Pointer to lzma_stream_flags structure + * + * \return Possible lzma_ret values: + * - LZMA_OK: Both are equal. If either had backward_size set + * to LZMA_VLI_UNKNOWN, backward_size values were not + * compared or validated. + * - LZMA_DATA_ERROR: The structures differ. + * - LZMA_OPTIONS_ERROR: version in either structure is greater + * than the maximum supported version (currently zero). + * - LZMA_PROG_ERROR: Invalid value, e.g. invalid check or + * backward_size. + */ +extern LZMA_API(lzma_ret) lzma_stream_flags_compare( + const lzma_stream_flags *a, const lzma_stream_flags *b) + lzma_nothrow lzma_attr_pure; diff --git a/src/liblzma/api/lzma/version.h b/src/liblzma/api/lzma/version.h new file mode 100644 index 0000000000..86b3556359 --- /dev/null +++ b/src/liblzma/api/lzma/version.h @@ -0,0 +1,134 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/version.h + * \brief Version number + * \note Never include this file directly. Use instead. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** \brief Major version number of the liblzma release. */ +#define LZMA_VERSION_MAJOR 5 + +/** \brief Minor version number of the liblzma release. */ +#define LZMA_VERSION_MINOR 8 + +/** \brief Patch version number of the liblzma release. */ +#define LZMA_VERSION_PATCH 1 + +/** + * \brief Version stability marker + * + * This will always be one of three values: + * - LZMA_VERSION_STABILITY_ALPHA + * - LZMA_VERSION_STABILITY_BETA + * - LZMA_VERSION_STABILITY_STABLE + */ +#define LZMA_VERSION_STABILITY LZMA_VERSION_STABILITY_STABLE + +/** \brief Commit version number of the liblzma release */ +#ifndef LZMA_VERSION_COMMIT +# define LZMA_VERSION_COMMIT "" +#endif + + +/* + * Map symbolic stability levels to integers. + */ +#define LZMA_VERSION_STABILITY_ALPHA 0 +#define LZMA_VERSION_STABILITY_BETA 1 +#define LZMA_VERSION_STABILITY_STABLE 2 + + +/** + * \brief Compile-time version number + * + * The version number is of format xyyyzzzs where + * - x = major + * - yyy = minor + * - zzz = revision + * - s indicates stability: 0 = alpha, 1 = beta, 2 = stable + * + * The same xyyyzzz triplet is never reused with different stability levels. + * For example, if 5.1.0alpha has been released, there will never be 5.1.0beta + * or 5.1.0 stable. + * + * \note The version number of liblzma has nothing to with + * the version number of Igor Pavlov's LZMA SDK. + */ +#define LZMA_VERSION (LZMA_VERSION_MAJOR * UINT32_C(10000000) \ + + LZMA_VERSION_MINOR * UINT32_C(10000) \ + + LZMA_VERSION_PATCH * UINT32_C(10) \ + + LZMA_VERSION_STABILITY) + + +/* + * Macros to construct the compile-time version string + */ +#if LZMA_VERSION_STABILITY == LZMA_VERSION_STABILITY_ALPHA +# define LZMA_VERSION_STABILITY_STRING "alpha" +#elif LZMA_VERSION_STABILITY == LZMA_VERSION_STABILITY_BETA +# define LZMA_VERSION_STABILITY_STRING "beta" +#elif LZMA_VERSION_STABILITY == LZMA_VERSION_STABILITY_STABLE +# define LZMA_VERSION_STABILITY_STRING "" +#else +# error Incorrect LZMA_VERSION_STABILITY +#endif + +#define LZMA_VERSION_STRING_C_(major, minor, patch, stability, commit) \ + #major "." #minor "." #patch stability commit + +#define LZMA_VERSION_STRING_C(major, minor, patch, stability, commit) \ + LZMA_VERSION_STRING_C_(major, minor, patch, stability, commit) + + +/** + * \brief Compile-time version as a string + * + * This can be for example "4.999.5alpha", "4.999.8beta", or "5.0.0" (stable + * versions don't have any "stable" suffix). In future, a snapshot built + * from source code repository may include an additional suffix, for example + * "4.999.8beta-21-g1d92". The commit ID won't be available in numeric form + * in LZMA_VERSION macro. + */ +#define LZMA_VERSION_STRING LZMA_VERSION_STRING_C( \ + LZMA_VERSION_MAJOR, LZMA_VERSION_MINOR, \ + LZMA_VERSION_PATCH, LZMA_VERSION_STABILITY_STRING, \ + LZMA_VERSION_COMMIT) + + +/* #ifndef is needed for use with windres (MinGW-w64 or Cygwin). */ +#ifndef LZMA_H_INTERNAL_RC + +/** + * \brief Run-time version number as an integer + * + * This allows an application to compare if it was built against the same, + * older, or newer version of liblzma that is currently running. + * + * \return The value of LZMA_VERSION macro at the compile time of liblzma + */ +extern LZMA_API(uint32_t) lzma_version_number(void) + lzma_nothrow lzma_attr_const; + + +/** + * \brief Run-time version as a string + * + * This function may be useful to display which version of liblzma an + * application is currently using. + * + * \return Run-time version of liblzma + */ +extern LZMA_API(const char *) lzma_version_string(void) + lzma_nothrow lzma_attr_const; + +#endif diff --git a/src/liblzma/api/lzma/vli.h b/src/liblzma/api/lzma/vli.h new file mode 100644 index 0000000000..6b049021b9 --- /dev/null +++ b/src/liblzma/api/lzma/vli.h @@ -0,0 +1,166 @@ +/* SPDX-License-Identifier: 0BSD */ + +/** + * \file lzma/vli.h + * \brief Variable-length integer handling + * \note Never include this file directly. Use instead. + * + * In the .xz format, most integers are encoded in a variable-length + * representation, which is sometimes called little endian base-128 encoding. + * This saves space when smaller values are more likely than bigger values. + * + * The encoding scheme encodes seven bits to every byte, using minimum + * number of bytes required to represent the given value. Encodings that use + * non-minimum number of bytes are invalid, thus every integer has exactly + * one encoded representation. The maximum number of bits in a VLI is 63, + * thus the vli argument must be less than or equal to UINT64_MAX / 2. You + * should use LZMA_VLI_MAX for clarity. + */ + +/* + * Author: Lasse Collin + */ + +#ifndef LZMA_H_INTERNAL +# error Never include this file directly. Use instead. +#endif + + +/** + * \brief Maximum supported value of a variable-length integer + */ +#define LZMA_VLI_MAX (UINT64_MAX / 2) + +/** + * \brief VLI value to denote that the value is unknown + */ +#define LZMA_VLI_UNKNOWN UINT64_MAX + +/** + * \brief Maximum supported encoded length of variable length integers + */ +#define LZMA_VLI_BYTES_MAX 9 + +/** + * \brief VLI constant suffix + */ +#define LZMA_VLI_C(n) UINT64_C(n) + + +/** + * \brief Variable-length integer type + * + * Valid VLI values are in the range [0, LZMA_VLI_MAX]. Unknown value is + * indicated with LZMA_VLI_UNKNOWN, which is the maximum value of the + * underlying integer type. + * + * lzma_vli will be uint64_t for the foreseeable future. If a bigger size + * is needed in the future, it is guaranteed that 2 * LZMA_VLI_MAX will + * not overflow lzma_vli. This simplifies integer overflow detection. + */ +typedef uint64_t lzma_vli; + + +/** + * \brief Validate a variable-length integer + * + * This is useful to test that application has given acceptable values + * for example in the uncompressed_size and compressed_size variables. + * + * \return True if the integer is representable as a VLI or if it + * indicates an unknown value. False otherwise. + */ +#define lzma_vli_is_valid(vli) \ + ((vli) <= LZMA_VLI_MAX || (vli) == LZMA_VLI_UNKNOWN) + + +/** + * \brief Encode a variable-length integer + * + * This function has two modes: single-call and multi-call. Single-call mode + * encodes the whole integer at once; it is an error if the output buffer is + * too small. Multi-call mode saves the position in *vli_pos, and thus it is + * possible to continue encoding if the buffer becomes full before the whole + * integer has been encoded. + * + * \param vli Integer to be encoded + * \param[out] vli_pos How many VLI-encoded bytes have already been written + * out. When starting to encode a new integer in + * multi-call mode, *vli_pos must be set to zero. + * To use single-call encoding, set vli_pos to NULL. + * \param[out] out Beginning of the output buffer + * \param[out] out_pos The next byte will be written to out[*out_pos]. + * \param out_size Size of the out buffer; the first byte into + * which no data is written to is out[out_size]. + * + * \return Slightly different return values are used in multi-call and + * single-call modes. + * + * Single-call (vli_pos == NULL): + * - LZMA_OK: Integer successfully encoded. + * - LZMA_PROG_ERROR: Arguments are not sane. This can be due + * to too little output space; single-call mode doesn't use + * LZMA_BUF_ERROR, since the application should have checked + * the encoded size with lzma_vli_size(). + * + * Multi-call (vli_pos != NULL): + * - LZMA_OK: So far all OK, but the integer is not + * completely written out yet. + * - LZMA_STREAM_END: Integer successfully encoded. + * - LZMA_BUF_ERROR: No output space was provided. + * - LZMA_PROG_ERROR: Arguments are not sane. + */ +extern LZMA_API(lzma_ret) lzma_vli_encode(lzma_vli vli, size_t *vli_pos, + uint8_t *out, size_t *out_pos, size_t out_size) lzma_nothrow; + + +/** + * \brief Decode a variable-length integer + * + * Like lzma_vli_encode(), this function has single-call and multi-call modes. + * + * \param[out] vli Pointer to decoded integer. The decoder will + * initialize it to zero when *vli_pos == 0, so + * application isn't required to initialize *vli. + * \param[out] vli_pos How many bytes have already been decoded. When + * starting to decode a new integer in multi-call + * mode, *vli_pos must be initialized to zero. To + * use single-call decoding, set vli_pos to NULL. + * \param in Beginning of the input buffer + * \param[out] in_pos The next byte will be read from in[*in_pos]. + * \param in_size Size of the input buffer; the first byte that + * won't be read is in[in_size]. + * + * \return Slightly different return values are used in multi-call and + * single-call modes. + * + * Single-call (vli_pos == NULL): + * - LZMA_OK: Integer successfully decoded. + * - LZMA_DATA_ERROR: Integer is corrupt. This includes hitting + * the end of the input buffer before the whole integer was + * decoded; providing no input at all will use LZMA_DATA_ERROR. + * - LZMA_PROG_ERROR: Arguments are not sane. + * + * Multi-call (vli_pos != NULL): + * - LZMA_OK: So far all OK, but the integer is not + * completely decoded yet. + * - LZMA_STREAM_END: Integer successfully decoded. + * - LZMA_DATA_ERROR: Integer is corrupt. + * - LZMA_BUF_ERROR: No input was provided. + * - LZMA_PROG_ERROR: Arguments are not sane. + */ +extern LZMA_API(lzma_ret) lzma_vli_decode(lzma_vli *vli, size_t *vli_pos, + const uint8_t *in, size_t *in_pos, size_t in_size) + lzma_nothrow; + + +/** + * \brief Get the number of bytes required to encode a VLI + * + * \param vli Integer whose encoded size is to be determined + * + * \return Number of bytes on success (1-9). If vli isn't valid, + * zero is returned. + */ +extern LZMA_API(uint32_t) lzma_vli_size(lzma_vli vli) + lzma_nothrow lzma_attr_pure; diff --git a/src/liblzma/check/Makefile.inc b/src/liblzma/check/Makefile.inc new file mode 100644 index 0000000000..00a26e687f --- /dev/null +++ b/src/liblzma/check/Makefile.inc @@ -0,0 +1,50 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +## Note: There is no check for COND_CHECK_CRC32 because +## currently crc32 is always enabled. + +EXTRA_DIST += \ + check/crc_clmul_consts_gen.c \ + check/crc32_tablegen.c \ + check/crc64_tablegen.c + +liblzma_la_SOURCES += \ + check/check.c \ + check/check.h \ + check/crc_common.h \ + check/crc_x86_clmul.h \ + check/crc32_arm64.h \ + check/crc32_loongarch.h + +if COND_SMALL +liblzma_la_SOURCES += check/crc32_small.c +else +liblzma_la_SOURCES += \ + check/crc32_fast.c \ + check/crc32_table_le.h \ + check/crc32_table_be.h +if COND_ASM_X86 +liblzma_la_SOURCES += check/crc32_x86.S +endif +endif + +if COND_CHECK_CRC64 +if COND_SMALL +liblzma_la_SOURCES += check/crc64_small.c +else +liblzma_la_SOURCES += \ + check/crc64_fast.c \ + check/crc64_table_le.h \ + check/crc64_table_be.h +if COND_ASM_X86 +liblzma_la_SOURCES += check/crc64_x86.S +endif +endif +endif + +if COND_CHECK_SHA256 +if COND_INTERNAL_SHA256 +liblzma_la_SOURCES += check/sha256.c +endif +endif diff --git a/src/liblzma/check/check.c b/src/liblzma/check/check.c new file mode 100644 index 0000000000..7734ace185 --- /dev/null +++ b/src/liblzma/check/check.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file check.c +/// \brief Single API to access different integrity checks +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "check.h" + + +extern LZMA_API(lzma_bool) +lzma_check_is_supported(lzma_check type) +{ + if ((unsigned int)(type) > LZMA_CHECK_ID_MAX) + return false; + + static const lzma_bool available_checks[LZMA_CHECK_ID_MAX + 1] = { + true, // LZMA_CHECK_NONE + +#ifdef HAVE_CHECK_CRC32 + true, +#else + false, +#endif + + false, // Reserved + false, // Reserved + +#ifdef HAVE_CHECK_CRC64 + true, +#else + false, +#endif + + false, // Reserved + false, // Reserved + false, // Reserved + false, // Reserved + false, // Reserved + +#ifdef HAVE_CHECK_SHA256 + true, +#else + false, +#endif + + false, // Reserved + false, // Reserved + false, // Reserved + false, // Reserved + false, // Reserved + }; + + return available_checks[(unsigned int)(type)]; +} + + +extern LZMA_API(uint32_t) +lzma_check_size(lzma_check type) +{ + if ((unsigned int)(type) > LZMA_CHECK_ID_MAX) + return UINT32_MAX; + + // See file-format.txt section 2.1.1.2. + static const uint8_t check_sizes[LZMA_CHECK_ID_MAX + 1] = { + 0, + 4, 4, 4, + 8, 8, 8, + 16, 16, 16, + 32, 32, 32, + 64, 64, 64 + }; + + return check_sizes[(unsigned int)(type)]; +} + + +extern void +lzma_check_init(lzma_check_state *check, lzma_check type) +{ + switch (type) { + case LZMA_CHECK_NONE: + break; + +#ifdef HAVE_CHECK_CRC32 + case LZMA_CHECK_CRC32: + check->state.crc32 = 0; + break; +#endif + +#ifdef HAVE_CHECK_CRC64 + case LZMA_CHECK_CRC64: + check->state.crc64 = 0; + break; +#endif + +#ifdef HAVE_CHECK_SHA256 + case LZMA_CHECK_SHA256: + lzma_sha256_init(check); + break; +#endif + + default: + break; + } + + return; +} + + +extern void +lzma_check_update(lzma_check_state *check, lzma_check type, + const uint8_t *buf, size_t size) +{ + switch (type) { +#ifdef HAVE_CHECK_CRC32 + case LZMA_CHECK_CRC32: + check->state.crc32 = lzma_crc32(buf, size, check->state.crc32); + break; +#endif + +#ifdef HAVE_CHECK_CRC64 + case LZMA_CHECK_CRC64: + check->state.crc64 = lzma_crc64(buf, size, check->state.crc64); + break; +#endif + +#ifdef HAVE_CHECK_SHA256 + case LZMA_CHECK_SHA256: + lzma_sha256_update(buf, size, check); + break; +#endif + + default: + break; + } + + return; +} + + +extern void +lzma_check_finish(lzma_check_state *check, lzma_check type) +{ + switch (type) { +#ifdef HAVE_CHECK_CRC32 + case LZMA_CHECK_CRC32: + check->buffer.u32[0] = conv32le(check->state.crc32); + break; +#endif + +#ifdef HAVE_CHECK_CRC64 + case LZMA_CHECK_CRC64: + check->buffer.u64[0] = conv64le(check->state.crc64); + break; +#endif + +#ifdef HAVE_CHECK_SHA256 + case LZMA_CHECK_SHA256: + lzma_sha256_finish(check); + break; +#endif + + default: + break; + } + + return; +} diff --git a/src/liblzma/check/check.h b/src/liblzma/check/check.h new file mode 100644 index 0000000000..16a5633421 --- /dev/null +++ b/src/liblzma/check/check.h @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file check.h +/// \brief Internal API to different integrity check functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_CHECK_H +#define LZMA_CHECK_H + +#include "common.h" + +// If the function for external SHA-256 is missing, use the internal SHA-256 +// code. Due to how configure works, these defines can only get defined when +// both a usable header and a type have already been found. +#if !(defined(HAVE_CC_SHA256_INIT) \ + || defined(HAVE_SHA256_INIT) \ + || defined(HAVE_SHA256INIT)) +# define HAVE_INTERNAL_SHA256 1 +#endif + +#if defined(HAVE_INTERNAL_SHA256) +// Nothing +#elif defined(HAVE_COMMONCRYPTO_COMMONDIGEST_H) +# include +#elif defined(HAVE_SHA256_H) +# include +# include +#elif defined(HAVE_SHA2_H) +# include +# include +#endif + +#if defined(HAVE_INTERNAL_SHA256) +/// State for the internal SHA-256 implementation +typedef struct { + /// Internal state + uint32_t state[8]; + + /// Size of the message excluding padding + uint64_t size; +} lzma_sha256_state; +#elif defined(HAVE_CC_SHA256_CTX) +typedef CC_SHA256_CTX lzma_sha256_state; +#elif defined(HAVE_SHA256_CTX) +typedef SHA256_CTX lzma_sha256_state; +#elif defined(HAVE_SHA2_CTX) +typedef SHA2_CTX lzma_sha256_state; +#endif + +#if defined(HAVE_INTERNAL_SHA256) +// Nothing +#elif defined(HAVE_CC_SHA256_INIT) +# define LZMA_SHA256FUNC(x) CC_SHA256_ ## x +#elif defined(HAVE_SHA256_INIT) +# define LZMA_SHA256FUNC(x) SHA256_ ## x +#elif defined(HAVE_SHA256INIT) +# define LZMA_SHA256FUNC(x) SHA256 ## x +#endif + +// Index hashing needs the best possible hash function (preferably +// a cryptographic hash) for maximum reliability. +#if defined(HAVE_CHECK_SHA256) +# define LZMA_CHECK_BEST LZMA_CHECK_SHA256 +#elif defined(HAVE_CHECK_CRC64) +# define LZMA_CHECK_BEST LZMA_CHECK_CRC64 +#else +# define LZMA_CHECK_BEST LZMA_CHECK_CRC32 +#endif + + +/// \brief Structure to hold internal state of the check being calculated +/// +/// \note This is not in the public API because this structure may +/// change in future if new integrity check algorithms are added. +typedef struct { + /// Buffer to hold the final result and a temporary buffer for SHA256. + union { + uint8_t u8[64]; + uint32_t u32[16]; + uint64_t u64[8]; + } buffer; + + /// Check-specific data + union { + uint32_t crc32; + uint64_t crc64; + lzma_sha256_state sha256; + } state; + +} lzma_check_state; + + +/// \brief Initialize *check depending on type +extern void lzma_check_init(lzma_check_state *check, lzma_check type); + +/// Update the check state +extern void lzma_check_update(lzma_check_state *check, lzma_check type, + const uint8_t *buf, size_t size); + +/// Finish the check calculation and store the result to check->buffer.u8. +extern void lzma_check_finish(lzma_check_state *check, lzma_check type); + + +#ifndef LZMA_SHA256FUNC + +/// Prepare SHA-256 state for new input. +extern void lzma_sha256_init(lzma_check_state *check); + +/// Update the SHA-256 hash state +extern void lzma_sha256_update( + const uint8_t *buf, size_t size, lzma_check_state *check); + +/// Finish the SHA-256 calculation and store the result to check->buffer.u8. +extern void lzma_sha256_finish(lzma_check_state *check); + + +#else + +static inline void +lzma_sha256_init(lzma_check_state *check) +{ + LZMA_SHA256FUNC(Init)(&check->state.sha256); +} + + +static inline void +lzma_sha256_update(const uint8_t *buf, size_t size, lzma_check_state *check) +{ +#if defined(HAVE_CC_SHA256_INIT) && SIZE_MAX > UINT32_MAX + // Darwin's CC_SHA256_Update takes uint32_t as the buffer size, + // so use a loop to support size_t. + while (size > UINT32_MAX) { + LZMA_SHA256FUNC(Update)(&check->state.sha256, buf, UINT32_MAX); + buf += UINT32_MAX; + size -= UINT32_MAX; + } +#endif + + LZMA_SHA256FUNC(Update)(&check->state.sha256, buf, size); +} + + +static inline void +lzma_sha256_finish(lzma_check_state *check) +{ + LZMA_SHA256FUNC(Final)(check->buffer.u8, &check->state.sha256); +} + +#endif + +#endif diff --git a/src/liblzma/check/crc32_arm64.h b/src/liblzma/check/crc32_arm64.h new file mode 100644 index 0000000000..fb0e8f0105 --- /dev/null +++ b/src/liblzma/check/crc32_arm64.h @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc32_arm64.h +/// \brief CRC32 calculation with ARM64 optimization +// +// Authors: Chenxi Mao +// Jia Tan +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_CRC32_ARM64_H +#define LZMA_CRC32_ARM64_H + +// MSVC always has the CRC intrinsics available when building for ARM64 +// there is no need to include any header files. +#ifndef _MSC_VER +# include +#endif + +// If both versions are going to be built, we need runtime detection +// to check if the instructions are supported. +#if defined(CRC32_GENERIC) && defined(CRC32_ARCH_OPTIMIZED) +# if defined(HAVE_GETAUXVAL) || defined(HAVE_ELF_AUX_INFO) +# include +# elif defined(_WIN32) +# include +# elif defined(__APPLE__) && defined(HAVE_SYSCTLBYNAME) +# include +# endif +#endif + +// Some EDG-based compilers support ARM64 and define __GNUC__ +// (such as Nvidia's nvcc), but do not support function attributes. +// +// NOTE: Build systems check for this too, keep them in sync with this. +#if (defined(__GNUC__) || defined(__clang__)) && !defined(__EDG__) +# define crc_attr_target __attribute__((__target__("+crc"))) +#else +# define crc_attr_target +#endif + + +crc_attr_target +static uint32_t +crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc) +{ + crc = ~crc; + + if (size >= 8) { + // Align the input buffer because this was shown to be + // significantly faster than unaligned accesses. + const size_t align = (0 - (uintptr_t)buf) & 7; + + if (align & 1) + crc = __crc32b(crc, *buf++); + + if (align & 2) { + crc = __crc32h(crc, aligned_read16le(buf)); + buf += 2; + } + + if (align & 4) { + crc = __crc32w(crc, aligned_read32le(buf)); + buf += 4; + } + + size -= align; + + // Process 8 bytes at a time. The end point is determined by + // ignoring the least significant three bits of size to + // ensure we do not process past the bounds of the buffer. + // This guarantees that limit is a multiple of 8 and is + // strictly less than size. + for (const uint8_t *limit = buf + (size & ~(size_t)7); + buf < limit; buf += 8) + crc = __crc32d(crc, aligned_read64le(buf)); + + size &= 7; + } + + // Process the remaining bytes that are not 8 byte aligned. + if (size & 4) { + crc = __crc32w(crc, aligned_read32le(buf)); + buf += 4; + } + + if (size & 2) { + crc = __crc32h(crc, aligned_read16le(buf)); + buf += 2; + } + + if (size & 1) + crc = __crc32b(crc, *buf); + + return ~crc; +} + + +#if defined(CRC32_GENERIC) && defined(CRC32_ARCH_OPTIMIZED) +static inline bool +is_arch_extension_supported(void) +{ +#if defined(HAVE_GETAUXVAL) + return (getauxval(AT_HWCAP) & HWCAP_CRC32) != 0; + +#elif defined(HAVE_ELF_AUX_INFO) + unsigned long feature_flags; + + if (elf_aux_info(AT_HWCAP, &feature_flags, sizeof(feature_flags)) != 0) + return false; + + return (feature_flags & HWCAP_CRC32) != 0; + +#elif defined(_WIN32) + return IsProcessorFeaturePresent( + PF_ARM_V8_CRC32_INSTRUCTIONS_AVAILABLE); + +#elif defined(__APPLE__) && defined(HAVE_SYSCTLBYNAME) + int has_crc32 = 0; + size_t size = sizeof(has_crc32); + + // The sysctlbyname() function requires a string identifier for the + // CPU feature it tests. The Apple documentation lists the string + // "hw.optional.armv8_crc32", which can be found here: + // https://developer.apple.com/documentation/kernel/1387446-sysctlbyname/determining_instruction_set_characteristics#3915619 + if (sysctlbyname("hw.optional.armv8_crc32", &has_crc32, + &size, NULL, 0) != 0) + return false; + + return has_crc32; + +#else + // If a runtime detection method cannot be found, then this must + // be a compile time error. The checks in crc_common.h should ensure + // a runtime detection method is always found if this function is + // built. It would be possible to just return false here, but this + // is inefficient for binary size and runtime since only the generic + // method could ever be used. +# error Runtime detection method unavailable. +#endif +} +#endif + +#endif // LZMA_CRC32_ARM64_H diff --git a/src/liblzma/check/crc32_fast.c b/src/liblzma/check/crc32_fast.c new file mode 100644 index 0000000000..0d5c3c7dd9 --- /dev/null +++ b/src/liblzma/check/crc32_fast.c @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc32.c +/// \brief CRC32 calculation +// +// Authors: Lasse Collin +// Ilya Kurdyukov +// +/////////////////////////////////////////////////////////////////////////////// + +#include "check.h" +#include "crc_common.h" + +#if defined(CRC_X86_CLMUL) +# define BUILDING_CRC_CLMUL 32 +# include "crc_x86_clmul.h" +#elif defined(CRC32_ARM64) +# include "crc32_arm64.h" +#elif defined(CRC32_LOONGARCH) +# include "crc32_loongarch.h" +#endif + +#ifdef CRC32_GENERIC + +/////////////////// +// Generic CRC32 // +/////////////////// + +#ifdef WORDS_BIGENDIAN +# include "crc32_table_be.h" +#else +# include "crc32_table_le.h" +#endif + + +#ifdef HAVE_CRC_X86_ASM +extern uint32_t lzma_crc32_generic( + const uint8_t *buf, size_t size, uint32_t crc); +#else +static uint32_t +lzma_crc32_generic(const uint8_t *buf, size_t size, uint32_t crc) +{ + crc = ~crc; + +#ifdef WORDS_BIGENDIAN + crc = byteswap32(crc); +#endif + + if (size > 8) { + // Fix the alignment, if needed. The if statement above + // ensures that this won't read past the end of buf[]. + while ((uintptr_t)(buf) & 7) { + crc = lzma_crc32_table[0][*buf++ ^ A(crc)] ^ S8(crc); + --size; + } + + // Calculate the position where to stop. + const uint8_t *const limit = buf + (size & ~(size_t)(7)); + + // Calculate how many bytes must be calculated separately + // before returning the result. + size &= (size_t)(7); + + // Calculate the CRC32 using the slice-by-eight algorithm. + while (buf < limit) { + crc ^= aligned_read32ne(buf); + buf += 4; + + crc = lzma_crc32_table[7][A(crc)] + ^ lzma_crc32_table[6][B(crc)] + ^ lzma_crc32_table[5][C(crc)] + ^ lzma_crc32_table[4][D(crc)]; + + const uint32_t tmp = aligned_read32ne(buf); + buf += 4; + + // At least with some compilers, it is critical for + // performance, that the crc variable is XORed + // between the two table-lookup pairs. + crc = lzma_crc32_table[3][A(tmp)] + ^ lzma_crc32_table[2][B(tmp)] + ^ crc + ^ lzma_crc32_table[1][C(tmp)] + ^ lzma_crc32_table[0][D(tmp)]; + } + } + + while (size-- != 0) + crc = lzma_crc32_table[0][*buf++ ^ A(crc)] ^ S8(crc); + +#ifdef WORDS_BIGENDIAN + crc = byteswap32(crc); +#endif + + return ~crc; +} +#endif // HAVE_CRC_X86_ASM +#endif // CRC32_GENERIC + + +#if defined(CRC32_GENERIC) && defined(CRC32_ARCH_OPTIMIZED) + +////////////////////////// +// Function dispatching // +////////////////////////// + +// If both the generic and arch-optimized implementations are built, then +// the function to use is selected at runtime because the system running +// the binary might not have the arch-specific instruction set extension(s) +// available. The dispatch methods in order of priority: +// +// 1. Constructor. This method uses __attribute__((__constructor__)) to +// set crc32_func at load time. This avoids extra computation (and any +// unlikely threading bugs) on the first call to lzma_crc32() to decide +// which implementation should be used. +// +// 2. First Call Resolution. On the very first call to lzma_crc32(), the +// call will be directed to crc32_dispatch() instead. This will set the +// appropriate implementation function and will not be called again. +// This method does not use any kind of locking but is safe because if +// multiple threads run the dispatcher simultaneously then they will all +// set crc32_func to the same value. + +typedef uint32_t (*crc32_func_type)( + const uint8_t *buf, size_t size, uint32_t crc); + +// This resolver is shared between all dispatch methods. +static crc32_func_type +crc32_resolve(void) +{ + return is_arch_extension_supported() + ? &crc32_arch_optimized : &lzma_crc32_generic; +} + + +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +// Constructor method. +# define CRC32_SET_FUNC_ATTR __attribute__((__constructor__)) +static crc32_func_type crc32_func; +#else +// First Call Resolution method. +# define CRC32_SET_FUNC_ATTR +static uint32_t crc32_dispatch(const uint8_t *buf, size_t size, uint32_t crc); +static crc32_func_type crc32_func = &crc32_dispatch; +#endif + +CRC32_SET_FUNC_ATTR +static void +crc32_set_func(void) +{ + crc32_func = crc32_resolve(); + return; +} + +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +static uint32_t +crc32_dispatch(const uint8_t *buf, size_t size, uint32_t crc) +{ + // When __attribute__((__constructor__)) isn't supported, set the + // function pointer without any locking. If multiple threads run + // the detection code in parallel, they will all end up setting + // the pointer to the same value. This avoids the use of + // mythread_once() on every call to lzma_crc32() but this likely + // isn't strictly standards compliant. Let's change it if it breaks. + crc32_set_func(); + return crc32_func(buf, size, crc); +} + +#endif +#endif + + +extern LZMA_API(uint32_t) +lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc) +{ +#if defined(CRC32_GENERIC) && defined(CRC32_ARCH_OPTIMIZED) +/* +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR + // See crc32_dispatch(). This would be the alternative which uses + // locking and doesn't use crc32_dispatch(). Note that on Windows + // this method needs Vista threads. + mythread_once(crc64_set_func); +#endif +*/ + return crc32_func(buf, size, crc); + +#elif defined(CRC32_ARCH_OPTIMIZED) + return crc32_arch_optimized(buf, size, crc); + +#else + return lzma_crc32_generic(buf, size, crc); +#endif +} diff --git a/src/liblzma/check/crc32_loongarch.h b/src/liblzma/check/crc32_loongarch.h new file mode 100644 index 0000000000..ec738b83d7 --- /dev/null +++ b/src/liblzma/check/crc32_loongarch.h @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc32_loongarch.h +/// \brief CRC32 calculation with LoongArch optimization +// +// Authors: Xi Ruoyao +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_CRC32_LOONGARCH_H +#define LZMA_CRC32_LOONGARCH_H + +#include + + +static uint32_t +crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc_unsigned) +{ + int32_t crc = (int32_t)~crc_unsigned; + + if (size >= 8) { + const size_t align = (0 - (uintptr_t)buf) & 7; + + if (align & 1) + crc = __crc_w_b_w((int8_t)*buf++, crc); + + if (align & 2) { + crc = __crc_w_h_w((int16_t)aligned_read16le(buf), crc); + buf += 2; + } + + if (align & 4) { + crc = __crc_w_w_w((int32_t)aligned_read32le(buf), crc); + buf += 4; + } + + size -= align; + + for (const uint8_t *limit = buf + (size & ~(size_t)7); + buf < limit; buf += 8) + crc = __crc_w_d_w((int64_t)aligned_read64le(buf), crc); + + size &= 7; + } + + if (size & 4) { + crc = __crc_w_w_w((int32_t)aligned_read32le(buf), crc); + buf += 4; + } + + if (size & 2) { + crc = __crc_w_h_w((int16_t)aligned_read16le(buf), crc); + buf += 2; + } + + if (size & 1) + crc = __crc_w_b_w((int8_t)*buf, crc); + + return (uint32_t)~crc; +} + +#endif // LZMA_CRC32_LOONGARCH_H diff --git a/src/liblzma/check/crc32_small.c b/src/liblzma/check/crc32_small.c new file mode 100644 index 0000000000..4a62830c80 --- /dev/null +++ b/src/liblzma/check/crc32_small.c @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc32_small.c +/// \brief CRC32 calculation (size-optimized) +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "check.h" +#include "crc_common.h" + + +// The table is used by the LZ encoder too, thus it's not static like +// in crc64_small.c. +uint32_t lzma_crc32_table[1][256]; + + +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +__attribute__((__constructor__)) +#endif +static void +crc32_init(void) +{ + static const uint32_t poly32 = UINT32_C(0xEDB88320); + + for (size_t b = 0; b < 256; ++b) { + uint32_t r = b; + for (size_t i = 0; i < 8; ++i) { + if (r & 1) + r = (r >> 1) ^ poly32; + else + r >>= 1; + } + + lzma_crc32_table[0][b] = r; + } + + return; +} + + +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +extern void +lzma_crc32_init(void) +{ + mythread_once(crc32_init); + return; +} +#endif + + +extern LZMA_API(uint32_t) +lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc) +{ +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR + lzma_crc32_init(); +#endif + + crc = ~crc; + + while (size != 0) { + crc = lzma_crc32_table[0][*buf++ ^ (crc & 0xFF)] ^ (crc >> 8); + --size; + } + + return ~crc; +} diff --git a/src/liblzma/check/crc32_table_be.h b/src/liblzma/check/crc32_table_be.h new file mode 100644 index 0000000000..505c23074c --- /dev/null +++ b/src/liblzma/check/crc32_table_be.h @@ -0,0 +1,527 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by crc32_tablegen.c. + +const uint32_t lzma_crc32_table[8][256] = { + { + 0x00000000, 0x96300777, 0x2C610EEE, 0xBA510999, + 0x19C46D07, 0x8FF46A70, 0x35A563E9, 0xA395649E, + 0x3288DB0E, 0xA4B8DC79, 0x1EE9D5E0, 0x88D9D297, + 0x2B4CB609, 0xBD7CB17E, 0x072DB8E7, 0x911DBF90, + 0x6410B71D, 0xF220B06A, 0x4871B9F3, 0xDE41BE84, + 0x7DD4DA1A, 0xEBE4DD6D, 0x51B5D4F4, 0xC785D383, + 0x56986C13, 0xC0A86B64, 0x7AF962FD, 0xECC9658A, + 0x4F5C0114, 0xD96C0663, 0x633D0FFA, 0xF50D088D, + 0xC8206E3B, 0x5E10694C, 0xE44160D5, 0x727167A2, + 0xD1E4033C, 0x47D4044B, 0xFD850DD2, 0x6BB50AA5, + 0xFAA8B535, 0x6C98B242, 0xD6C9BBDB, 0x40F9BCAC, + 0xE36CD832, 0x755CDF45, 0xCF0DD6DC, 0x593DD1AB, + 0xAC30D926, 0x3A00DE51, 0x8051D7C8, 0x1661D0BF, + 0xB5F4B421, 0x23C4B356, 0x9995BACF, 0x0FA5BDB8, + 0x9EB80228, 0x0888055F, 0xB2D90CC6, 0x24E90BB1, + 0x877C6F2F, 0x114C6858, 0xAB1D61C1, 0x3D2D66B6, + 0x9041DC76, 0x0671DB01, 0xBC20D298, 0x2A10D5EF, + 0x8985B171, 0x1FB5B606, 0xA5E4BF9F, 0x33D4B8E8, + 0xA2C90778, 0x34F9000F, 0x8EA80996, 0x18980EE1, + 0xBB0D6A7F, 0x2D3D6D08, 0x976C6491, 0x015C63E6, + 0xF4516B6B, 0x62616C1C, 0xD8306585, 0x4E0062F2, + 0xED95066C, 0x7BA5011B, 0xC1F40882, 0x57C40FF5, + 0xC6D9B065, 0x50E9B712, 0xEAB8BE8B, 0x7C88B9FC, + 0xDF1DDD62, 0x492DDA15, 0xF37CD38C, 0x654CD4FB, + 0x5861B24D, 0xCE51B53A, 0x7400BCA3, 0xE230BBD4, + 0x41A5DF4A, 0xD795D83D, 0x6DC4D1A4, 0xFBF4D6D3, + 0x6AE96943, 0xFCD96E34, 0x468867AD, 0xD0B860DA, + 0x732D0444, 0xE51D0333, 0x5F4C0AAA, 0xC97C0DDD, + 0x3C710550, 0xAA410227, 0x10100BBE, 0x86200CC9, + 0x25B56857, 0xB3856F20, 0x09D466B9, 0x9FE461CE, + 0x0EF9DE5E, 0x98C9D929, 0x2298D0B0, 0xB4A8D7C7, + 0x173DB359, 0x810DB42E, 0x3B5CBDB7, 0xAD6CBAC0, + 0x2083B8ED, 0xB6B3BF9A, 0x0CE2B603, 0x9AD2B174, + 0x3947D5EA, 0xAF77D29D, 0x1526DB04, 0x8316DC73, + 0x120B63E3, 0x843B6494, 0x3E6A6D0D, 0xA85A6A7A, + 0x0BCF0EE4, 0x9DFF0993, 0x27AE000A, 0xB19E077D, + 0x44930FF0, 0xD2A30887, 0x68F2011E, 0xFEC20669, + 0x5D5762F7, 0xCB676580, 0x71366C19, 0xE7066B6E, + 0x761BD4FE, 0xE02BD389, 0x5A7ADA10, 0xCC4ADD67, + 0x6FDFB9F9, 0xF9EFBE8E, 0x43BEB717, 0xD58EB060, + 0xE8A3D6D6, 0x7E93D1A1, 0xC4C2D838, 0x52F2DF4F, + 0xF167BBD1, 0x6757BCA6, 0xDD06B53F, 0x4B36B248, + 0xDA2B0DD8, 0x4C1B0AAF, 0xF64A0336, 0x607A0441, + 0xC3EF60DF, 0x55DF67A8, 0xEF8E6E31, 0x79BE6946, + 0x8CB361CB, 0x1A8366BC, 0xA0D26F25, 0x36E26852, + 0x95770CCC, 0x03470BBB, 0xB9160222, 0x2F260555, + 0xBE3BBAC5, 0x280BBDB2, 0x925AB42B, 0x046AB35C, + 0xA7FFD7C2, 0x31CFD0B5, 0x8B9ED92C, 0x1DAEDE5B, + 0xB0C2649B, 0x26F263EC, 0x9CA36A75, 0x0A936D02, + 0xA906099C, 0x3F360EEB, 0x85670772, 0x13570005, + 0x824ABF95, 0x147AB8E2, 0xAE2BB17B, 0x381BB60C, + 0x9B8ED292, 0x0DBED5E5, 0xB7EFDC7C, 0x21DFDB0B, + 0xD4D2D386, 0x42E2D4F1, 0xF8B3DD68, 0x6E83DA1F, + 0xCD16BE81, 0x5B26B9F6, 0xE177B06F, 0x7747B718, + 0xE65A0888, 0x706A0FFF, 0xCA3B0666, 0x5C0B0111, + 0xFF9E658F, 0x69AE62F8, 0xD3FF6B61, 0x45CF6C16, + 0x78E20AA0, 0xEED20DD7, 0x5483044E, 0xC2B30339, + 0x612667A7, 0xF71660D0, 0x4D476949, 0xDB776E3E, + 0x4A6AD1AE, 0xDC5AD6D9, 0x660BDF40, 0xF03BD837, + 0x53AEBCA9, 0xC59EBBDE, 0x7FCFB247, 0xE9FFB530, + 0x1CF2BDBD, 0x8AC2BACA, 0x3093B353, 0xA6A3B424, + 0x0536D0BA, 0x9306D7CD, 0x2957DE54, 0xBF67D923, + 0x2E7A66B3, 0xB84A61C4, 0x021B685D, 0x942B6F2A, + 0x37BE0BB4, 0xA18E0CC3, 0x1BDF055A, 0x8DEF022D + }, { + 0x00000000, 0x41311B19, 0x82623632, 0xC3532D2B, + 0x04C56C64, 0x45F4777D, 0x86A75A56, 0xC796414F, + 0x088AD9C8, 0x49BBC2D1, 0x8AE8EFFA, 0xCBD9F4E3, + 0x0C4FB5AC, 0x4D7EAEB5, 0x8E2D839E, 0xCF1C9887, + 0x5112C24A, 0x1023D953, 0xD370F478, 0x9241EF61, + 0x55D7AE2E, 0x14E6B537, 0xD7B5981C, 0x96848305, + 0x59981B82, 0x18A9009B, 0xDBFA2DB0, 0x9ACB36A9, + 0x5D5D77E6, 0x1C6C6CFF, 0xDF3F41D4, 0x9E0E5ACD, + 0xA2248495, 0xE3159F8C, 0x2046B2A7, 0x6177A9BE, + 0xA6E1E8F1, 0xE7D0F3E8, 0x2483DEC3, 0x65B2C5DA, + 0xAAAE5D5D, 0xEB9F4644, 0x28CC6B6F, 0x69FD7076, + 0xAE6B3139, 0xEF5A2A20, 0x2C09070B, 0x6D381C12, + 0xF33646DF, 0xB2075DC6, 0x715470ED, 0x30656BF4, + 0xF7F32ABB, 0xB6C231A2, 0x75911C89, 0x34A00790, + 0xFBBC9F17, 0xBA8D840E, 0x79DEA925, 0x38EFB23C, + 0xFF79F373, 0xBE48E86A, 0x7D1BC541, 0x3C2ADE58, + 0x054F79F0, 0x447E62E9, 0x872D4FC2, 0xC61C54DB, + 0x018A1594, 0x40BB0E8D, 0x83E823A6, 0xC2D938BF, + 0x0DC5A038, 0x4CF4BB21, 0x8FA7960A, 0xCE968D13, + 0x0900CC5C, 0x4831D745, 0x8B62FA6E, 0xCA53E177, + 0x545DBBBA, 0x156CA0A3, 0xD63F8D88, 0x970E9691, + 0x5098D7DE, 0x11A9CCC7, 0xD2FAE1EC, 0x93CBFAF5, + 0x5CD76272, 0x1DE6796B, 0xDEB55440, 0x9F844F59, + 0x58120E16, 0x1923150F, 0xDA703824, 0x9B41233D, + 0xA76BFD65, 0xE65AE67C, 0x2509CB57, 0x6438D04E, + 0xA3AE9101, 0xE29F8A18, 0x21CCA733, 0x60FDBC2A, + 0xAFE124AD, 0xEED03FB4, 0x2D83129F, 0x6CB20986, + 0xAB2448C9, 0xEA1553D0, 0x29467EFB, 0x687765E2, + 0xF6793F2F, 0xB7482436, 0x741B091D, 0x352A1204, + 0xF2BC534B, 0xB38D4852, 0x70DE6579, 0x31EF7E60, + 0xFEF3E6E7, 0xBFC2FDFE, 0x7C91D0D5, 0x3DA0CBCC, + 0xFA368A83, 0xBB07919A, 0x7854BCB1, 0x3965A7A8, + 0x4B98833B, 0x0AA99822, 0xC9FAB509, 0x88CBAE10, + 0x4F5DEF5F, 0x0E6CF446, 0xCD3FD96D, 0x8C0EC274, + 0x43125AF3, 0x022341EA, 0xC1706CC1, 0x804177D8, + 0x47D73697, 0x06E62D8E, 0xC5B500A5, 0x84841BBC, + 0x1A8A4171, 0x5BBB5A68, 0x98E87743, 0xD9D96C5A, + 0x1E4F2D15, 0x5F7E360C, 0x9C2D1B27, 0xDD1C003E, + 0x120098B9, 0x533183A0, 0x9062AE8B, 0xD153B592, + 0x16C5F4DD, 0x57F4EFC4, 0x94A7C2EF, 0xD596D9F6, + 0xE9BC07AE, 0xA88D1CB7, 0x6BDE319C, 0x2AEF2A85, + 0xED796BCA, 0xAC4870D3, 0x6F1B5DF8, 0x2E2A46E1, + 0xE136DE66, 0xA007C57F, 0x6354E854, 0x2265F34D, + 0xE5F3B202, 0xA4C2A91B, 0x67918430, 0x26A09F29, + 0xB8AEC5E4, 0xF99FDEFD, 0x3ACCF3D6, 0x7BFDE8CF, + 0xBC6BA980, 0xFD5AB299, 0x3E099FB2, 0x7F3884AB, + 0xB0241C2C, 0xF1150735, 0x32462A1E, 0x73773107, + 0xB4E17048, 0xF5D06B51, 0x3683467A, 0x77B25D63, + 0x4ED7FACB, 0x0FE6E1D2, 0xCCB5CCF9, 0x8D84D7E0, + 0x4A1296AF, 0x0B238DB6, 0xC870A09D, 0x8941BB84, + 0x465D2303, 0x076C381A, 0xC43F1531, 0x850E0E28, + 0x42984F67, 0x03A9547E, 0xC0FA7955, 0x81CB624C, + 0x1FC53881, 0x5EF42398, 0x9DA70EB3, 0xDC9615AA, + 0x1B0054E5, 0x5A314FFC, 0x996262D7, 0xD85379CE, + 0x174FE149, 0x567EFA50, 0x952DD77B, 0xD41CCC62, + 0x138A8D2D, 0x52BB9634, 0x91E8BB1F, 0xD0D9A006, + 0xECF37E5E, 0xADC26547, 0x6E91486C, 0x2FA05375, + 0xE836123A, 0xA9070923, 0x6A542408, 0x2B653F11, + 0xE479A796, 0xA548BC8F, 0x661B91A4, 0x272A8ABD, + 0xE0BCCBF2, 0xA18DD0EB, 0x62DEFDC0, 0x23EFE6D9, + 0xBDE1BC14, 0xFCD0A70D, 0x3F838A26, 0x7EB2913F, + 0xB924D070, 0xF815CB69, 0x3B46E642, 0x7A77FD5B, + 0xB56B65DC, 0xF45A7EC5, 0x370953EE, 0x763848F7, + 0xB1AE09B8, 0xF09F12A1, 0x33CC3F8A, 0x72FD2493 + }, { + 0x00000000, 0x376AC201, 0x6ED48403, 0x59BE4602, + 0xDCA80907, 0xEBC2CB06, 0xB27C8D04, 0x85164F05, + 0xB851130E, 0x8F3BD10F, 0xD685970D, 0xE1EF550C, + 0x64F91A09, 0x5393D808, 0x0A2D9E0A, 0x3D475C0B, + 0x70A3261C, 0x47C9E41D, 0x1E77A21F, 0x291D601E, + 0xAC0B2F1B, 0x9B61ED1A, 0xC2DFAB18, 0xF5B56919, + 0xC8F23512, 0xFF98F713, 0xA626B111, 0x914C7310, + 0x145A3C15, 0x2330FE14, 0x7A8EB816, 0x4DE47A17, + 0xE0464D38, 0xD72C8F39, 0x8E92C93B, 0xB9F80B3A, + 0x3CEE443F, 0x0B84863E, 0x523AC03C, 0x6550023D, + 0x58175E36, 0x6F7D9C37, 0x36C3DA35, 0x01A91834, + 0x84BF5731, 0xB3D59530, 0xEA6BD332, 0xDD011133, + 0x90E56B24, 0xA78FA925, 0xFE31EF27, 0xC95B2D26, + 0x4C4D6223, 0x7B27A022, 0x2299E620, 0x15F32421, + 0x28B4782A, 0x1FDEBA2B, 0x4660FC29, 0x710A3E28, + 0xF41C712D, 0xC376B32C, 0x9AC8F52E, 0xADA2372F, + 0xC08D9A70, 0xF7E75871, 0xAE591E73, 0x9933DC72, + 0x1C259377, 0x2B4F5176, 0x72F11774, 0x459BD575, + 0x78DC897E, 0x4FB64B7F, 0x16080D7D, 0x2162CF7C, + 0xA4748079, 0x931E4278, 0xCAA0047A, 0xFDCAC67B, + 0xB02EBC6C, 0x87447E6D, 0xDEFA386F, 0xE990FA6E, + 0x6C86B56B, 0x5BEC776A, 0x02523168, 0x3538F369, + 0x087FAF62, 0x3F156D63, 0x66AB2B61, 0x51C1E960, + 0xD4D7A665, 0xE3BD6464, 0xBA032266, 0x8D69E067, + 0x20CBD748, 0x17A11549, 0x4E1F534B, 0x7975914A, + 0xFC63DE4F, 0xCB091C4E, 0x92B75A4C, 0xA5DD984D, + 0x989AC446, 0xAFF00647, 0xF64E4045, 0xC1248244, + 0x4432CD41, 0x73580F40, 0x2AE64942, 0x1D8C8B43, + 0x5068F154, 0x67023355, 0x3EBC7557, 0x09D6B756, + 0x8CC0F853, 0xBBAA3A52, 0xE2147C50, 0xD57EBE51, + 0xE839E25A, 0xDF53205B, 0x86ED6659, 0xB187A458, + 0x3491EB5D, 0x03FB295C, 0x5A456F5E, 0x6D2FAD5F, + 0x801B35E1, 0xB771F7E0, 0xEECFB1E2, 0xD9A573E3, + 0x5CB33CE6, 0x6BD9FEE7, 0x3267B8E5, 0x050D7AE4, + 0x384A26EF, 0x0F20E4EE, 0x569EA2EC, 0x61F460ED, + 0xE4E22FE8, 0xD388EDE9, 0x8A36ABEB, 0xBD5C69EA, + 0xF0B813FD, 0xC7D2D1FC, 0x9E6C97FE, 0xA90655FF, + 0x2C101AFA, 0x1B7AD8FB, 0x42C49EF9, 0x75AE5CF8, + 0x48E900F3, 0x7F83C2F2, 0x263D84F0, 0x115746F1, + 0x944109F4, 0xA32BCBF5, 0xFA958DF7, 0xCDFF4FF6, + 0x605D78D9, 0x5737BAD8, 0x0E89FCDA, 0x39E33EDB, + 0xBCF571DE, 0x8B9FB3DF, 0xD221F5DD, 0xE54B37DC, + 0xD80C6BD7, 0xEF66A9D6, 0xB6D8EFD4, 0x81B22DD5, + 0x04A462D0, 0x33CEA0D1, 0x6A70E6D3, 0x5D1A24D2, + 0x10FE5EC5, 0x27949CC4, 0x7E2ADAC6, 0x494018C7, + 0xCC5657C2, 0xFB3C95C3, 0xA282D3C1, 0x95E811C0, + 0xA8AF4DCB, 0x9FC58FCA, 0xC67BC9C8, 0xF1110BC9, + 0x740744CC, 0x436D86CD, 0x1AD3C0CF, 0x2DB902CE, + 0x4096AF91, 0x77FC6D90, 0x2E422B92, 0x1928E993, + 0x9C3EA696, 0xAB546497, 0xF2EA2295, 0xC580E094, + 0xF8C7BC9F, 0xCFAD7E9E, 0x9613389C, 0xA179FA9D, + 0x246FB598, 0x13057799, 0x4ABB319B, 0x7DD1F39A, + 0x3035898D, 0x075F4B8C, 0x5EE10D8E, 0x698BCF8F, + 0xEC9D808A, 0xDBF7428B, 0x82490489, 0xB523C688, + 0x88649A83, 0xBF0E5882, 0xE6B01E80, 0xD1DADC81, + 0x54CC9384, 0x63A65185, 0x3A181787, 0x0D72D586, + 0xA0D0E2A9, 0x97BA20A8, 0xCE0466AA, 0xF96EA4AB, + 0x7C78EBAE, 0x4B1229AF, 0x12AC6FAD, 0x25C6ADAC, + 0x1881F1A7, 0x2FEB33A6, 0x765575A4, 0x413FB7A5, + 0xC429F8A0, 0xF3433AA1, 0xAAFD7CA3, 0x9D97BEA2, + 0xD073C4B5, 0xE71906B4, 0xBEA740B6, 0x89CD82B7, + 0x0CDBCDB2, 0x3BB10FB3, 0x620F49B1, 0x55658BB0, + 0x6822D7BB, 0x5F4815BA, 0x06F653B8, 0x319C91B9, + 0xB48ADEBC, 0x83E01CBD, 0xDA5E5ABF, 0xED3498BE + }, { + 0x00000000, 0x6567BCB8, 0x8BC809AA, 0xEEAFB512, + 0x5797628F, 0x32F0DE37, 0xDC5F6B25, 0xB938D79D, + 0xEF28B4C5, 0x8A4F087D, 0x64E0BD6F, 0x018701D7, + 0xB8BFD64A, 0xDDD86AF2, 0x3377DFE0, 0x56106358, + 0x9F571950, 0xFA30A5E8, 0x149F10FA, 0x71F8AC42, + 0xC8C07BDF, 0xADA7C767, 0x43087275, 0x266FCECD, + 0x707FAD95, 0x1518112D, 0xFBB7A43F, 0x9ED01887, + 0x27E8CF1A, 0x428F73A2, 0xAC20C6B0, 0xC9477A08, + 0x3EAF32A0, 0x5BC88E18, 0xB5673B0A, 0xD00087B2, + 0x6938502F, 0x0C5FEC97, 0xE2F05985, 0x8797E53D, + 0xD1878665, 0xB4E03ADD, 0x5A4F8FCF, 0x3F283377, + 0x8610E4EA, 0xE3775852, 0x0DD8ED40, 0x68BF51F8, + 0xA1F82BF0, 0xC49F9748, 0x2A30225A, 0x4F579EE2, + 0xF66F497F, 0x9308F5C7, 0x7DA740D5, 0x18C0FC6D, + 0x4ED09F35, 0x2BB7238D, 0xC518969F, 0xA07F2A27, + 0x1947FDBA, 0x7C204102, 0x928FF410, 0xF7E848A8, + 0x3D58149B, 0x583FA823, 0xB6901D31, 0xD3F7A189, + 0x6ACF7614, 0x0FA8CAAC, 0xE1077FBE, 0x8460C306, + 0xD270A05E, 0xB7171CE6, 0x59B8A9F4, 0x3CDF154C, + 0x85E7C2D1, 0xE0807E69, 0x0E2FCB7B, 0x6B4877C3, + 0xA20F0DCB, 0xC768B173, 0x29C70461, 0x4CA0B8D9, + 0xF5986F44, 0x90FFD3FC, 0x7E5066EE, 0x1B37DA56, + 0x4D27B90E, 0x284005B6, 0xC6EFB0A4, 0xA3880C1C, + 0x1AB0DB81, 0x7FD76739, 0x9178D22B, 0xF41F6E93, + 0x03F7263B, 0x66909A83, 0x883F2F91, 0xED589329, + 0x546044B4, 0x3107F80C, 0xDFA84D1E, 0xBACFF1A6, + 0xECDF92FE, 0x89B82E46, 0x67179B54, 0x027027EC, + 0xBB48F071, 0xDE2F4CC9, 0x3080F9DB, 0x55E74563, + 0x9CA03F6B, 0xF9C783D3, 0x176836C1, 0x720F8A79, + 0xCB375DE4, 0xAE50E15C, 0x40FF544E, 0x2598E8F6, + 0x73888BAE, 0x16EF3716, 0xF8408204, 0x9D273EBC, + 0x241FE921, 0x41785599, 0xAFD7E08B, 0xCAB05C33, + 0x3BB659ED, 0x5ED1E555, 0xB07E5047, 0xD519ECFF, + 0x6C213B62, 0x094687DA, 0xE7E932C8, 0x828E8E70, + 0xD49EED28, 0xB1F95190, 0x5F56E482, 0x3A31583A, + 0x83098FA7, 0xE66E331F, 0x08C1860D, 0x6DA63AB5, + 0xA4E140BD, 0xC186FC05, 0x2F294917, 0x4A4EF5AF, + 0xF3762232, 0x96119E8A, 0x78BE2B98, 0x1DD99720, + 0x4BC9F478, 0x2EAE48C0, 0xC001FDD2, 0xA566416A, + 0x1C5E96F7, 0x79392A4F, 0x97969F5D, 0xF2F123E5, + 0x05196B4D, 0x607ED7F5, 0x8ED162E7, 0xEBB6DE5F, + 0x528E09C2, 0x37E9B57A, 0xD9460068, 0xBC21BCD0, + 0xEA31DF88, 0x8F566330, 0x61F9D622, 0x049E6A9A, + 0xBDA6BD07, 0xD8C101BF, 0x366EB4AD, 0x53090815, + 0x9A4E721D, 0xFF29CEA5, 0x11867BB7, 0x74E1C70F, + 0xCDD91092, 0xA8BEAC2A, 0x46111938, 0x2376A580, + 0x7566C6D8, 0x10017A60, 0xFEAECF72, 0x9BC973CA, + 0x22F1A457, 0x479618EF, 0xA939ADFD, 0xCC5E1145, + 0x06EE4D76, 0x6389F1CE, 0x8D2644DC, 0xE841F864, + 0x51792FF9, 0x341E9341, 0xDAB12653, 0xBFD69AEB, + 0xE9C6F9B3, 0x8CA1450B, 0x620EF019, 0x07694CA1, + 0xBE519B3C, 0xDB362784, 0x35999296, 0x50FE2E2E, + 0x99B95426, 0xFCDEE89E, 0x12715D8C, 0x7716E134, + 0xCE2E36A9, 0xAB498A11, 0x45E63F03, 0x208183BB, + 0x7691E0E3, 0x13F65C5B, 0xFD59E949, 0x983E55F1, + 0x2106826C, 0x44613ED4, 0xAACE8BC6, 0xCFA9377E, + 0x38417FD6, 0x5D26C36E, 0xB389767C, 0xD6EECAC4, + 0x6FD61D59, 0x0AB1A1E1, 0xE41E14F3, 0x8179A84B, + 0xD769CB13, 0xB20E77AB, 0x5CA1C2B9, 0x39C67E01, + 0x80FEA99C, 0xE5991524, 0x0B36A036, 0x6E511C8E, + 0xA7166686, 0xC271DA3E, 0x2CDE6F2C, 0x49B9D394, + 0xF0810409, 0x95E6B8B1, 0x7B490DA3, 0x1E2EB11B, + 0x483ED243, 0x2D596EFB, 0xC3F6DBE9, 0xA6916751, + 0x1FA9B0CC, 0x7ACE0C74, 0x9461B966, 0xF10605DE + }, { + 0x00000000, 0xB029603D, 0x6053C07A, 0xD07AA047, + 0xC0A680F5, 0x708FE0C8, 0xA0F5408F, 0x10DC20B2, + 0xC14B7030, 0x7162100D, 0xA118B04A, 0x1131D077, + 0x01EDF0C5, 0xB1C490F8, 0x61BE30BF, 0xD1975082, + 0x8297E060, 0x32BE805D, 0xE2C4201A, 0x52ED4027, + 0x42316095, 0xF21800A8, 0x2262A0EF, 0x924BC0D2, + 0x43DC9050, 0xF3F5F06D, 0x238F502A, 0x93A63017, + 0x837A10A5, 0x33537098, 0xE329D0DF, 0x5300B0E2, + 0x042FC1C1, 0xB406A1FC, 0x647C01BB, 0xD4556186, + 0xC4894134, 0x74A02109, 0xA4DA814E, 0x14F3E173, + 0xC564B1F1, 0x754DD1CC, 0xA537718B, 0x151E11B6, + 0x05C23104, 0xB5EB5139, 0x6591F17E, 0xD5B89143, + 0x86B821A1, 0x3691419C, 0xE6EBE1DB, 0x56C281E6, + 0x461EA154, 0xF637C169, 0x264D612E, 0x96640113, + 0x47F35191, 0xF7DA31AC, 0x27A091EB, 0x9789F1D6, + 0x8755D164, 0x377CB159, 0xE706111E, 0x572F7123, + 0x4958F358, 0xF9719365, 0x290B3322, 0x9922531F, + 0x89FE73AD, 0x39D71390, 0xE9ADB3D7, 0x5984D3EA, + 0x88138368, 0x383AE355, 0xE8404312, 0x5869232F, + 0x48B5039D, 0xF89C63A0, 0x28E6C3E7, 0x98CFA3DA, + 0xCBCF1338, 0x7BE67305, 0xAB9CD342, 0x1BB5B37F, + 0x0B6993CD, 0xBB40F3F0, 0x6B3A53B7, 0xDB13338A, + 0x0A846308, 0xBAAD0335, 0x6AD7A372, 0xDAFEC34F, + 0xCA22E3FD, 0x7A0B83C0, 0xAA712387, 0x1A5843BA, + 0x4D773299, 0xFD5E52A4, 0x2D24F2E3, 0x9D0D92DE, + 0x8DD1B26C, 0x3DF8D251, 0xED827216, 0x5DAB122B, + 0x8C3C42A9, 0x3C152294, 0xEC6F82D3, 0x5C46E2EE, + 0x4C9AC25C, 0xFCB3A261, 0x2CC90226, 0x9CE0621B, + 0xCFE0D2F9, 0x7FC9B2C4, 0xAFB31283, 0x1F9A72BE, + 0x0F46520C, 0xBF6F3231, 0x6F159276, 0xDF3CF24B, + 0x0EABA2C9, 0xBE82C2F4, 0x6EF862B3, 0xDED1028E, + 0xCE0D223C, 0x7E244201, 0xAE5EE246, 0x1E77827B, + 0x92B0E6B1, 0x2299868C, 0xF2E326CB, 0x42CA46F6, + 0x52166644, 0xE23F0679, 0x3245A63E, 0x826CC603, + 0x53FB9681, 0xE3D2F6BC, 0x33A856FB, 0x838136C6, + 0x935D1674, 0x23747649, 0xF30ED60E, 0x4327B633, + 0x102706D1, 0xA00E66EC, 0x7074C6AB, 0xC05DA696, + 0xD0818624, 0x60A8E619, 0xB0D2465E, 0x00FB2663, + 0xD16C76E1, 0x614516DC, 0xB13FB69B, 0x0116D6A6, + 0x11CAF614, 0xA1E39629, 0x7199366E, 0xC1B05653, + 0x969F2770, 0x26B6474D, 0xF6CCE70A, 0x46E58737, + 0x5639A785, 0xE610C7B8, 0x366A67FF, 0x864307C2, + 0x57D45740, 0xE7FD377D, 0x3787973A, 0x87AEF707, + 0x9772D7B5, 0x275BB788, 0xF72117CF, 0x470877F2, + 0x1408C710, 0xA421A72D, 0x745B076A, 0xC4726757, + 0xD4AE47E5, 0x648727D8, 0xB4FD879F, 0x04D4E7A2, + 0xD543B720, 0x656AD71D, 0xB510775A, 0x05391767, + 0x15E537D5, 0xA5CC57E8, 0x75B6F7AF, 0xC59F9792, + 0xDBE815E9, 0x6BC175D4, 0xBBBBD593, 0x0B92B5AE, + 0x1B4E951C, 0xAB67F521, 0x7B1D5566, 0xCB34355B, + 0x1AA365D9, 0xAA8A05E4, 0x7AF0A5A3, 0xCAD9C59E, + 0xDA05E52C, 0x6A2C8511, 0xBA562556, 0x0A7F456B, + 0x597FF589, 0xE95695B4, 0x392C35F3, 0x890555CE, + 0x99D9757C, 0x29F01541, 0xF98AB506, 0x49A3D53B, + 0x983485B9, 0x281DE584, 0xF86745C3, 0x484E25FE, + 0x5892054C, 0xE8BB6571, 0x38C1C536, 0x88E8A50B, + 0xDFC7D428, 0x6FEEB415, 0xBF941452, 0x0FBD746F, + 0x1F6154DD, 0xAF4834E0, 0x7F3294A7, 0xCF1BF49A, + 0x1E8CA418, 0xAEA5C425, 0x7EDF6462, 0xCEF6045F, + 0xDE2A24ED, 0x6E0344D0, 0xBE79E497, 0x0E5084AA, + 0x5D503448, 0xED795475, 0x3D03F432, 0x8D2A940F, + 0x9DF6B4BD, 0x2DDFD480, 0xFDA574C7, 0x4D8C14FA, + 0x9C1B4478, 0x2C322445, 0xFC488402, 0x4C61E43F, + 0x5CBDC48D, 0xEC94A4B0, 0x3CEE04F7, 0x8CC764CA + }, { + 0x00000000, 0xA5D35CCB, 0x0BA1C84D, 0xAE729486, + 0x1642919B, 0xB391CD50, 0x1DE359D6, 0xB830051D, + 0x6D8253EC, 0xC8510F27, 0x66239BA1, 0xC3F0C76A, + 0x7BC0C277, 0xDE139EBC, 0x70610A3A, 0xD5B256F1, + 0x9B02D603, 0x3ED18AC8, 0x90A31E4E, 0x35704285, + 0x8D404798, 0x28931B53, 0x86E18FD5, 0x2332D31E, + 0xF68085EF, 0x5353D924, 0xFD214DA2, 0x58F21169, + 0xE0C21474, 0x451148BF, 0xEB63DC39, 0x4EB080F2, + 0x3605AC07, 0x93D6F0CC, 0x3DA4644A, 0x98773881, + 0x20473D9C, 0x85946157, 0x2BE6F5D1, 0x8E35A91A, + 0x5B87FFEB, 0xFE54A320, 0x502637A6, 0xF5F56B6D, + 0x4DC56E70, 0xE81632BB, 0x4664A63D, 0xE3B7FAF6, + 0xAD077A04, 0x08D426CF, 0xA6A6B249, 0x0375EE82, + 0xBB45EB9F, 0x1E96B754, 0xB0E423D2, 0x15377F19, + 0xC08529E8, 0x65567523, 0xCB24E1A5, 0x6EF7BD6E, + 0xD6C7B873, 0x7314E4B8, 0xDD66703E, 0x78B52CF5, + 0x6C0A580F, 0xC9D904C4, 0x67AB9042, 0xC278CC89, + 0x7A48C994, 0xDF9B955F, 0x71E901D9, 0xD43A5D12, + 0x01880BE3, 0xA45B5728, 0x0A29C3AE, 0xAFFA9F65, + 0x17CA9A78, 0xB219C6B3, 0x1C6B5235, 0xB9B80EFE, + 0xF7088E0C, 0x52DBD2C7, 0xFCA94641, 0x597A1A8A, + 0xE14A1F97, 0x4499435C, 0xEAEBD7DA, 0x4F388B11, + 0x9A8ADDE0, 0x3F59812B, 0x912B15AD, 0x34F84966, + 0x8CC84C7B, 0x291B10B0, 0x87698436, 0x22BAD8FD, + 0x5A0FF408, 0xFFDCA8C3, 0x51AE3C45, 0xF47D608E, + 0x4C4D6593, 0xE99E3958, 0x47ECADDE, 0xE23FF115, + 0x378DA7E4, 0x925EFB2F, 0x3C2C6FA9, 0x99FF3362, + 0x21CF367F, 0x841C6AB4, 0x2A6EFE32, 0x8FBDA2F9, + 0xC10D220B, 0x64DE7EC0, 0xCAACEA46, 0x6F7FB68D, + 0xD74FB390, 0x729CEF5B, 0xDCEE7BDD, 0x793D2716, + 0xAC8F71E7, 0x095C2D2C, 0xA72EB9AA, 0x02FDE561, + 0xBACDE07C, 0x1F1EBCB7, 0xB16C2831, 0x14BF74FA, + 0xD814B01E, 0x7DC7ECD5, 0xD3B57853, 0x76662498, + 0xCE562185, 0x6B857D4E, 0xC5F7E9C8, 0x6024B503, + 0xB596E3F2, 0x1045BF39, 0xBE372BBF, 0x1BE47774, + 0xA3D47269, 0x06072EA2, 0xA875BA24, 0x0DA6E6EF, + 0x4316661D, 0xE6C53AD6, 0x48B7AE50, 0xED64F29B, + 0x5554F786, 0xF087AB4D, 0x5EF53FCB, 0xFB266300, + 0x2E9435F1, 0x8B47693A, 0x2535FDBC, 0x80E6A177, + 0x38D6A46A, 0x9D05F8A1, 0x33776C27, 0x96A430EC, + 0xEE111C19, 0x4BC240D2, 0xE5B0D454, 0x4063889F, + 0xF8538D82, 0x5D80D149, 0xF3F245CF, 0x56211904, + 0x83934FF5, 0x2640133E, 0x883287B8, 0x2DE1DB73, + 0x95D1DE6E, 0x300282A5, 0x9E701623, 0x3BA34AE8, + 0x7513CA1A, 0xD0C096D1, 0x7EB20257, 0xDB615E9C, + 0x63515B81, 0xC682074A, 0x68F093CC, 0xCD23CF07, + 0x189199F6, 0xBD42C53D, 0x133051BB, 0xB6E30D70, + 0x0ED3086D, 0xAB0054A6, 0x0572C020, 0xA0A19CEB, + 0xB41EE811, 0x11CDB4DA, 0xBFBF205C, 0x1A6C7C97, + 0xA25C798A, 0x078F2541, 0xA9FDB1C7, 0x0C2EED0C, + 0xD99CBBFD, 0x7C4FE736, 0xD23D73B0, 0x77EE2F7B, + 0xCFDE2A66, 0x6A0D76AD, 0xC47FE22B, 0x61ACBEE0, + 0x2F1C3E12, 0x8ACF62D9, 0x24BDF65F, 0x816EAA94, + 0x395EAF89, 0x9C8DF342, 0x32FF67C4, 0x972C3B0F, + 0x429E6DFE, 0xE74D3135, 0x493FA5B3, 0xECECF978, + 0x54DCFC65, 0xF10FA0AE, 0x5F7D3428, 0xFAAE68E3, + 0x821B4416, 0x27C818DD, 0x89BA8C5B, 0x2C69D090, + 0x9459D58D, 0x318A8946, 0x9FF81DC0, 0x3A2B410B, + 0xEF9917FA, 0x4A4A4B31, 0xE438DFB7, 0x41EB837C, + 0xF9DB8661, 0x5C08DAAA, 0xF27A4E2C, 0x57A912E7, + 0x19199215, 0xBCCACEDE, 0x12B85A58, 0xB76B0693, + 0x0F5B038E, 0xAA885F45, 0x04FACBC3, 0xA1299708, + 0x749BC1F9, 0xD1489D32, 0x7F3A09B4, 0xDAE9557F, + 0x62D95062, 0xC70A0CA9, 0x6978982F, 0xCCABC4E4 + }, { + 0x00000000, 0xB40B77A6, 0x29119F97, 0x9D1AE831, + 0x13244FF4, 0xA72F3852, 0x3A35D063, 0x8E3EA7C5, + 0x674EEF33, 0xD3459895, 0x4E5F70A4, 0xFA540702, + 0x746AA0C7, 0xC061D761, 0x5D7B3F50, 0xE97048F6, + 0xCE9CDE67, 0x7A97A9C1, 0xE78D41F0, 0x53863656, + 0xDDB89193, 0x69B3E635, 0xF4A90E04, 0x40A279A2, + 0xA9D23154, 0x1DD946F2, 0x80C3AEC3, 0x34C8D965, + 0xBAF67EA0, 0x0EFD0906, 0x93E7E137, 0x27EC9691, + 0x9C39BDCF, 0x2832CA69, 0xB5282258, 0x012355FE, + 0x8F1DF23B, 0x3B16859D, 0xA60C6DAC, 0x12071A0A, + 0xFB7752FC, 0x4F7C255A, 0xD266CD6B, 0x666DBACD, + 0xE8531D08, 0x5C586AAE, 0xC142829F, 0x7549F539, + 0x52A563A8, 0xE6AE140E, 0x7BB4FC3F, 0xCFBF8B99, + 0x41812C5C, 0xF58A5BFA, 0x6890B3CB, 0xDC9BC46D, + 0x35EB8C9B, 0x81E0FB3D, 0x1CFA130C, 0xA8F164AA, + 0x26CFC36F, 0x92C4B4C9, 0x0FDE5CF8, 0xBBD52B5E, + 0x79750B44, 0xCD7E7CE2, 0x506494D3, 0xE46FE375, + 0x6A5144B0, 0xDE5A3316, 0x4340DB27, 0xF74BAC81, + 0x1E3BE477, 0xAA3093D1, 0x372A7BE0, 0x83210C46, + 0x0D1FAB83, 0xB914DC25, 0x240E3414, 0x900543B2, + 0xB7E9D523, 0x03E2A285, 0x9EF84AB4, 0x2AF33D12, + 0xA4CD9AD7, 0x10C6ED71, 0x8DDC0540, 0x39D772E6, + 0xD0A73A10, 0x64AC4DB6, 0xF9B6A587, 0x4DBDD221, + 0xC38375E4, 0x77880242, 0xEA92EA73, 0x5E999DD5, + 0xE54CB68B, 0x5147C12D, 0xCC5D291C, 0x78565EBA, + 0xF668F97F, 0x42638ED9, 0xDF7966E8, 0x6B72114E, + 0x820259B8, 0x36092E1E, 0xAB13C62F, 0x1F18B189, + 0x9126164C, 0x252D61EA, 0xB83789DB, 0x0C3CFE7D, + 0x2BD068EC, 0x9FDB1F4A, 0x02C1F77B, 0xB6CA80DD, + 0x38F42718, 0x8CFF50BE, 0x11E5B88F, 0xA5EECF29, + 0x4C9E87DF, 0xF895F079, 0x658F1848, 0xD1846FEE, + 0x5FBAC82B, 0xEBB1BF8D, 0x76AB57BC, 0xC2A0201A, + 0xF2EA1688, 0x46E1612E, 0xDBFB891F, 0x6FF0FEB9, + 0xE1CE597C, 0x55C52EDA, 0xC8DFC6EB, 0x7CD4B14D, + 0x95A4F9BB, 0x21AF8E1D, 0xBCB5662C, 0x08BE118A, + 0x8680B64F, 0x328BC1E9, 0xAF9129D8, 0x1B9A5E7E, + 0x3C76C8EF, 0x887DBF49, 0x15675778, 0xA16C20DE, + 0x2F52871B, 0x9B59F0BD, 0x0643188C, 0xB2486F2A, + 0x5B3827DC, 0xEF33507A, 0x7229B84B, 0xC622CFED, + 0x481C6828, 0xFC171F8E, 0x610DF7BF, 0xD5068019, + 0x6ED3AB47, 0xDAD8DCE1, 0x47C234D0, 0xF3C94376, + 0x7DF7E4B3, 0xC9FC9315, 0x54E67B24, 0xE0ED0C82, + 0x099D4474, 0xBD9633D2, 0x208CDBE3, 0x9487AC45, + 0x1AB90B80, 0xAEB27C26, 0x33A89417, 0x87A3E3B1, + 0xA04F7520, 0x14440286, 0x895EEAB7, 0x3D559D11, + 0xB36B3AD4, 0x07604D72, 0x9A7AA543, 0x2E71D2E5, + 0xC7019A13, 0x730AEDB5, 0xEE100584, 0x5A1B7222, + 0xD425D5E7, 0x602EA241, 0xFD344A70, 0x493F3DD6, + 0x8B9F1DCC, 0x3F946A6A, 0xA28E825B, 0x1685F5FD, + 0x98BB5238, 0x2CB0259E, 0xB1AACDAF, 0x05A1BA09, + 0xECD1F2FF, 0x58DA8559, 0xC5C06D68, 0x71CB1ACE, + 0xFFF5BD0B, 0x4BFECAAD, 0xD6E4229C, 0x62EF553A, + 0x4503C3AB, 0xF108B40D, 0x6C125C3C, 0xD8192B9A, + 0x56278C5F, 0xE22CFBF9, 0x7F3613C8, 0xCB3D646E, + 0x224D2C98, 0x96465B3E, 0x0B5CB30F, 0xBF57C4A9, + 0x3169636C, 0x856214CA, 0x1878FCFB, 0xAC738B5D, + 0x17A6A003, 0xA3ADD7A5, 0x3EB73F94, 0x8ABC4832, + 0x0482EFF7, 0xB0899851, 0x2D937060, 0x999807C6, + 0x70E84F30, 0xC4E33896, 0x59F9D0A7, 0xEDF2A701, + 0x63CC00C4, 0xD7C77762, 0x4ADD9F53, 0xFED6E8F5, + 0xD93A7E64, 0x6D3109C2, 0xF02BE1F3, 0x44209655, + 0xCA1E3190, 0x7E154636, 0xE30FAE07, 0x5704D9A1, + 0xBE749157, 0x0A7FE6F1, 0x97650EC0, 0x236E7966, + 0xAD50DEA3, 0x195BA905, 0x84414134, 0x304A3692 + }, { + 0x00000000, 0x9E00AACC, 0x7D072542, 0xE3078F8E, + 0xFA0E4A84, 0x640EE048, 0x87096FC6, 0x1909C50A, + 0xB51BE5D3, 0x2B1B4F1F, 0xC81CC091, 0x561C6A5D, + 0x4F15AF57, 0xD115059B, 0x32128A15, 0xAC1220D9, + 0x2B31BB7C, 0xB53111B0, 0x56369E3E, 0xC83634F2, + 0xD13FF1F8, 0x4F3F5B34, 0xAC38D4BA, 0x32387E76, + 0x9E2A5EAF, 0x002AF463, 0xE32D7BED, 0x7D2DD121, + 0x6424142B, 0xFA24BEE7, 0x19233169, 0x87239BA5, + 0x566276F9, 0xC862DC35, 0x2B6553BB, 0xB565F977, + 0xAC6C3C7D, 0x326C96B1, 0xD16B193F, 0x4F6BB3F3, + 0xE379932A, 0x7D7939E6, 0x9E7EB668, 0x007E1CA4, + 0x1977D9AE, 0x87777362, 0x6470FCEC, 0xFA705620, + 0x7D53CD85, 0xE3536749, 0x0054E8C7, 0x9E54420B, + 0x875D8701, 0x195D2DCD, 0xFA5AA243, 0x645A088F, + 0xC8482856, 0x5648829A, 0xB54F0D14, 0x2B4FA7D8, + 0x324662D2, 0xAC46C81E, 0x4F414790, 0xD141ED5C, + 0xEDC29D29, 0x73C237E5, 0x90C5B86B, 0x0EC512A7, + 0x17CCD7AD, 0x89CC7D61, 0x6ACBF2EF, 0xF4CB5823, + 0x58D978FA, 0xC6D9D236, 0x25DE5DB8, 0xBBDEF774, + 0xA2D7327E, 0x3CD798B2, 0xDFD0173C, 0x41D0BDF0, + 0xC6F32655, 0x58F38C99, 0xBBF40317, 0x25F4A9DB, + 0x3CFD6CD1, 0xA2FDC61D, 0x41FA4993, 0xDFFAE35F, + 0x73E8C386, 0xEDE8694A, 0x0EEFE6C4, 0x90EF4C08, + 0x89E68902, 0x17E623CE, 0xF4E1AC40, 0x6AE1068C, + 0xBBA0EBD0, 0x25A0411C, 0xC6A7CE92, 0x58A7645E, + 0x41AEA154, 0xDFAE0B98, 0x3CA98416, 0xA2A92EDA, + 0x0EBB0E03, 0x90BBA4CF, 0x73BC2B41, 0xEDBC818D, + 0xF4B54487, 0x6AB5EE4B, 0x89B261C5, 0x17B2CB09, + 0x909150AC, 0x0E91FA60, 0xED9675EE, 0x7396DF22, + 0x6A9F1A28, 0xF49FB0E4, 0x17983F6A, 0x899895A6, + 0x258AB57F, 0xBB8A1FB3, 0x588D903D, 0xC68D3AF1, + 0xDF84FFFB, 0x41845537, 0xA283DAB9, 0x3C837075, + 0xDA853B53, 0x4485919F, 0xA7821E11, 0x3982B4DD, + 0x208B71D7, 0xBE8BDB1B, 0x5D8C5495, 0xC38CFE59, + 0x6F9EDE80, 0xF19E744C, 0x1299FBC2, 0x8C99510E, + 0x95909404, 0x0B903EC8, 0xE897B146, 0x76971B8A, + 0xF1B4802F, 0x6FB42AE3, 0x8CB3A56D, 0x12B30FA1, + 0x0BBACAAB, 0x95BA6067, 0x76BDEFE9, 0xE8BD4525, + 0x44AF65FC, 0xDAAFCF30, 0x39A840BE, 0xA7A8EA72, + 0xBEA12F78, 0x20A185B4, 0xC3A60A3A, 0x5DA6A0F6, + 0x8CE74DAA, 0x12E7E766, 0xF1E068E8, 0x6FE0C224, + 0x76E9072E, 0xE8E9ADE2, 0x0BEE226C, 0x95EE88A0, + 0x39FCA879, 0xA7FC02B5, 0x44FB8D3B, 0xDAFB27F7, + 0xC3F2E2FD, 0x5DF24831, 0xBEF5C7BF, 0x20F56D73, + 0xA7D6F6D6, 0x39D65C1A, 0xDAD1D394, 0x44D17958, + 0x5DD8BC52, 0xC3D8169E, 0x20DF9910, 0xBEDF33DC, + 0x12CD1305, 0x8CCDB9C9, 0x6FCA3647, 0xF1CA9C8B, + 0xE8C35981, 0x76C3F34D, 0x95C47CC3, 0x0BC4D60F, + 0x3747A67A, 0xA9470CB6, 0x4A408338, 0xD44029F4, + 0xCD49ECFE, 0x53494632, 0xB04EC9BC, 0x2E4E6370, + 0x825C43A9, 0x1C5CE965, 0xFF5B66EB, 0x615BCC27, + 0x7852092D, 0xE652A3E1, 0x05552C6F, 0x9B5586A3, + 0x1C761D06, 0x8276B7CA, 0x61713844, 0xFF719288, + 0xE6785782, 0x7878FD4E, 0x9B7F72C0, 0x057FD80C, + 0xA96DF8D5, 0x376D5219, 0xD46ADD97, 0x4A6A775B, + 0x5363B251, 0xCD63189D, 0x2E649713, 0xB0643DDF, + 0x6125D083, 0xFF257A4F, 0x1C22F5C1, 0x82225F0D, + 0x9B2B9A07, 0x052B30CB, 0xE62CBF45, 0x782C1589, + 0xD43E3550, 0x4A3E9F9C, 0xA9391012, 0x3739BADE, + 0x2E307FD4, 0xB030D518, 0x53375A96, 0xCD37F05A, + 0x4A146BFF, 0xD414C133, 0x37134EBD, 0xA913E471, + 0xB01A217B, 0x2E1A8BB7, 0xCD1D0439, 0x531DAEF5, + 0xFF0F8E2C, 0x610F24E0, 0x8208AB6E, 0x1C0801A2, + 0x0501C4A8, 0x9B016E64, 0x7806E1EA, 0xE6064B26 + } +}; diff --git a/src/liblzma/check/crc32_table_le.h b/src/liblzma/check/crc32_table_le.h new file mode 100644 index 0000000000..e89c21a7b2 --- /dev/null +++ b/src/liblzma/check/crc32_table_le.h @@ -0,0 +1,527 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by crc32_tablegen.c. + +const uint32_t lzma_crc32_table[8][256] = { + { + 0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, + 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3, + 0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, + 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91, + 0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, + 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7, + 0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, + 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5, + 0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, + 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B, + 0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, + 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59, + 0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, + 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F, + 0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, + 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D, + 0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, + 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433, + 0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, + 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01, + 0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, + 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457, + 0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, + 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65, + 0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, + 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB, + 0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, + 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9, + 0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, + 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F, + 0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, + 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD, + 0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, + 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683, + 0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, + 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1, + 0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, + 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7, + 0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, + 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5, + 0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, + 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B, + 0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, + 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79, + 0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, + 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F, + 0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, + 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D, + 0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, + 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713, + 0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, + 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21, + 0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, + 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777, + 0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, + 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45, + 0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, + 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB, + 0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, + 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9, + 0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, + 0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF, + 0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, + 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D + }, { + 0x00000000, 0x191B3141, 0x32366282, 0x2B2D53C3, + 0x646CC504, 0x7D77F445, 0x565AA786, 0x4F4196C7, + 0xC8D98A08, 0xD1C2BB49, 0xFAEFE88A, 0xE3F4D9CB, + 0xACB54F0C, 0xB5AE7E4D, 0x9E832D8E, 0x87981CCF, + 0x4AC21251, 0x53D92310, 0x78F470D3, 0x61EF4192, + 0x2EAED755, 0x37B5E614, 0x1C98B5D7, 0x05838496, + 0x821B9859, 0x9B00A918, 0xB02DFADB, 0xA936CB9A, + 0xE6775D5D, 0xFF6C6C1C, 0xD4413FDF, 0xCD5A0E9E, + 0x958424A2, 0x8C9F15E3, 0xA7B24620, 0xBEA97761, + 0xF1E8E1A6, 0xE8F3D0E7, 0xC3DE8324, 0xDAC5B265, + 0x5D5DAEAA, 0x44469FEB, 0x6F6BCC28, 0x7670FD69, + 0x39316BAE, 0x202A5AEF, 0x0B07092C, 0x121C386D, + 0xDF4636F3, 0xC65D07B2, 0xED705471, 0xF46B6530, + 0xBB2AF3F7, 0xA231C2B6, 0x891C9175, 0x9007A034, + 0x179FBCFB, 0x0E848DBA, 0x25A9DE79, 0x3CB2EF38, + 0x73F379FF, 0x6AE848BE, 0x41C51B7D, 0x58DE2A3C, + 0xF0794F05, 0xE9627E44, 0xC24F2D87, 0xDB541CC6, + 0x94158A01, 0x8D0EBB40, 0xA623E883, 0xBF38D9C2, + 0x38A0C50D, 0x21BBF44C, 0x0A96A78F, 0x138D96CE, + 0x5CCC0009, 0x45D73148, 0x6EFA628B, 0x77E153CA, + 0xBABB5D54, 0xA3A06C15, 0x888D3FD6, 0x91960E97, + 0xDED79850, 0xC7CCA911, 0xECE1FAD2, 0xF5FACB93, + 0x7262D75C, 0x6B79E61D, 0x4054B5DE, 0x594F849F, + 0x160E1258, 0x0F152319, 0x243870DA, 0x3D23419B, + 0x65FD6BA7, 0x7CE65AE6, 0x57CB0925, 0x4ED03864, + 0x0191AEA3, 0x188A9FE2, 0x33A7CC21, 0x2ABCFD60, + 0xAD24E1AF, 0xB43FD0EE, 0x9F12832D, 0x8609B26C, + 0xC94824AB, 0xD05315EA, 0xFB7E4629, 0xE2657768, + 0x2F3F79F6, 0x362448B7, 0x1D091B74, 0x04122A35, + 0x4B53BCF2, 0x52488DB3, 0x7965DE70, 0x607EEF31, + 0xE7E6F3FE, 0xFEFDC2BF, 0xD5D0917C, 0xCCCBA03D, + 0x838A36FA, 0x9A9107BB, 0xB1BC5478, 0xA8A76539, + 0x3B83984B, 0x2298A90A, 0x09B5FAC9, 0x10AECB88, + 0x5FEF5D4F, 0x46F46C0E, 0x6DD93FCD, 0x74C20E8C, + 0xF35A1243, 0xEA412302, 0xC16C70C1, 0xD8774180, + 0x9736D747, 0x8E2DE606, 0xA500B5C5, 0xBC1B8484, + 0x71418A1A, 0x685ABB5B, 0x4377E898, 0x5A6CD9D9, + 0x152D4F1E, 0x0C367E5F, 0x271B2D9C, 0x3E001CDD, + 0xB9980012, 0xA0833153, 0x8BAE6290, 0x92B553D1, + 0xDDF4C516, 0xC4EFF457, 0xEFC2A794, 0xF6D996D5, + 0xAE07BCE9, 0xB71C8DA8, 0x9C31DE6B, 0x852AEF2A, + 0xCA6B79ED, 0xD37048AC, 0xF85D1B6F, 0xE1462A2E, + 0x66DE36E1, 0x7FC507A0, 0x54E85463, 0x4DF36522, + 0x02B2F3E5, 0x1BA9C2A4, 0x30849167, 0x299FA026, + 0xE4C5AEB8, 0xFDDE9FF9, 0xD6F3CC3A, 0xCFE8FD7B, + 0x80A96BBC, 0x99B25AFD, 0xB29F093E, 0xAB84387F, + 0x2C1C24B0, 0x350715F1, 0x1E2A4632, 0x07317773, + 0x4870E1B4, 0x516BD0F5, 0x7A468336, 0x635DB277, + 0xCBFAD74E, 0xD2E1E60F, 0xF9CCB5CC, 0xE0D7848D, + 0xAF96124A, 0xB68D230B, 0x9DA070C8, 0x84BB4189, + 0x03235D46, 0x1A386C07, 0x31153FC4, 0x280E0E85, + 0x674F9842, 0x7E54A903, 0x5579FAC0, 0x4C62CB81, + 0x8138C51F, 0x9823F45E, 0xB30EA79D, 0xAA1596DC, + 0xE554001B, 0xFC4F315A, 0xD7626299, 0xCE7953D8, + 0x49E14F17, 0x50FA7E56, 0x7BD72D95, 0x62CC1CD4, + 0x2D8D8A13, 0x3496BB52, 0x1FBBE891, 0x06A0D9D0, + 0x5E7EF3EC, 0x4765C2AD, 0x6C48916E, 0x7553A02F, + 0x3A1236E8, 0x230907A9, 0x0824546A, 0x113F652B, + 0x96A779E4, 0x8FBC48A5, 0xA4911B66, 0xBD8A2A27, + 0xF2CBBCE0, 0xEBD08DA1, 0xC0FDDE62, 0xD9E6EF23, + 0x14BCE1BD, 0x0DA7D0FC, 0x268A833F, 0x3F91B27E, + 0x70D024B9, 0x69CB15F8, 0x42E6463B, 0x5BFD777A, + 0xDC656BB5, 0xC57E5AF4, 0xEE530937, 0xF7483876, + 0xB809AEB1, 0xA1129FF0, 0x8A3FCC33, 0x9324FD72 + }, { + 0x00000000, 0x01C26A37, 0x0384D46E, 0x0246BE59, + 0x0709A8DC, 0x06CBC2EB, 0x048D7CB2, 0x054F1685, + 0x0E1351B8, 0x0FD13B8F, 0x0D9785D6, 0x0C55EFE1, + 0x091AF964, 0x08D89353, 0x0A9E2D0A, 0x0B5C473D, + 0x1C26A370, 0x1DE4C947, 0x1FA2771E, 0x1E601D29, + 0x1B2F0BAC, 0x1AED619B, 0x18ABDFC2, 0x1969B5F5, + 0x1235F2C8, 0x13F798FF, 0x11B126A6, 0x10734C91, + 0x153C5A14, 0x14FE3023, 0x16B88E7A, 0x177AE44D, + 0x384D46E0, 0x398F2CD7, 0x3BC9928E, 0x3A0BF8B9, + 0x3F44EE3C, 0x3E86840B, 0x3CC03A52, 0x3D025065, + 0x365E1758, 0x379C7D6F, 0x35DAC336, 0x3418A901, + 0x3157BF84, 0x3095D5B3, 0x32D36BEA, 0x331101DD, + 0x246BE590, 0x25A98FA7, 0x27EF31FE, 0x262D5BC9, + 0x23624D4C, 0x22A0277B, 0x20E69922, 0x2124F315, + 0x2A78B428, 0x2BBADE1F, 0x29FC6046, 0x283E0A71, + 0x2D711CF4, 0x2CB376C3, 0x2EF5C89A, 0x2F37A2AD, + 0x709A8DC0, 0x7158E7F7, 0x731E59AE, 0x72DC3399, + 0x7793251C, 0x76514F2B, 0x7417F172, 0x75D59B45, + 0x7E89DC78, 0x7F4BB64F, 0x7D0D0816, 0x7CCF6221, + 0x798074A4, 0x78421E93, 0x7A04A0CA, 0x7BC6CAFD, + 0x6CBC2EB0, 0x6D7E4487, 0x6F38FADE, 0x6EFA90E9, + 0x6BB5866C, 0x6A77EC5B, 0x68315202, 0x69F33835, + 0x62AF7F08, 0x636D153F, 0x612BAB66, 0x60E9C151, + 0x65A6D7D4, 0x6464BDE3, 0x662203BA, 0x67E0698D, + 0x48D7CB20, 0x4915A117, 0x4B531F4E, 0x4A917579, + 0x4FDE63FC, 0x4E1C09CB, 0x4C5AB792, 0x4D98DDA5, + 0x46C49A98, 0x4706F0AF, 0x45404EF6, 0x448224C1, + 0x41CD3244, 0x400F5873, 0x4249E62A, 0x438B8C1D, + 0x54F16850, 0x55330267, 0x5775BC3E, 0x56B7D609, + 0x53F8C08C, 0x523AAABB, 0x507C14E2, 0x51BE7ED5, + 0x5AE239E8, 0x5B2053DF, 0x5966ED86, 0x58A487B1, + 0x5DEB9134, 0x5C29FB03, 0x5E6F455A, 0x5FAD2F6D, + 0xE1351B80, 0xE0F771B7, 0xE2B1CFEE, 0xE373A5D9, + 0xE63CB35C, 0xE7FED96B, 0xE5B86732, 0xE47A0D05, + 0xEF264A38, 0xEEE4200F, 0xECA29E56, 0xED60F461, + 0xE82FE2E4, 0xE9ED88D3, 0xEBAB368A, 0xEA695CBD, + 0xFD13B8F0, 0xFCD1D2C7, 0xFE976C9E, 0xFF5506A9, + 0xFA1A102C, 0xFBD87A1B, 0xF99EC442, 0xF85CAE75, + 0xF300E948, 0xF2C2837F, 0xF0843D26, 0xF1465711, + 0xF4094194, 0xF5CB2BA3, 0xF78D95FA, 0xF64FFFCD, + 0xD9785D60, 0xD8BA3757, 0xDAFC890E, 0xDB3EE339, + 0xDE71F5BC, 0xDFB39F8B, 0xDDF521D2, 0xDC374BE5, + 0xD76B0CD8, 0xD6A966EF, 0xD4EFD8B6, 0xD52DB281, + 0xD062A404, 0xD1A0CE33, 0xD3E6706A, 0xD2241A5D, + 0xC55EFE10, 0xC49C9427, 0xC6DA2A7E, 0xC7184049, + 0xC25756CC, 0xC3953CFB, 0xC1D382A2, 0xC011E895, + 0xCB4DAFA8, 0xCA8FC59F, 0xC8C97BC6, 0xC90B11F1, + 0xCC440774, 0xCD866D43, 0xCFC0D31A, 0xCE02B92D, + 0x91AF9640, 0x906DFC77, 0x922B422E, 0x93E92819, + 0x96A63E9C, 0x976454AB, 0x9522EAF2, 0x94E080C5, + 0x9FBCC7F8, 0x9E7EADCF, 0x9C381396, 0x9DFA79A1, + 0x98B56F24, 0x99770513, 0x9B31BB4A, 0x9AF3D17D, + 0x8D893530, 0x8C4B5F07, 0x8E0DE15E, 0x8FCF8B69, + 0x8A809DEC, 0x8B42F7DB, 0x89044982, 0x88C623B5, + 0x839A6488, 0x82580EBF, 0x801EB0E6, 0x81DCDAD1, + 0x8493CC54, 0x8551A663, 0x8717183A, 0x86D5720D, + 0xA9E2D0A0, 0xA820BA97, 0xAA6604CE, 0xABA46EF9, + 0xAEEB787C, 0xAF29124B, 0xAD6FAC12, 0xACADC625, + 0xA7F18118, 0xA633EB2F, 0xA4755576, 0xA5B73F41, + 0xA0F829C4, 0xA13A43F3, 0xA37CFDAA, 0xA2BE979D, + 0xB5C473D0, 0xB40619E7, 0xB640A7BE, 0xB782CD89, + 0xB2CDDB0C, 0xB30FB13B, 0xB1490F62, 0xB08B6555, + 0xBBD72268, 0xBA15485F, 0xB853F606, 0xB9919C31, + 0xBCDE8AB4, 0xBD1CE083, 0xBF5A5EDA, 0xBE9834ED + }, { + 0x00000000, 0xB8BC6765, 0xAA09C88B, 0x12B5AFEE, + 0x8F629757, 0x37DEF032, 0x256B5FDC, 0x9DD738B9, + 0xC5B428EF, 0x7D084F8A, 0x6FBDE064, 0xD7018701, + 0x4AD6BFB8, 0xF26AD8DD, 0xE0DF7733, 0x58631056, + 0x5019579F, 0xE8A530FA, 0xFA109F14, 0x42ACF871, + 0xDF7BC0C8, 0x67C7A7AD, 0x75720843, 0xCDCE6F26, + 0x95AD7F70, 0x2D111815, 0x3FA4B7FB, 0x8718D09E, + 0x1ACFE827, 0xA2738F42, 0xB0C620AC, 0x087A47C9, + 0xA032AF3E, 0x188EC85B, 0x0A3B67B5, 0xB28700D0, + 0x2F503869, 0x97EC5F0C, 0x8559F0E2, 0x3DE59787, + 0x658687D1, 0xDD3AE0B4, 0xCF8F4F5A, 0x7733283F, + 0xEAE41086, 0x525877E3, 0x40EDD80D, 0xF851BF68, + 0xF02BF8A1, 0x48979FC4, 0x5A22302A, 0xE29E574F, + 0x7F496FF6, 0xC7F50893, 0xD540A77D, 0x6DFCC018, + 0x359FD04E, 0x8D23B72B, 0x9F9618C5, 0x272A7FA0, + 0xBAFD4719, 0x0241207C, 0x10F48F92, 0xA848E8F7, + 0x9B14583D, 0x23A83F58, 0x311D90B6, 0x89A1F7D3, + 0x1476CF6A, 0xACCAA80F, 0xBE7F07E1, 0x06C36084, + 0x5EA070D2, 0xE61C17B7, 0xF4A9B859, 0x4C15DF3C, + 0xD1C2E785, 0x697E80E0, 0x7BCB2F0E, 0xC377486B, + 0xCB0D0FA2, 0x73B168C7, 0x6104C729, 0xD9B8A04C, + 0x446F98F5, 0xFCD3FF90, 0xEE66507E, 0x56DA371B, + 0x0EB9274D, 0xB6054028, 0xA4B0EFC6, 0x1C0C88A3, + 0x81DBB01A, 0x3967D77F, 0x2BD27891, 0x936E1FF4, + 0x3B26F703, 0x839A9066, 0x912F3F88, 0x299358ED, + 0xB4446054, 0x0CF80731, 0x1E4DA8DF, 0xA6F1CFBA, + 0xFE92DFEC, 0x462EB889, 0x549B1767, 0xEC277002, + 0x71F048BB, 0xC94C2FDE, 0xDBF98030, 0x6345E755, + 0x6B3FA09C, 0xD383C7F9, 0xC1366817, 0x798A0F72, + 0xE45D37CB, 0x5CE150AE, 0x4E54FF40, 0xF6E89825, + 0xAE8B8873, 0x1637EF16, 0x048240F8, 0xBC3E279D, + 0x21E91F24, 0x99557841, 0x8BE0D7AF, 0x335CB0CA, + 0xED59B63B, 0x55E5D15E, 0x47507EB0, 0xFFEC19D5, + 0x623B216C, 0xDA874609, 0xC832E9E7, 0x708E8E82, + 0x28ED9ED4, 0x9051F9B1, 0x82E4565F, 0x3A58313A, + 0xA78F0983, 0x1F336EE6, 0x0D86C108, 0xB53AA66D, + 0xBD40E1A4, 0x05FC86C1, 0x1749292F, 0xAFF54E4A, + 0x322276F3, 0x8A9E1196, 0x982BBE78, 0x2097D91D, + 0x78F4C94B, 0xC048AE2E, 0xD2FD01C0, 0x6A4166A5, + 0xF7965E1C, 0x4F2A3979, 0x5D9F9697, 0xE523F1F2, + 0x4D6B1905, 0xF5D77E60, 0xE762D18E, 0x5FDEB6EB, + 0xC2098E52, 0x7AB5E937, 0x680046D9, 0xD0BC21BC, + 0x88DF31EA, 0x3063568F, 0x22D6F961, 0x9A6A9E04, + 0x07BDA6BD, 0xBF01C1D8, 0xADB46E36, 0x15080953, + 0x1D724E9A, 0xA5CE29FF, 0xB77B8611, 0x0FC7E174, + 0x9210D9CD, 0x2AACBEA8, 0x38191146, 0x80A57623, + 0xD8C66675, 0x607A0110, 0x72CFAEFE, 0xCA73C99B, + 0x57A4F122, 0xEF189647, 0xFDAD39A9, 0x45115ECC, + 0x764DEE06, 0xCEF18963, 0xDC44268D, 0x64F841E8, + 0xF92F7951, 0x41931E34, 0x5326B1DA, 0xEB9AD6BF, + 0xB3F9C6E9, 0x0B45A18C, 0x19F00E62, 0xA14C6907, + 0x3C9B51BE, 0x842736DB, 0x96929935, 0x2E2EFE50, + 0x2654B999, 0x9EE8DEFC, 0x8C5D7112, 0x34E11677, + 0xA9362ECE, 0x118A49AB, 0x033FE645, 0xBB838120, + 0xE3E09176, 0x5B5CF613, 0x49E959FD, 0xF1553E98, + 0x6C820621, 0xD43E6144, 0xC68BCEAA, 0x7E37A9CF, + 0xD67F4138, 0x6EC3265D, 0x7C7689B3, 0xC4CAEED6, + 0x591DD66F, 0xE1A1B10A, 0xF3141EE4, 0x4BA87981, + 0x13CB69D7, 0xAB770EB2, 0xB9C2A15C, 0x017EC639, + 0x9CA9FE80, 0x241599E5, 0x36A0360B, 0x8E1C516E, + 0x866616A7, 0x3EDA71C2, 0x2C6FDE2C, 0x94D3B949, + 0x090481F0, 0xB1B8E695, 0xA30D497B, 0x1BB12E1E, + 0x43D23E48, 0xFB6E592D, 0xE9DBF6C3, 0x516791A6, + 0xCCB0A91F, 0x740CCE7A, 0x66B96194, 0xDE0506F1 + }, { + 0x00000000, 0x3D6029B0, 0x7AC05360, 0x47A07AD0, + 0xF580A6C0, 0xC8E08F70, 0x8F40F5A0, 0xB220DC10, + 0x30704BC1, 0x0D106271, 0x4AB018A1, 0x77D03111, + 0xC5F0ED01, 0xF890C4B1, 0xBF30BE61, 0x825097D1, + 0x60E09782, 0x5D80BE32, 0x1A20C4E2, 0x2740ED52, + 0x95603142, 0xA80018F2, 0xEFA06222, 0xD2C04B92, + 0x5090DC43, 0x6DF0F5F3, 0x2A508F23, 0x1730A693, + 0xA5107A83, 0x98705333, 0xDFD029E3, 0xE2B00053, + 0xC1C12F04, 0xFCA106B4, 0xBB017C64, 0x866155D4, + 0x344189C4, 0x0921A074, 0x4E81DAA4, 0x73E1F314, + 0xF1B164C5, 0xCCD14D75, 0x8B7137A5, 0xB6111E15, + 0x0431C205, 0x3951EBB5, 0x7EF19165, 0x4391B8D5, + 0xA121B886, 0x9C419136, 0xDBE1EBE6, 0xE681C256, + 0x54A11E46, 0x69C137F6, 0x2E614D26, 0x13016496, + 0x9151F347, 0xAC31DAF7, 0xEB91A027, 0xD6F18997, + 0x64D15587, 0x59B17C37, 0x1E1106E7, 0x23712F57, + 0x58F35849, 0x659371F9, 0x22330B29, 0x1F532299, + 0xAD73FE89, 0x9013D739, 0xD7B3ADE9, 0xEAD38459, + 0x68831388, 0x55E33A38, 0x124340E8, 0x2F236958, + 0x9D03B548, 0xA0639CF8, 0xE7C3E628, 0xDAA3CF98, + 0x3813CFCB, 0x0573E67B, 0x42D39CAB, 0x7FB3B51B, + 0xCD93690B, 0xF0F340BB, 0xB7533A6B, 0x8A3313DB, + 0x0863840A, 0x3503ADBA, 0x72A3D76A, 0x4FC3FEDA, + 0xFDE322CA, 0xC0830B7A, 0x872371AA, 0xBA43581A, + 0x9932774D, 0xA4525EFD, 0xE3F2242D, 0xDE920D9D, + 0x6CB2D18D, 0x51D2F83D, 0x167282ED, 0x2B12AB5D, + 0xA9423C8C, 0x9422153C, 0xD3826FEC, 0xEEE2465C, + 0x5CC29A4C, 0x61A2B3FC, 0x2602C92C, 0x1B62E09C, + 0xF9D2E0CF, 0xC4B2C97F, 0x8312B3AF, 0xBE729A1F, + 0x0C52460F, 0x31326FBF, 0x7692156F, 0x4BF23CDF, + 0xC9A2AB0E, 0xF4C282BE, 0xB362F86E, 0x8E02D1DE, + 0x3C220DCE, 0x0142247E, 0x46E25EAE, 0x7B82771E, + 0xB1E6B092, 0x8C869922, 0xCB26E3F2, 0xF646CA42, + 0x44661652, 0x79063FE2, 0x3EA64532, 0x03C66C82, + 0x8196FB53, 0xBCF6D2E3, 0xFB56A833, 0xC6368183, + 0x74165D93, 0x49767423, 0x0ED60EF3, 0x33B62743, + 0xD1062710, 0xEC660EA0, 0xABC67470, 0x96A65DC0, + 0x248681D0, 0x19E6A860, 0x5E46D2B0, 0x6326FB00, + 0xE1766CD1, 0xDC164561, 0x9BB63FB1, 0xA6D61601, + 0x14F6CA11, 0x2996E3A1, 0x6E369971, 0x5356B0C1, + 0x70279F96, 0x4D47B626, 0x0AE7CCF6, 0x3787E546, + 0x85A73956, 0xB8C710E6, 0xFF676A36, 0xC2074386, + 0x4057D457, 0x7D37FDE7, 0x3A978737, 0x07F7AE87, + 0xB5D77297, 0x88B75B27, 0xCF1721F7, 0xF2770847, + 0x10C70814, 0x2DA721A4, 0x6A075B74, 0x576772C4, + 0xE547AED4, 0xD8278764, 0x9F87FDB4, 0xA2E7D404, + 0x20B743D5, 0x1DD76A65, 0x5A7710B5, 0x67173905, + 0xD537E515, 0xE857CCA5, 0xAFF7B675, 0x92979FC5, + 0xE915E8DB, 0xD475C16B, 0x93D5BBBB, 0xAEB5920B, + 0x1C954E1B, 0x21F567AB, 0x66551D7B, 0x5B3534CB, + 0xD965A31A, 0xE4058AAA, 0xA3A5F07A, 0x9EC5D9CA, + 0x2CE505DA, 0x11852C6A, 0x562556BA, 0x6B457F0A, + 0x89F57F59, 0xB49556E9, 0xF3352C39, 0xCE550589, + 0x7C75D999, 0x4115F029, 0x06B58AF9, 0x3BD5A349, + 0xB9853498, 0x84E51D28, 0xC34567F8, 0xFE254E48, + 0x4C059258, 0x7165BBE8, 0x36C5C138, 0x0BA5E888, + 0x28D4C7DF, 0x15B4EE6F, 0x521494BF, 0x6F74BD0F, + 0xDD54611F, 0xE03448AF, 0xA794327F, 0x9AF41BCF, + 0x18A48C1E, 0x25C4A5AE, 0x6264DF7E, 0x5F04F6CE, + 0xED242ADE, 0xD044036E, 0x97E479BE, 0xAA84500E, + 0x4834505D, 0x755479ED, 0x32F4033D, 0x0F942A8D, + 0xBDB4F69D, 0x80D4DF2D, 0xC774A5FD, 0xFA148C4D, + 0x78441B9C, 0x4524322C, 0x028448FC, 0x3FE4614C, + 0x8DC4BD5C, 0xB0A494EC, 0xF704EE3C, 0xCA64C78C + }, { + 0x00000000, 0xCB5CD3A5, 0x4DC8A10B, 0x869472AE, + 0x9B914216, 0x50CD91B3, 0xD659E31D, 0x1D0530B8, + 0xEC53826D, 0x270F51C8, 0xA19B2366, 0x6AC7F0C3, + 0x77C2C07B, 0xBC9E13DE, 0x3A0A6170, 0xF156B2D5, + 0x03D6029B, 0xC88AD13E, 0x4E1EA390, 0x85427035, + 0x9847408D, 0x531B9328, 0xD58FE186, 0x1ED33223, + 0xEF8580F6, 0x24D95353, 0xA24D21FD, 0x6911F258, + 0x7414C2E0, 0xBF481145, 0x39DC63EB, 0xF280B04E, + 0x07AC0536, 0xCCF0D693, 0x4A64A43D, 0x81387798, + 0x9C3D4720, 0x57619485, 0xD1F5E62B, 0x1AA9358E, + 0xEBFF875B, 0x20A354FE, 0xA6372650, 0x6D6BF5F5, + 0x706EC54D, 0xBB3216E8, 0x3DA66446, 0xF6FAB7E3, + 0x047A07AD, 0xCF26D408, 0x49B2A6A6, 0x82EE7503, + 0x9FEB45BB, 0x54B7961E, 0xD223E4B0, 0x197F3715, + 0xE82985C0, 0x23755665, 0xA5E124CB, 0x6EBDF76E, + 0x73B8C7D6, 0xB8E41473, 0x3E7066DD, 0xF52CB578, + 0x0F580A6C, 0xC404D9C9, 0x4290AB67, 0x89CC78C2, + 0x94C9487A, 0x5F959BDF, 0xD901E971, 0x125D3AD4, + 0xE30B8801, 0x28575BA4, 0xAEC3290A, 0x659FFAAF, + 0x789ACA17, 0xB3C619B2, 0x35526B1C, 0xFE0EB8B9, + 0x0C8E08F7, 0xC7D2DB52, 0x4146A9FC, 0x8A1A7A59, + 0x971F4AE1, 0x5C439944, 0xDAD7EBEA, 0x118B384F, + 0xE0DD8A9A, 0x2B81593F, 0xAD152B91, 0x6649F834, + 0x7B4CC88C, 0xB0101B29, 0x36846987, 0xFDD8BA22, + 0x08F40F5A, 0xC3A8DCFF, 0x453CAE51, 0x8E607DF4, + 0x93654D4C, 0x58399EE9, 0xDEADEC47, 0x15F13FE2, + 0xE4A78D37, 0x2FFB5E92, 0xA96F2C3C, 0x6233FF99, + 0x7F36CF21, 0xB46A1C84, 0x32FE6E2A, 0xF9A2BD8F, + 0x0B220DC1, 0xC07EDE64, 0x46EAACCA, 0x8DB67F6F, + 0x90B34FD7, 0x5BEF9C72, 0xDD7BEEDC, 0x16273D79, + 0xE7718FAC, 0x2C2D5C09, 0xAAB92EA7, 0x61E5FD02, + 0x7CE0CDBA, 0xB7BC1E1F, 0x31286CB1, 0xFA74BF14, + 0x1EB014D8, 0xD5ECC77D, 0x5378B5D3, 0x98246676, + 0x852156CE, 0x4E7D856B, 0xC8E9F7C5, 0x03B52460, + 0xF2E396B5, 0x39BF4510, 0xBF2B37BE, 0x7477E41B, + 0x6972D4A3, 0xA22E0706, 0x24BA75A8, 0xEFE6A60D, + 0x1D661643, 0xD63AC5E6, 0x50AEB748, 0x9BF264ED, + 0x86F75455, 0x4DAB87F0, 0xCB3FF55E, 0x006326FB, + 0xF135942E, 0x3A69478B, 0xBCFD3525, 0x77A1E680, + 0x6AA4D638, 0xA1F8059D, 0x276C7733, 0xEC30A496, + 0x191C11EE, 0xD240C24B, 0x54D4B0E5, 0x9F886340, + 0x828D53F8, 0x49D1805D, 0xCF45F2F3, 0x04192156, + 0xF54F9383, 0x3E134026, 0xB8873288, 0x73DBE12D, + 0x6EDED195, 0xA5820230, 0x2316709E, 0xE84AA33B, + 0x1ACA1375, 0xD196C0D0, 0x5702B27E, 0x9C5E61DB, + 0x815B5163, 0x4A0782C6, 0xCC93F068, 0x07CF23CD, + 0xF6999118, 0x3DC542BD, 0xBB513013, 0x700DE3B6, + 0x6D08D30E, 0xA65400AB, 0x20C07205, 0xEB9CA1A0, + 0x11E81EB4, 0xDAB4CD11, 0x5C20BFBF, 0x977C6C1A, + 0x8A795CA2, 0x41258F07, 0xC7B1FDA9, 0x0CED2E0C, + 0xFDBB9CD9, 0x36E74F7C, 0xB0733DD2, 0x7B2FEE77, + 0x662ADECF, 0xAD760D6A, 0x2BE27FC4, 0xE0BEAC61, + 0x123E1C2F, 0xD962CF8A, 0x5FF6BD24, 0x94AA6E81, + 0x89AF5E39, 0x42F38D9C, 0xC467FF32, 0x0F3B2C97, + 0xFE6D9E42, 0x35314DE7, 0xB3A53F49, 0x78F9ECEC, + 0x65FCDC54, 0xAEA00FF1, 0x28347D5F, 0xE368AEFA, + 0x16441B82, 0xDD18C827, 0x5B8CBA89, 0x90D0692C, + 0x8DD55994, 0x46898A31, 0xC01DF89F, 0x0B412B3A, + 0xFA1799EF, 0x314B4A4A, 0xB7DF38E4, 0x7C83EB41, + 0x6186DBF9, 0xAADA085C, 0x2C4E7AF2, 0xE712A957, + 0x15921919, 0xDECECABC, 0x585AB812, 0x93066BB7, + 0x8E035B0F, 0x455F88AA, 0xC3CBFA04, 0x089729A1, + 0xF9C19B74, 0x329D48D1, 0xB4093A7F, 0x7F55E9DA, + 0x6250D962, 0xA90C0AC7, 0x2F987869, 0xE4C4ABCC + }, { + 0x00000000, 0xA6770BB4, 0x979F1129, 0x31E81A9D, + 0xF44F2413, 0x52382FA7, 0x63D0353A, 0xC5A73E8E, + 0x33EF4E67, 0x959845D3, 0xA4705F4E, 0x020754FA, + 0xC7A06A74, 0x61D761C0, 0x503F7B5D, 0xF64870E9, + 0x67DE9CCE, 0xC1A9977A, 0xF0418DE7, 0x56368653, + 0x9391B8DD, 0x35E6B369, 0x040EA9F4, 0xA279A240, + 0x5431D2A9, 0xF246D91D, 0xC3AEC380, 0x65D9C834, + 0xA07EF6BA, 0x0609FD0E, 0x37E1E793, 0x9196EC27, + 0xCFBD399C, 0x69CA3228, 0x582228B5, 0xFE552301, + 0x3BF21D8F, 0x9D85163B, 0xAC6D0CA6, 0x0A1A0712, + 0xFC5277FB, 0x5A257C4F, 0x6BCD66D2, 0xCDBA6D66, + 0x081D53E8, 0xAE6A585C, 0x9F8242C1, 0x39F54975, + 0xA863A552, 0x0E14AEE6, 0x3FFCB47B, 0x998BBFCF, + 0x5C2C8141, 0xFA5B8AF5, 0xCBB39068, 0x6DC49BDC, + 0x9B8CEB35, 0x3DFBE081, 0x0C13FA1C, 0xAA64F1A8, + 0x6FC3CF26, 0xC9B4C492, 0xF85CDE0F, 0x5E2BD5BB, + 0x440B7579, 0xE27C7ECD, 0xD3946450, 0x75E36FE4, + 0xB044516A, 0x16335ADE, 0x27DB4043, 0x81AC4BF7, + 0x77E43B1E, 0xD19330AA, 0xE07B2A37, 0x460C2183, + 0x83AB1F0D, 0x25DC14B9, 0x14340E24, 0xB2430590, + 0x23D5E9B7, 0x85A2E203, 0xB44AF89E, 0x123DF32A, + 0xD79ACDA4, 0x71EDC610, 0x4005DC8D, 0xE672D739, + 0x103AA7D0, 0xB64DAC64, 0x87A5B6F9, 0x21D2BD4D, + 0xE47583C3, 0x42028877, 0x73EA92EA, 0xD59D995E, + 0x8BB64CE5, 0x2DC14751, 0x1C295DCC, 0xBA5E5678, + 0x7FF968F6, 0xD98E6342, 0xE86679DF, 0x4E11726B, + 0xB8590282, 0x1E2E0936, 0x2FC613AB, 0x89B1181F, + 0x4C162691, 0xEA612D25, 0xDB8937B8, 0x7DFE3C0C, + 0xEC68D02B, 0x4A1FDB9F, 0x7BF7C102, 0xDD80CAB6, + 0x1827F438, 0xBE50FF8C, 0x8FB8E511, 0x29CFEEA5, + 0xDF879E4C, 0x79F095F8, 0x48188F65, 0xEE6F84D1, + 0x2BC8BA5F, 0x8DBFB1EB, 0xBC57AB76, 0x1A20A0C2, + 0x8816EAF2, 0x2E61E146, 0x1F89FBDB, 0xB9FEF06F, + 0x7C59CEE1, 0xDA2EC555, 0xEBC6DFC8, 0x4DB1D47C, + 0xBBF9A495, 0x1D8EAF21, 0x2C66B5BC, 0x8A11BE08, + 0x4FB68086, 0xE9C18B32, 0xD82991AF, 0x7E5E9A1B, + 0xEFC8763C, 0x49BF7D88, 0x78576715, 0xDE206CA1, + 0x1B87522F, 0xBDF0599B, 0x8C184306, 0x2A6F48B2, + 0xDC27385B, 0x7A5033EF, 0x4BB82972, 0xEDCF22C6, + 0x28681C48, 0x8E1F17FC, 0xBFF70D61, 0x198006D5, + 0x47ABD36E, 0xE1DCD8DA, 0xD034C247, 0x7643C9F3, + 0xB3E4F77D, 0x1593FCC9, 0x247BE654, 0x820CEDE0, + 0x74449D09, 0xD23396BD, 0xE3DB8C20, 0x45AC8794, + 0x800BB91A, 0x267CB2AE, 0x1794A833, 0xB1E3A387, + 0x20754FA0, 0x86024414, 0xB7EA5E89, 0x119D553D, + 0xD43A6BB3, 0x724D6007, 0x43A57A9A, 0xE5D2712E, + 0x139A01C7, 0xB5ED0A73, 0x840510EE, 0x22721B5A, + 0xE7D525D4, 0x41A22E60, 0x704A34FD, 0xD63D3F49, + 0xCC1D9F8B, 0x6A6A943F, 0x5B828EA2, 0xFDF58516, + 0x3852BB98, 0x9E25B02C, 0xAFCDAAB1, 0x09BAA105, + 0xFFF2D1EC, 0x5985DA58, 0x686DC0C5, 0xCE1ACB71, + 0x0BBDF5FF, 0xADCAFE4B, 0x9C22E4D6, 0x3A55EF62, + 0xABC30345, 0x0DB408F1, 0x3C5C126C, 0x9A2B19D8, + 0x5F8C2756, 0xF9FB2CE2, 0xC813367F, 0x6E643DCB, + 0x982C4D22, 0x3E5B4696, 0x0FB35C0B, 0xA9C457BF, + 0x6C636931, 0xCA146285, 0xFBFC7818, 0x5D8B73AC, + 0x03A0A617, 0xA5D7ADA3, 0x943FB73E, 0x3248BC8A, + 0xF7EF8204, 0x519889B0, 0x6070932D, 0xC6079899, + 0x304FE870, 0x9638E3C4, 0xA7D0F959, 0x01A7F2ED, + 0xC400CC63, 0x6277C7D7, 0x539FDD4A, 0xF5E8D6FE, + 0x647E3AD9, 0xC209316D, 0xF3E12BF0, 0x55962044, + 0x90311ECA, 0x3646157E, 0x07AE0FE3, 0xA1D90457, + 0x579174BE, 0xF1E67F0A, 0xC00E6597, 0x66796E23, + 0xA3DE50AD, 0x05A95B19, 0x34414184, 0x92364A30 + }, { + 0x00000000, 0xCCAA009E, 0x4225077D, 0x8E8F07E3, + 0x844A0EFA, 0x48E00E64, 0xC66F0987, 0x0AC50919, + 0xD3E51BB5, 0x1F4F1B2B, 0x91C01CC8, 0x5D6A1C56, + 0x57AF154F, 0x9B0515D1, 0x158A1232, 0xD92012AC, + 0x7CBB312B, 0xB01131B5, 0x3E9E3656, 0xF23436C8, + 0xF8F13FD1, 0x345B3F4F, 0xBAD438AC, 0x767E3832, + 0xAF5E2A9E, 0x63F42A00, 0xED7B2DE3, 0x21D12D7D, + 0x2B142464, 0xE7BE24FA, 0x69312319, 0xA59B2387, + 0xF9766256, 0x35DC62C8, 0xBB53652B, 0x77F965B5, + 0x7D3C6CAC, 0xB1966C32, 0x3F196BD1, 0xF3B36B4F, + 0x2A9379E3, 0xE639797D, 0x68B67E9E, 0xA41C7E00, + 0xAED97719, 0x62737787, 0xECFC7064, 0x205670FA, + 0x85CD537D, 0x496753E3, 0xC7E85400, 0x0B42549E, + 0x01875D87, 0xCD2D5D19, 0x43A25AFA, 0x8F085A64, + 0x562848C8, 0x9A824856, 0x140D4FB5, 0xD8A74F2B, + 0xD2624632, 0x1EC846AC, 0x9047414F, 0x5CED41D1, + 0x299DC2ED, 0xE537C273, 0x6BB8C590, 0xA712C50E, + 0xADD7CC17, 0x617DCC89, 0xEFF2CB6A, 0x2358CBF4, + 0xFA78D958, 0x36D2D9C6, 0xB85DDE25, 0x74F7DEBB, + 0x7E32D7A2, 0xB298D73C, 0x3C17D0DF, 0xF0BDD041, + 0x5526F3C6, 0x998CF358, 0x1703F4BB, 0xDBA9F425, + 0xD16CFD3C, 0x1DC6FDA2, 0x9349FA41, 0x5FE3FADF, + 0x86C3E873, 0x4A69E8ED, 0xC4E6EF0E, 0x084CEF90, + 0x0289E689, 0xCE23E617, 0x40ACE1F4, 0x8C06E16A, + 0xD0EBA0BB, 0x1C41A025, 0x92CEA7C6, 0x5E64A758, + 0x54A1AE41, 0x980BAEDF, 0x1684A93C, 0xDA2EA9A2, + 0x030EBB0E, 0xCFA4BB90, 0x412BBC73, 0x8D81BCED, + 0x8744B5F4, 0x4BEEB56A, 0xC561B289, 0x09CBB217, + 0xAC509190, 0x60FA910E, 0xEE7596ED, 0x22DF9673, + 0x281A9F6A, 0xE4B09FF4, 0x6A3F9817, 0xA6959889, + 0x7FB58A25, 0xB31F8ABB, 0x3D908D58, 0xF13A8DC6, + 0xFBFF84DF, 0x37558441, 0xB9DA83A2, 0x7570833C, + 0x533B85DA, 0x9F918544, 0x111E82A7, 0xDDB48239, + 0xD7718B20, 0x1BDB8BBE, 0x95548C5D, 0x59FE8CC3, + 0x80DE9E6F, 0x4C749EF1, 0xC2FB9912, 0x0E51998C, + 0x04949095, 0xC83E900B, 0x46B197E8, 0x8A1B9776, + 0x2F80B4F1, 0xE32AB46F, 0x6DA5B38C, 0xA10FB312, + 0xABCABA0B, 0x6760BA95, 0xE9EFBD76, 0x2545BDE8, + 0xFC65AF44, 0x30CFAFDA, 0xBE40A839, 0x72EAA8A7, + 0x782FA1BE, 0xB485A120, 0x3A0AA6C3, 0xF6A0A65D, + 0xAA4DE78C, 0x66E7E712, 0xE868E0F1, 0x24C2E06F, + 0x2E07E976, 0xE2ADE9E8, 0x6C22EE0B, 0xA088EE95, + 0x79A8FC39, 0xB502FCA7, 0x3B8DFB44, 0xF727FBDA, + 0xFDE2F2C3, 0x3148F25D, 0xBFC7F5BE, 0x736DF520, + 0xD6F6D6A7, 0x1A5CD639, 0x94D3D1DA, 0x5879D144, + 0x52BCD85D, 0x9E16D8C3, 0x1099DF20, 0xDC33DFBE, + 0x0513CD12, 0xC9B9CD8C, 0x4736CA6F, 0x8B9CCAF1, + 0x8159C3E8, 0x4DF3C376, 0xC37CC495, 0x0FD6C40B, + 0x7AA64737, 0xB60C47A9, 0x3883404A, 0xF42940D4, + 0xFEEC49CD, 0x32464953, 0xBCC94EB0, 0x70634E2E, + 0xA9435C82, 0x65E95C1C, 0xEB665BFF, 0x27CC5B61, + 0x2D095278, 0xE1A352E6, 0x6F2C5505, 0xA386559B, + 0x061D761C, 0xCAB77682, 0x44387161, 0x889271FF, + 0x825778E6, 0x4EFD7878, 0xC0727F9B, 0x0CD87F05, + 0xD5F86DA9, 0x19526D37, 0x97DD6AD4, 0x5B776A4A, + 0x51B26353, 0x9D1863CD, 0x1397642E, 0xDF3D64B0, + 0x83D02561, 0x4F7A25FF, 0xC1F5221C, 0x0D5F2282, + 0x079A2B9B, 0xCB302B05, 0x45BF2CE6, 0x89152C78, + 0x50353ED4, 0x9C9F3E4A, 0x121039A9, 0xDEBA3937, + 0xD47F302E, 0x18D530B0, 0x965A3753, 0x5AF037CD, + 0xFF6B144A, 0x33C114D4, 0xBD4E1337, 0x71E413A9, + 0x7B211AB0, 0xB78B1A2E, 0x39041DCD, 0xF5AE1D53, + 0x2C8E0FFF, 0xE0240F61, 0x6EAB0882, 0xA201081C, + 0xA8C40105, 0x646E019B, 0xEAE10678, 0x264B06E6 + } +}; diff --git a/src/liblzma/check/crc32_tablegen.c b/src/liblzma/check/crc32_tablegen.c new file mode 100644 index 0000000000..b8cf459f8e --- /dev/null +++ b/src/liblzma/check/crc32_tablegen.c @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc32_tablegen.c +/// \brief Generate crc32_table_le.h and crc32_table_be.h +/// +/// Compiling: gcc -std=c99 -o crc32_tablegen crc32_tablegen.c +/// Add -DWORDS_BIGENDIAN to generate big endian table. +/// Add -DLZ_HASH_TABLE to generate lz_encoder_hash_table.h (little endian). +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include +#include "../../common/tuklib_integer.h" + + +static uint32_t crc32_table[8][256]; + + +static void +init_crc32_table(void) +{ + static const uint32_t poly32 = UINT32_C(0xEDB88320); + + for (size_t s = 0; s < 8; ++s) { + for (size_t b = 0; b < 256; ++b) { + uint32_t r = s == 0 ? b : crc32_table[s - 1][b]; + + for (size_t i = 0; i < 8; ++i) { + if (r & 1) + r = (r >> 1) ^ poly32; + else + r >>= 1; + } + + crc32_table[s][b] = r; + } + } + +#ifdef WORDS_BIGENDIAN + for (size_t s = 0; s < 8; ++s) + for (size_t b = 0; b < 256; ++b) + crc32_table[s][b] = byteswap32(crc32_table[s][b]); +#endif + + return; +} + + +static void +print_crc32_table(void) +{ + // Split the SPDX string so that it won't accidentally match + // when tools search for the string. + printf("// SPDX" "-License-Identifier" ": 0BSD\n\n" + "// This file has been generated by crc32_tablegen.c.\n\n" + "const uint32_t lzma_crc32_table[8][256] = {\n\t{"); + + for (size_t s = 0; s < 8; ++s) { + for (size_t b = 0; b < 256; ++b) { + if ((b % 4) == 0) + printf("\n\t\t"); + + printf("0x%08" PRIX32, crc32_table[s][b]); + + if (b != 255) + printf(",%s", (b+1) % 4 == 0 ? "" : " "); + } + + if (s == 7) + printf("\n\t}\n};\n"); + else + printf("\n\t}, {"); + } + + return; +} + + +static void +print_lz_table(void) +{ + // Split the SPDX string so that it won't accidentally match + // when tools search for the string. + printf("// SPDX" "-License-Identifier" ": 0BSD\n\n" + "// This file has been generated by crc32_tablegen.c.\n\n" + "const uint32_t lzma_lz_hash_table[256] = {"); + + for (size_t b = 0; b < 256; ++b) { + if ((b % 4) == 0) + printf("\n\t"); + + printf("0x%08" PRIX32, crc32_table[0][b]); + + if (b != 255) + printf(",%s", (b+1) % 4 == 0 ? "" : " "); + } + + printf("\n};\n"); + + return; +} + + +int +main(void) +{ + init_crc32_table(); + +#ifdef LZ_HASH_TABLE + print_lz_table(); +#else + print_crc32_table(); +#endif + + return 0; +} diff --git a/src/liblzma/check/crc32_x86.S b/src/liblzma/check/crc32_x86.S new file mode 100644 index 0000000000..37ee063d10 --- /dev/null +++ b/src/liblzma/check/crc32_x86.S @@ -0,0 +1,308 @@ +/* SPDX-License-Identifier: 0BSD */ + +/* + * Speed-optimized CRC32 using slicing-by-eight algorithm + * + * This uses only i386 instructions, but it is optimized for i686 and later + * (including e.g. Pentium II/III/IV, Athlon XP, and Core 2). For i586 + * (e.g. Pentium), slicing-by-four would be better, and even the C version + * of slicing-by-eight built with gcc -march=i586 tends to be a little bit + * better than this. Very few probably run this code on i586 or older x86 + * so this shouldn't be a problem in practice. + * + * Authors: Igor Pavlov (original version) + * Lasse Collin (AT&T syntax, PIC support, better portability) + * + * This code needs lzma_crc32_table, which can be created using the + * following C code: + +uint32_t lzma_crc32_table[8][256]; + +void +init_table(void) +{ + // IEEE-802.3 + static const uint32_t poly32 = UINT32_C(0xEDB88320); + + // Castagnoli + // static const uint32_t poly32 = UINT32_C(0x82F63B78); + + // Koopman + // static const uint32_t poly32 = UINT32_C(0xEB31D82E); + + for (size_t s = 0; s < 8; ++s) { + for (size_t b = 0; b < 256; ++b) { + uint32_t r = s == 0 ? b : lzma_crc32_table[s - 1][b]; + + for (size_t i = 0; i < 8; ++i) { + if (r & 1) + r = (r >> 1) ^ poly32; + else + r >>= 1; + } + + lzma_crc32_table[s][b] = r; + } + } +} + + * The prototype of the CRC32 function: + * extern uint32_t lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc); + */ + +/* When Intel CET is enabled, include in assembly code to mark + Intel CET support. */ +#ifdef __CET__ +# include +#else +# define _CET_ENDBR +#endif + +/* + * On some systems, the functions need to be prefixed. The prefix is + * usually an underscore. + */ +#ifndef __USER_LABEL_PREFIX__ +# define __USER_LABEL_PREFIX__ +#endif +#define MAKE_SYM_CAT(prefix, sym) prefix ## sym +#define MAKE_SYM(prefix, sym) MAKE_SYM_CAT(prefix, sym) +#define LZMA_CRC32 MAKE_SYM(__USER_LABEL_PREFIX__, lzma_crc32_generic) +#define LZMA_CRC32_TABLE MAKE_SYM(__USER_LABEL_PREFIX__, lzma_crc32_table) + +/* + * Solaris assembler doesn't have .p2align, and Darwin uses .align + * differently than GNU/Linux and Solaris. + */ +#if defined(__APPLE__) || defined(__MSDOS__) +# define ALIGN(pow2, abs) .align pow2 +#else +# define ALIGN(pow2, abs) .align abs +#endif + + .text + .globl LZMA_CRC32 +#ifdef __ELF__ + .hidden LZMA_CRC32 +#endif + +#if !defined(__APPLE__) && !defined(_WIN32) && !defined(__CYGWIN__) \ + && !defined(__MSDOS__) + .type LZMA_CRC32, @function +#endif + + ALIGN(4, 16) +LZMA_CRC32: + _CET_ENDBR + /* + * Register usage: + * %eax crc + * %esi buf + * %edi size or buf + size + * %ebx lzma_crc32_table + * %ebp Table index + * %ecx Temporary + * %edx Temporary + */ + pushl %ebx + pushl %esi + pushl %edi + pushl %ebp + movl 0x14(%esp), %esi /* buf */ + movl 0x18(%esp), %edi /* size */ + movl 0x1C(%esp), %eax /* crc */ + + /* + * Store the address of lzma_crc32_table to %ebx. This is needed to + * get position-independent code (PIC). + * + * The PIC macro is defined by libtool, while __PIC__ is defined + * by GCC but only on some systems. Testing for both makes it simpler + * to test this code without libtool, and keeps the code working also + * when built with libtool but using something else than GCC. + * + * I understood that libtool may define PIC on Windows even though + * the code in Windows DLLs is not PIC in sense that it is in ELF + * binaries, so we need a separate check to always use the non-PIC + * code on Windows. + */ +#if (!defined(PIC) && !defined(__PIC__)) \ + || (defined(_WIN32) || defined(__CYGWIN__)) + /* Not PIC */ + movl $ LZMA_CRC32_TABLE, %ebx +#elif defined(__APPLE__) + /* Mach-O */ + call .L_get_pc +.L_pic: + leal .L_lzma_crc32_table$non_lazy_ptr-.L_pic(%ebx), %ebx + movl (%ebx), %ebx +#else + /* ELF */ + call .L_get_pc + addl $_GLOBAL_OFFSET_TABLE_, %ebx + movl LZMA_CRC32_TABLE@GOT(%ebx), %ebx +#endif + + /* Complement the initial value. */ + notl %eax + + ALIGN(4, 16) +.L_align: + /* + * Check if there is enough input to use slicing-by-eight. + * We need 16 bytes, because the loop pre-reads eight bytes. + */ + cmpl $16, %edi + jb .L_rest + + /* Check if we have reached alignment of eight bytes. */ + testl $7, %esi + jz .L_slice + + /* Calculate CRC of the next input byte. */ + movzbl (%esi), %ebp + incl %esi + movzbl %al, %ecx + xorl %ecx, %ebp + shrl $8, %eax + xorl (%ebx, %ebp, 4), %eax + decl %edi + jmp .L_align + + ALIGN(2, 4) +.L_slice: + /* + * If we get here, there's at least 16 bytes of aligned input + * available. Make %edi multiple of eight bytes. Store the possible + * remainder over the "size" variable in the argument stack. + */ + movl %edi, 0x18(%esp) + andl $-8, %edi + subl %edi, 0x18(%esp) + + /* + * Let %edi be buf + size - 8 while running the main loop. This way + * we can compare for equality to determine when exit the loop. + */ + addl %esi, %edi + subl $8, %edi + + /* Read in the first eight aligned bytes. */ + xorl (%esi), %eax + movl 4(%esi), %ecx + movzbl %cl, %ebp + +.L_loop: + movl 0x0C00(%ebx, %ebp, 4), %edx + movzbl %ch, %ebp + xorl 0x0800(%ebx, %ebp, 4), %edx + shrl $16, %ecx + xorl 8(%esi), %edx + movzbl %cl, %ebp + xorl 0x0400(%ebx, %ebp, 4), %edx + movzbl %ch, %ebp + xorl (%ebx, %ebp, 4), %edx + movzbl %al, %ebp + + /* + * Read the next four bytes, for which the CRC is calculated + * on the next iteration of the loop. + */ + movl 12(%esi), %ecx + + xorl 0x1C00(%ebx, %ebp, 4), %edx + movzbl %ah, %ebp + shrl $16, %eax + xorl 0x1800(%ebx, %ebp, 4), %edx + movzbl %ah, %ebp + movzbl %al, %eax + movl 0x1400(%ebx, %eax, 4), %eax + addl $8, %esi + xorl %edx, %eax + xorl 0x1000(%ebx, %ebp, 4), %eax + + /* Check for end of aligned input. */ + cmpl %edi, %esi + movzbl %cl, %ebp + jne .L_loop + + /* + * Process the remaining eight bytes, which we have already + * copied to %ecx and %edx. + */ + movl 0x0C00(%ebx, %ebp, 4), %edx + movzbl %ch, %ebp + xorl 0x0800(%ebx, %ebp, 4), %edx + shrl $16, %ecx + movzbl %cl, %ebp + xorl 0x0400(%ebx, %ebp, 4), %edx + movzbl %ch, %ebp + xorl (%ebx, %ebp, 4), %edx + movzbl %al, %ebp + + xorl 0x1C00(%ebx, %ebp, 4), %edx + movzbl %ah, %ebp + shrl $16, %eax + xorl 0x1800(%ebx, %ebp, 4), %edx + movzbl %ah, %ebp + movzbl %al, %eax + movl 0x1400(%ebx, %eax, 4), %eax + addl $8, %esi + xorl %edx, %eax + xorl 0x1000(%ebx, %ebp, 4), %eax + + /* Copy the number of remaining bytes to %edi. */ + movl 0x18(%esp), %edi + +.L_rest: + /* Check for end of input. */ + testl %edi, %edi + jz .L_return + + /* Calculate CRC of the next input byte. */ + movzbl (%esi), %ebp + incl %esi + movzbl %al, %ecx + xorl %ecx, %ebp + shrl $8, %eax + xorl (%ebx, %ebp, 4), %eax + decl %edi + jmp .L_rest + +.L_return: + /* Complement the final value. */ + notl %eax + + popl %ebp + popl %edi + popl %esi + popl %ebx + ret + +#if defined(PIC) || defined(__PIC__) + ALIGN(4, 16) +.L_get_pc: + movl (%esp), %ebx + ret +#endif + +#if defined(__APPLE__) && (defined(PIC) || defined(__PIC__)) + /* Mach-O PIC */ + .section __IMPORT,__pointers,non_lazy_symbol_pointers +.L_lzma_crc32_table$non_lazy_ptr: + .indirect_symbol LZMA_CRC32_TABLE + .long 0 + +#elif !defined(_WIN32) && !defined(__CYGWIN__) && !defined(__MSDOS__) + /* ELF */ + .size LZMA_CRC32, .-LZMA_CRC32 +#endif + +/* + * This is needed to support non-executable stack. It's ugly to + * use __FreeBSD__ and __linux__ here, but I don't know a way to detect when + * we are using GNU assembler. + */ +#if defined(__ELF__) && (defined(__FreeBSD__) || defined(__linux__)) + .section .note.GNU-stack,"",@progbits +#endif diff --git a/src/liblzma/check/crc64_fast.c b/src/liblzma/check/crc64_fast.c new file mode 100644 index 0000000000..8a6770a431 --- /dev/null +++ b/src/liblzma/check/crc64_fast.c @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc64.c +/// \brief CRC64 calculation +// +// Authors: Lasse Collin +// Ilya Kurdyukov +// +/////////////////////////////////////////////////////////////////////////////// + +#include "check.h" +#include "crc_common.h" + +#if defined(CRC_X86_CLMUL) +# define BUILDING_CRC_CLMUL 64 +# include "crc_x86_clmul.h" +#endif + + +#ifdef CRC64_GENERIC + +///////////////////////////////// +// Generic slice-by-four CRC64 // +///////////////////////////////// + +#if defined(WORDS_BIGENDIAN) +# include "crc64_table_be.h" +#else +# include "crc64_table_le.h" +#endif + + +#ifdef HAVE_CRC_X86_ASM +extern uint64_t lzma_crc64_generic( + const uint8_t *buf, size_t size, uint64_t crc); +#else + +#ifdef WORDS_BIGENDIAN +# define A1(x) ((x) >> 56) +#else +# define A1 A +#endif + + +// See the comments in crc32_fast.c. They aren't duplicated here. +static uint64_t +lzma_crc64_generic(const uint8_t *buf, size_t size, uint64_t crc) +{ + crc = ~crc; + +#ifdef WORDS_BIGENDIAN + crc = byteswap64(crc); +#endif + + if (size > 4) { + while ((uintptr_t)(buf) & 3) { + crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc); + --size; + } + + const uint8_t *const limit = buf + (size & ~(size_t)(3)); + size &= (size_t)(3); + + while (buf < limit) { +#ifdef WORDS_BIGENDIAN + const uint32_t tmp = (uint32_t)(crc >> 32) + ^ aligned_read32ne(buf); +#else + const uint32_t tmp = (uint32_t)crc + ^ aligned_read32ne(buf); +#endif + buf += 4; + + crc = lzma_crc64_table[3][A(tmp)] + ^ lzma_crc64_table[2][B(tmp)] + ^ S32(crc) + ^ lzma_crc64_table[1][C(tmp)] + ^ lzma_crc64_table[0][D(tmp)]; + } + } + + while (size-- != 0) + crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc); + +#ifdef WORDS_BIGENDIAN + crc = byteswap64(crc); +#endif + + return ~crc; +} +#endif // HAVE_CRC_X86_ASM +#endif // CRC64_GENERIC + + +#if defined(CRC64_GENERIC) && defined(CRC64_ARCH_OPTIMIZED) + +////////////////////////// +// Function dispatching // +////////////////////////// + +// If both the generic and arch-optimized implementations are usable, then +// the function that is used is selected at runtime. See crc32_fast.c. + +typedef uint64_t (*crc64_func_type)( + const uint8_t *buf, size_t size, uint64_t crc); + +static crc64_func_type +crc64_resolve(void) +{ + return is_arch_extension_supported() + ? &crc64_arch_optimized : &lzma_crc64_generic; +} + +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +# define CRC64_SET_FUNC_ATTR __attribute__((__constructor__)) +static crc64_func_type crc64_func; +#else +# define CRC64_SET_FUNC_ATTR +static uint64_t crc64_dispatch(const uint8_t *buf, size_t size, uint64_t crc); +static crc64_func_type crc64_func = &crc64_dispatch; +#endif + + +CRC64_SET_FUNC_ATTR +static void +crc64_set_func(void) +{ + crc64_func = crc64_resolve(); + return; +} + + +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +static uint64_t +crc64_dispatch(const uint8_t *buf, size_t size, uint64_t crc) +{ + crc64_set_func(); + return crc64_func(buf, size, crc); +} +#endif +#endif + + +extern LZMA_API(uint64_t) +lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) +{ +#if defined(_MSC_VER) && !defined(__INTEL_COMPILER) && !defined(__clang__) \ + && defined(_M_IX86) && defined(CRC64_ARCH_OPTIMIZED) + // VS2015-2022 might corrupt the ebx register on 32-bit x86 when + // the CLMUL code is enabled. This hack forces MSVC to store and + // restore ebx. This is only needed here, not in lzma_crc32(). + __asm mov ebx, ebx +#endif + +#if defined(CRC64_GENERIC) && defined(CRC64_ARCH_OPTIMIZED) + return crc64_func(buf, size, crc); + +#elif defined(CRC64_ARCH_OPTIMIZED) + // If arch-optimized version is used unconditionally without runtime + // CPU detection then omitting the generic version and its 8 KiB + // lookup table makes the library smaller. + return crc64_arch_optimized(buf, size, crc); + +#else + return lzma_crc64_generic(buf, size, crc); +#endif +} diff --git a/src/liblzma/check/crc64_small.c b/src/liblzma/check/crc64_small.c new file mode 100644 index 0000000000..ee4ea26f67 --- /dev/null +++ b/src/liblzma/check/crc64_small.c @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc64_small.c +/// \brief CRC64 calculation (size-optimized) +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "check.h" + + +static uint64_t crc64_table[256]; + + +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +__attribute__((__constructor__)) +#endif +static void +crc64_init(void) +{ + static const uint64_t poly64 = UINT64_C(0xC96C5795D7870F42); + + for (size_t b = 0; b < 256; ++b) { + uint64_t r = b; + for (size_t i = 0; i < 8; ++i) { + if (r & 1) + r = (r >> 1) ^ poly64; + else + r >>= 1; + } + + crc64_table[b] = r; + } + + return; +} + + +extern LZMA_API(uint64_t) +lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) +{ +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR + mythread_once(crc64_init); +#endif + + crc = ~crc; + + while (size != 0) { + crc = crc64_table[*buf++ ^ (crc & 0xFF)] ^ (crc >> 8); + --size; + } + + return ~crc; +} diff --git a/src/liblzma/check/crc64_table_be.h b/src/liblzma/check/crc64_table_be.h new file mode 100644 index 0000000000..db76cc70e0 --- /dev/null +++ b/src/liblzma/check/crc64_table_be.h @@ -0,0 +1,523 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by crc64_tablegen.c. + +const uint64_t lzma_crc64_table[4][256] = { + { + UINT64_C(0x0000000000000000), UINT64_C(0x6F5FA703BE4C2EB3), + UINT64_C(0x5BA040A8573684F4), UINT64_C(0x34FFE7ABE97AAA47), + UINT64_C(0x335E8FFF84C3D07B), UINT64_C(0x5C0128FC3A8FFEC8), + UINT64_C(0x68FECF57D3F5548F), UINT64_C(0x07A168546DB97A3C), + UINT64_C(0x66BC1EFF0987A1F7), UINT64_C(0x09E3B9FCB7CB8F44), + UINT64_C(0x3D1C5E575EB12503), UINT64_C(0x5243F954E0FD0BB0), + UINT64_C(0x55E291008D44718C), UINT64_C(0x3ABD360333085F3F), + UINT64_C(0x0E42D1A8DA72F578), UINT64_C(0x611D76AB643EDBCB), + UINT64_C(0x4966335138A19B7D), UINT64_C(0x2639945286EDB5CE), + UINT64_C(0x12C673F96F971F89), UINT64_C(0x7D99D4FAD1DB313A), + UINT64_C(0x7A38BCAEBC624B06), UINT64_C(0x15671BAD022E65B5), + UINT64_C(0x2198FC06EB54CFF2), UINT64_C(0x4EC75B055518E141), + UINT64_C(0x2FDA2DAE31263A8A), UINT64_C(0x40858AAD8F6A1439), + UINT64_C(0x747A6D066610BE7E), UINT64_C(0x1B25CA05D85C90CD), + UINT64_C(0x1C84A251B5E5EAF1), UINT64_C(0x73DB05520BA9C442), + UINT64_C(0x4724E2F9E2D36E05), UINT64_C(0x287B45FA5C9F40B6), + UINT64_C(0x92CC66A2704237FB), UINT64_C(0xFD93C1A1CE0E1948), + UINT64_C(0xC96C260A2774B30F), UINT64_C(0xA633810999389DBC), + UINT64_C(0xA192E95DF481E780), UINT64_C(0xCECD4E5E4ACDC933), + UINT64_C(0xFA32A9F5A3B76374), UINT64_C(0x956D0EF61DFB4DC7), + UINT64_C(0xF470785D79C5960C), UINT64_C(0x9B2FDF5EC789B8BF), + UINT64_C(0xAFD038F52EF312F8), UINT64_C(0xC08F9FF690BF3C4B), + UINT64_C(0xC72EF7A2FD064677), UINT64_C(0xA87150A1434A68C4), + UINT64_C(0x9C8EB70AAA30C283), UINT64_C(0xF3D11009147CEC30), + UINT64_C(0xDBAA55F348E3AC86), UINT64_C(0xB4F5F2F0F6AF8235), + UINT64_C(0x800A155B1FD52872), UINT64_C(0xEF55B258A19906C1), + UINT64_C(0xE8F4DA0CCC207CFD), UINT64_C(0x87AB7D0F726C524E), + UINT64_C(0xB3549AA49B16F809), UINT64_C(0xDC0B3DA7255AD6BA), + UINT64_C(0xBD164B0C41640D71), UINT64_C(0xD249EC0FFF2823C2), + UINT64_C(0xE6B60BA416528985), UINT64_C(0x89E9ACA7A81EA736), + UINT64_C(0x8E48C4F3C5A7DD0A), UINT64_C(0xE11763F07BEBF3B9), + UINT64_C(0xD5E8845B929159FE), UINT64_C(0xBAB723582CDD774D), + UINT64_C(0xA187C3EBCA2BB664), UINT64_C(0xCED864E8746798D7), + UINT64_C(0xFA2783439D1D3290), UINT64_C(0x9578244023511C23), + UINT64_C(0x92D94C144EE8661F), UINT64_C(0xFD86EB17F0A448AC), + UINT64_C(0xC9790CBC19DEE2EB), UINT64_C(0xA626ABBFA792CC58), + UINT64_C(0xC73BDD14C3AC1793), UINT64_C(0xA8647A177DE03920), + UINT64_C(0x9C9B9DBC949A9367), UINT64_C(0xF3C43ABF2AD6BDD4), + UINT64_C(0xF46552EB476FC7E8), UINT64_C(0x9B3AF5E8F923E95B), + UINT64_C(0xAFC512431059431C), UINT64_C(0xC09AB540AE156DAF), + UINT64_C(0xE8E1F0BAF28A2D19), UINT64_C(0x87BE57B94CC603AA), + UINT64_C(0xB341B012A5BCA9ED), UINT64_C(0xDC1E17111BF0875E), + UINT64_C(0xDBBF7F457649FD62), UINT64_C(0xB4E0D846C805D3D1), + UINT64_C(0x801F3FED217F7996), UINT64_C(0xEF4098EE9F335725), + UINT64_C(0x8E5DEE45FB0D8CEE), UINT64_C(0xE10249464541A25D), + UINT64_C(0xD5FDAEEDAC3B081A), UINT64_C(0xBAA209EE127726A9), + UINT64_C(0xBD0361BA7FCE5C95), UINT64_C(0xD25CC6B9C1827226), + UINT64_C(0xE6A3211228F8D861), UINT64_C(0x89FC861196B4F6D2), + UINT64_C(0x334BA549BA69819F), UINT64_C(0x5C14024A0425AF2C), + UINT64_C(0x68EBE5E1ED5F056B), UINT64_C(0x07B442E253132BD8), + UINT64_C(0x00152AB63EAA51E4), UINT64_C(0x6F4A8DB580E67F57), + UINT64_C(0x5BB56A1E699CD510), UINT64_C(0x34EACD1DD7D0FBA3), + UINT64_C(0x55F7BBB6B3EE2068), UINT64_C(0x3AA81CB50DA20EDB), + UINT64_C(0x0E57FB1EE4D8A49C), UINT64_C(0x61085C1D5A948A2F), + UINT64_C(0x66A93449372DF013), UINT64_C(0x09F6934A8961DEA0), + UINT64_C(0x3D0974E1601B74E7), UINT64_C(0x5256D3E2DE575A54), + UINT64_C(0x7A2D961882C81AE2), UINT64_C(0x1572311B3C843451), + UINT64_C(0x218DD6B0D5FE9E16), UINT64_C(0x4ED271B36BB2B0A5), + UINT64_C(0x497319E7060BCA99), UINT64_C(0x262CBEE4B847E42A), + UINT64_C(0x12D3594F513D4E6D), UINT64_C(0x7D8CFE4CEF7160DE), + UINT64_C(0x1C9188E78B4FBB15), UINT64_C(0x73CE2FE4350395A6), + UINT64_C(0x4731C84FDC793FE1), UINT64_C(0x286E6F4C62351152), + UINT64_C(0x2FCF07180F8C6B6E), UINT64_C(0x4090A01BB1C045DD), + UINT64_C(0x746F47B058BAEF9A), UINT64_C(0x1B30E0B3E6F6C129), + UINT64_C(0x420F87D795576CC9), UINT64_C(0x2D5020D42B1B427A), + UINT64_C(0x19AFC77FC261E83D), UINT64_C(0x76F0607C7C2DC68E), + UINT64_C(0x715108281194BCB2), UINT64_C(0x1E0EAF2BAFD89201), + UINT64_C(0x2AF1488046A23846), UINT64_C(0x45AEEF83F8EE16F5), + UINT64_C(0x24B399289CD0CD3E), UINT64_C(0x4BEC3E2B229CE38D), + UINT64_C(0x7F13D980CBE649CA), UINT64_C(0x104C7E8375AA6779), + UINT64_C(0x17ED16D718131D45), UINT64_C(0x78B2B1D4A65F33F6), + UINT64_C(0x4C4D567F4F2599B1), UINT64_C(0x2312F17CF169B702), + UINT64_C(0x0B69B486ADF6F7B4), UINT64_C(0x6436138513BAD907), + UINT64_C(0x50C9F42EFAC07340), UINT64_C(0x3F96532D448C5DF3), + UINT64_C(0x38373B79293527CF), UINT64_C(0x57689C7A9779097C), + UINT64_C(0x63977BD17E03A33B), UINT64_C(0x0CC8DCD2C04F8D88), + UINT64_C(0x6DD5AA79A4715643), UINT64_C(0x028A0D7A1A3D78F0), + UINT64_C(0x3675EAD1F347D2B7), UINT64_C(0x592A4DD24D0BFC04), + UINT64_C(0x5E8B258620B28638), UINT64_C(0x31D482859EFEA88B), + UINT64_C(0x052B652E778402CC), UINT64_C(0x6A74C22DC9C82C7F), + UINT64_C(0xD0C3E175E5155B32), UINT64_C(0xBF9C46765B597581), + UINT64_C(0x8B63A1DDB223DFC6), UINT64_C(0xE43C06DE0C6FF175), + UINT64_C(0xE39D6E8A61D68B49), UINT64_C(0x8CC2C989DF9AA5FA), + UINT64_C(0xB83D2E2236E00FBD), UINT64_C(0xD762892188AC210E), + UINT64_C(0xB67FFF8AEC92FAC5), UINT64_C(0xD920588952DED476), + UINT64_C(0xEDDFBF22BBA47E31), UINT64_C(0x8280182105E85082), + UINT64_C(0x8521707568512ABE), UINT64_C(0xEA7ED776D61D040D), + UINT64_C(0xDE8130DD3F67AE4A), UINT64_C(0xB1DE97DE812B80F9), + UINT64_C(0x99A5D224DDB4C04F), UINT64_C(0xF6FA752763F8EEFC), + UINT64_C(0xC205928C8A8244BB), UINT64_C(0xAD5A358F34CE6A08), + UINT64_C(0xAAFB5DDB59771034), UINT64_C(0xC5A4FAD8E73B3E87), + UINT64_C(0xF15B1D730E4194C0), UINT64_C(0x9E04BA70B00DBA73), + UINT64_C(0xFF19CCDBD43361B8), UINT64_C(0x90466BD86A7F4F0B), + UINT64_C(0xA4B98C738305E54C), UINT64_C(0xCBE62B703D49CBFF), + UINT64_C(0xCC47432450F0B1C3), UINT64_C(0xA318E427EEBC9F70), + UINT64_C(0x97E7038C07C63537), UINT64_C(0xF8B8A48FB98A1B84), + UINT64_C(0xE388443C5F7CDAAD), UINT64_C(0x8CD7E33FE130F41E), + UINT64_C(0xB8280494084A5E59), UINT64_C(0xD777A397B60670EA), + UINT64_C(0xD0D6CBC3DBBF0AD6), UINT64_C(0xBF896CC065F32465), + UINT64_C(0x8B768B6B8C898E22), UINT64_C(0xE4292C6832C5A091), + UINT64_C(0x85345AC356FB7B5A), UINT64_C(0xEA6BFDC0E8B755E9), + UINT64_C(0xDE941A6B01CDFFAE), UINT64_C(0xB1CBBD68BF81D11D), + UINT64_C(0xB66AD53CD238AB21), UINT64_C(0xD935723F6C748592), + UINT64_C(0xEDCA9594850E2FD5), UINT64_C(0x829532973B420166), + UINT64_C(0xAAEE776D67DD41D0), UINT64_C(0xC5B1D06ED9916F63), + UINT64_C(0xF14E37C530EBC524), UINT64_C(0x9E1190C68EA7EB97), + UINT64_C(0x99B0F892E31E91AB), UINT64_C(0xF6EF5F915D52BF18), + UINT64_C(0xC210B83AB428155F), UINT64_C(0xAD4F1F390A643BEC), + UINT64_C(0xCC5269926E5AE027), UINT64_C(0xA30DCE91D016CE94), + UINT64_C(0x97F2293A396C64D3), UINT64_C(0xF8AD8E3987204A60), + UINT64_C(0xFF0CE66DEA99305C), UINT64_C(0x9053416E54D51EEF), + UINT64_C(0xA4ACA6C5BDAFB4A8), UINT64_C(0xCBF301C603E39A1B), + UINT64_C(0x7144229E2F3EED56), UINT64_C(0x1E1B859D9172C3E5), + UINT64_C(0x2AE46236780869A2), UINT64_C(0x45BBC535C6444711), + UINT64_C(0x421AAD61ABFD3D2D), UINT64_C(0x2D450A6215B1139E), + UINT64_C(0x19BAEDC9FCCBB9D9), UINT64_C(0x76E54ACA4287976A), + UINT64_C(0x17F83C6126B94CA1), UINT64_C(0x78A79B6298F56212), + UINT64_C(0x4C587CC9718FC855), UINT64_C(0x2307DBCACFC3E6E6), + UINT64_C(0x24A6B39EA27A9CDA), UINT64_C(0x4BF9149D1C36B269), + UINT64_C(0x7F06F336F54C182E), UINT64_C(0x105954354B00369D), + UINT64_C(0x382211CF179F762B), UINT64_C(0x577DB6CCA9D35898), + UINT64_C(0x6382516740A9F2DF), UINT64_C(0x0CDDF664FEE5DC6C), + UINT64_C(0x0B7C9E30935CA650), UINT64_C(0x642339332D1088E3), + UINT64_C(0x50DCDE98C46A22A4), UINT64_C(0x3F83799B7A260C17), + UINT64_C(0x5E9E0F301E18D7DC), UINT64_C(0x31C1A833A054F96F), + UINT64_C(0x053E4F98492E5328), UINT64_C(0x6A61E89BF7627D9B), + UINT64_C(0x6DC080CF9ADB07A7), UINT64_C(0x029F27CC24972914), + UINT64_C(0x3660C067CDED8353), UINT64_C(0x593F676473A1ADE0) + }, { + UINT64_C(0x0000000000000000), UINT64_C(0x0DF1D05C9279E954), + UINT64_C(0x1AE2A1B924F3D2A9), UINT64_C(0x171371E5B68A3BFD), + UINT64_C(0xB1DA4DDC62497DC1), UINT64_C(0xBC2B9D80F0309495), + UINT64_C(0xAB38EC6546BAAF68), UINT64_C(0xA6C93C39D4C3463C), + UINT64_C(0xE7AB9517EE3D2210), UINT64_C(0xEA5A454B7C44CB44), + UINT64_C(0xFD4934AECACEF0B9), UINT64_C(0xF0B8E4F258B719ED), + UINT64_C(0x5671D8CB8C745FD1), UINT64_C(0x5B8008971E0DB685), + UINT64_C(0x4C937972A8878D78), UINT64_C(0x4162A92E3AFE642C), + UINT64_C(0xCE572B2FDC7B4420), UINT64_C(0xC3A6FB734E02AD74), + UINT64_C(0xD4B58A96F8889689), UINT64_C(0xD9445ACA6AF17FDD), + UINT64_C(0x7F8D66F3BE3239E1), UINT64_C(0x727CB6AF2C4BD0B5), + UINT64_C(0x656FC74A9AC1EB48), UINT64_C(0x689E171608B8021C), + UINT64_C(0x29FCBE3832466630), UINT64_C(0x240D6E64A03F8F64), + UINT64_C(0x331E1F8116B5B499), UINT64_C(0x3EEFCFDD84CC5DCD), + UINT64_C(0x9826F3E4500F1BF1), UINT64_C(0x95D723B8C276F2A5), + UINT64_C(0x82C4525D74FCC958), UINT64_C(0x8F358201E685200C), + UINT64_C(0x9CAF565EB8F78840), UINT64_C(0x915E86022A8E6114), + UINT64_C(0x864DF7E79C045AE9), UINT64_C(0x8BBC27BB0E7DB3BD), + UINT64_C(0x2D751B82DABEF581), UINT64_C(0x2084CBDE48C71CD5), + UINT64_C(0x3797BA3BFE4D2728), UINT64_C(0x3A666A676C34CE7C), + UINT64_C(0x7B04C34956CAAA50), UINT64_C(0x76F51315C4B34304), + UINT64_C(0x61E662F0723978F9), UINT64_C(0x6C17B2ACE04091AD), + UINT64_C(0xCADE8E953483D791), UINT64_C(0xC72F5EC9A6FA3EC5), + UINT64_C(0xD03C2F2C10700538), UINT64_C(0xDDCDFF708209EC6C), + UINT64_C(0x52F87D71648CCC60), UINT64_C(0x5F09AD2DF6F52534), + UINT64_C(0x481ADCC8407F1EC9), UINT64_C(0x45EB0C94D206F79D), + UINT64_C(0xE32230AD06C5B1A1), UINT64_C(0xEED3E0F194BC58F5), + UINT64_C(0xF9C0911422366308), UINT64_C(0xF4314148B04F8A5C), + UINT64_C(0xB553E8668AB1EE70), UINT64_C(0xB8A2383A18C80724), + UINT64_C(0xAFB149DFAE423CD9), UINT64_C(0xA24099833C3BD58D), + UINT64_C(0x0489A5BAE8F893B1), UINT64_C(0x097875E67A817AE5), + UINT64_C(0x1E6B0403CC0B4118), UINT64_C(0x139AD45F5E72A84C), + UINT64_C(0x385FADBC70EF1181), UINT64_C(0x35AE7DE0E296F8D5), + UINT64_C(0x22BD0C05541CC328), UINT64_C(0x2F4CDC59C6652A7C), + UINT64_C(0x8985E06012A66C40), UINT64_C(0x8474303C80DF8514), + UINT64_C(0x936741D93655BEE9), UINT64_C(0x9E969185A42C57BD), + UINT64_C(0xDFF438AB9ED23391), UINT64_C(0xD205E8F70CABDAC5), + UINT64_C(0xC5169912BA21E138), UINT64_C(0xC8E7494E2858086C), + UINT64_C(0x6E2E7577FC9B4E50), UINT64_C(0x63DFA52B6EE2A704), + UINT64_C(0x74CCD4CED8689CF9), UINT64_C(0x793D04924A1175AD), + UINT64_C(0xF6088693AC9455A1), UINT64_C(0xFBF956CF3EEDBCF5), + UINT64_C(0xECEA272A88678708), UINT64_C(0xE11BF7761A1E6E5C), + UINT64_C(0x47D2CB4FCEDD2860), UINT64_C(0x4A231B135CA4C134), + UINT64_C(0x5D306AF6EA2EFAC9), UINT64_C(0x50C1BAAA7857139D), + UINT64_C(0x11A3138442A977B1), UINT64_C(0x1C52C3D8D0D09EE5), + UINT64_C(0x0B41B23D665AA518), UINT64_C(0x06B06261F4234C4C), + UINT64_C(0xA0795E5820E00A70), UINT64_C(0xAD888E04B299E324), + UINT64_C(0xBA9BFFE10413D8D9), UINT64_C(0xB76A2FBD966A318D), + UINT64_C(0xA4F0FBE2C81899C1), UINT64_C(0xA9012BBE5A617095), + UINT64_C(0xBE125A5BECEB4B68), UINT64_C(0xB3E38A077E92A23C), + UINT64_C(0x152AB63EAA51E400), UINT64_C(0x18DB666238280D54), + UINT64_C(0x0FC817878EA236A9), UINT64_C(0x0239C7DB1CDBDFFD), + UINT64_C(0x435B6EF52625BBD1), UINT64_C(0x4EAABEA9B45C5285), + UINT64_C(0x59B9CF4C02D66978), UINT64_C(0x54481F1090AF802C), + UINT64_C(0xF2812329446CC610), UINT64_C(0xFF70F375D6152F44), + UINT64_C(0xE8638290609F14B9), UINT64_C(0xE59252CCF2E6FDED), + UINT64_C(0x6AA7D0CD1463DDE1), UINT64_C(0x67560091861A34B5), + UINT64_C(0x7045717430900F48), UINT64_C(0x7DB4A128A2E9E61C), + UINT64_C(0xDB7D9D11762AA020), UINT64_C(0xD68C4D4DE4534974), + UINT64_C(0xC19F3CA852D97289), UINT64_C(0xCC6EECF4C0A09BDD), + UINT64_C(0x8D0C45DAFA5EFFF1), UINT64_C(0x80FD9586682716A5), + UINT64_C(0x97EEE463DEAD2D58), UINT64_C(0x9A1F343F4CD4C40C), + UINT64_C(0x3CD6080698178230), UINT64_C(0x3127D85A0A6E6B64), + UINT64_C(0x2634A9BFBCE45099), UINT64_C(0x2BC579E32E9DB9CD), + UINT64_C(0xF5A054D6CA71FB90), UINT64_C(0xF851848A580812C4), + UINT64_C(0xEF42F56FEE822939), UINT64_C(0xE2B325337CFBC06D), + UINT64_C(0x447A190AA8388651), UINT64_C(0x498BC9563A416F05), + UINT64_C(0x5E98B8B38CCB54F8), UINT64_C(0x536968EF1EB2BDAC), + UINT64_C(0x120BC1C1244CD980), UINT64_C(0x1FFA119DB63530D4), + UINT64_C(0x08E9607800BF0B29), UINT64_C(0x0518B02492C6E27D), + UINT64_C(0xA3D18C1D4605A441), UINT64_C(0xAE205C41D47C4D15), + UINT64_C(0xB9332DA462F676E8), UINT64_C(0xB4C2FDF8F08F9FBC), + UINT64_C(0x3BF77FF9160ABFB0), UINT64_C(0x3606AFA5847356E4), + UINT64_C(0x2115DE4032F96D19), UINT64_C(0x2CE40E1CA080844D), + UINT64_C(0x8A2D32257443C271), UINT64_C(0x87DCE279E63A2B25), + UINT64_C(0x90CF939C50B010D8), UINT64_C(0x9D3E43C0C2C9F98C), + UINT64_C(0xDC5CEAEEF8379DA0), UINT64_C(0xD1AD3AB26A4E74F4), + UINT64_C(0xC6BE4B57DCC44F09), UINT64_C(0xCB4F9B0B4EBDA65D), + UINT64_C(0x6D86A7329A7EE061), UINT64_C(0x6077776E08070935), + UINT64_C(0x7764068BBE8D32C8), UINT64_C(0x7A95D6D72CF4DB9C), + UINT64_C(0x690F0288728673D0), UINT64_C(0x64FED2D4E0FF9A84), + UINT64_C(0x73EDA3315675A179), UINT64_C(0x7E1C736DC40C482D), + UINT64_C(0xD8D54F5410CF0E11), UINT64_C(0xD5249F0882B6E745), + UINT64_C(0xC237EEED343CDCB8), UINT64_C(0xCFC63EB1A64535EC), + UINT64_C(0x8EA4979F9CBB51C0), UINT64_C(0x835547C30EC2B894), + UINT64_C(0x94463626B8488369), UINT64_C(0x99B7E67A2A316A3D), + UINT64_C(0x3F7EDA43FEF22C01), UINT64_C(0x328F0A1F6C8BC555), + UINT64_C(0x259C7BFADA01FEA8), UINT64_C(0x286DABA6487817FC), + UINT64_C(0xA75829A7AEFD37F0), UINT64_C(0xAAA9F9FB3C84DEA4), + UINT64_C(0xBDBA881E8A0EE559), UINT64_C(0xB04B584218770C0D), + UINT64_C(0x1682647BCCB44A31), UINT64_C(0x1B73B4275ECDA365), + UINT64_C(0x0C60C5C2E8479898), UINT64_C(0x0191159E7A3E71CC), + UINT64_C(0x40F3BCB040C015E0), UINT64_C(0x4D026CECD2B9FCB4), + UINT64_C(0x5A111D096433C749), UINT64_C(0x57E0CD55F64A2E1D), + UINT64_C(0xF129F16C22896821), UINT64_C(0xFCD82130B0F08175), + UINT64_C(0xEBCB50D5067ABA88), UINT64_C(0xE63A8089940353DC), + UINT64_C(0xCDFFF96ABA9EEA11), UINT64_C(0xC00E293628E70345), + UINT64_C(0xD71D58D39E6D38B8), UINT64_C(0xDAEC888F0C14D1EC), + UINT64_C(0x7C25B4B6D8D797D0), UINT64_C(0x71D464EA4AAE7E84), + UINT64_C(0x66C7150FFC244579), UINT64_C(0x6B36C5536E5DAC2D), + UINT64_C(0x2A546C7D54A3C801), UINT64_C(0x27A5BC21C6DA2155), + UINT64_C(0x30B6CDC470501AA8), UINT64_C(0x3D471D98E229F3FC), + UINT64_C(0x9B8E21A136EAB5C0), UINT64_C(0x967FF1FDA4935C94), + UINT64_C(0x816C801812196769), UINT64_C(0x8C9D504480608E3D), + UINT64_C(0x03A8D24566E5AE31), UINT64_C(0x0E590219F49C4765), + UINT64_C(0x194A73FC42167C98), UINT64_C(0x14BBA3A0D06F95CC), + UINT64_C(0xB2729F9904ACD3F0), UINT64_C(0xBF834FC596D53AA4), + UINT64_C(0xA8903E20205F0159), UINT64_C(0xA561EE7CB226E80D), + UINT64_C(0xE403475288D88C21), UINT64_C(0xE9F2970E1AA16575), + UINT64_C(0xFEE1E6EBAC2B5E88), UINT64_C(0xF31036B73E52B7DC), + UINT64_C(0x55D90A8EEA91F1E0), UINT64_C(0x5828DAD278E818B4), + UINT64_C(0x4F3BAB37CE622349), UINT64_C(0x42CA7B6B5C1BCA1D), + UINT64_C(0x5150AF3402696251), UINT64_C(0x5CA17F6890108B05), + UINT64_C(0x4BB20E8D269AB0F8), UINT64_C(0x4643DED1B4E359AC), + UINT64_C(0xE08AE2E860201F90), UINT64_C(0xED7B32B4F259F6C4), + UINT64_C(0xFA68435144D3CD39), UINT64_C(0xF799930DD6AA246D), + UINT64_C(0xB6FB3A23EC544041), UINT64_C(0xBB0AEA7F7E2DA915), + UINT64_C(0xAC199B9AC8A792E8), UINT64_C(0xA1E84BC65ADE7BBC), + UINT64_C(0x072177FF8E1D3D80), UINT64_C(0x0AD0A7A31C64D4D4), + UINT64_C(0x1DC3D646AAEEEF29), UINT64_C(0x1032061A3897067D), + UINT64_C(0x9F07841BDE122671), UINT64_C(0x92F654474C6BCF25), + UINT64_C(0x85E525A2FAE1F4D8), UINT64_C(0x8814F5FE68981D8C), + UINT64_C(0x2EDDC9C7BC5B5BB0), UINT64_C(0x232C199B2E22B2E4), + UINT64_C(0x343F687E98A88919), UINT64_C(0x39CEB8220AD1604D), + UINT64_C(0x78AC110C302F0461), UINT64_C(0x755DC150A256ED35), + UINT64_C(0x624EB0B514DCD6C8), UINT64_C(0x6FBF60E986A53F9C), + UINT64_C(0xC9765CD0526679A0), UINT64_C(0xC4878C8CC01F90F4), + UINT64_C(0xD394FD697695AB09), UINT64_C(0xDE652D35E4EC425D) + }, { + UINT64_C(0x0000000000000000), UINT64_C(0xCB6D6A914AE10B3F), + UINT64_C(0x96DBD42295C2177E), UINT64_C(0x5DB6BEB3DF231C41), + UINT64_C(0x2CB7A9452A852FFC), UINT64_C(0xE7DAC3D4606424C3), + UINT64_C(0xBA6C7D67BF473882), UINT64_C(0x710117F6F5A633BD), + UINT64_C(0xDD705D247FA5876A), UINT64_C(0x161D37B535448C55), + UINT64_C(0x4BAB8906EA679014), UINT64_C(0x80C6E397A0869B2B), + UINT64_C(0xF1C7F4615520A896), UINT64_C(0x3AAA9EF01FC1A3A9), + UINT64_C(0x671C2043C0E2BFE8), UINT64_C(0xAC714AD28A03B4D7), + UINT64_C(0xBAE1BA48FE4A0FD5), UINT64_C(0x718CD0D9B4AB04EA), + UINT64_C(0x2C3A6E6A6B8818AB), UINT64_C(0xE75704FB21691394), + UINT64_C(0x9656130DD4CF2029), UINT64_C(0x5D3B799C9E2E2B16), + UINT64_C(0x008DC72F410D3757), UINT64_C(0xCBE0ADBE0BEC3C68), + UINT64_C(0x6791E76C81EF88BF), UINT64_C(0xACFC8DFDCB0E8380), + UINT64_C(0xF14A334E142D9FC1), UINT64_C(0x3A2759DF5ECC94FE), + UINT64_C(0x4B264E29AB6AA743), UINT64_C(0x804B24B8E18BAC7C), + UINT64_C(0xDDFD9A0B3EA8B03D), UINT64_C(0x1690F09A7449BB02), + UINT64_C(0xF1DD7B3ED73AC638), UINT64_C(0x3AB011AF9DDBCD07), + UINT64_C(0x6706AF1C42F8D146), UINT64_C(0xAC6BC58D0819DA79), + UINT64_C(0xDD6AD27BFDBFE9C4), UINT64_C(0x1607B8EAB75EE2FB), + UINT64_C(0x4BB10659687DFEBA), UINT64_C(0x80DC6CC8229CF585), + UINT64_C(0x2CAD261AA89F4152), UINT64_C(0xE7C04C8BE27E4A6D), + UINT64_C(0xBA76F2383D5D562C), UINT64_C(0x711B98A977BC5D13), + UINT64_C(0x001A8F5F821A6EAE), UINT64_C(0xCB77E5CEC8FB6591), + UINT64_C(0x96C15B7D17D879D0), UINT64_C(0x5DAC31EC5D3972EF), + UINT64_C(0x4B3CC1762970C9ED), UINT64_C(0x8051ABE76391C2D2), + UINT64_C(0xDDE71554BCB2DE93), UINT64_C(0x168A7FC5F653D5AC), + UINT64_C(0x678B683303F5E611), UINT64_C(0xACE602A24914ED2E), + UINT64_C(0xF150BC119637F16F), UINT64_C(0x3A3DD680DCD6FA50), + UINT64_C(0x964C9C5256D54E87), UINT64_C(0x5D21F6C31C3445B8), + UINT64_C(0x00974870C31759F9), UINT64_C(0xCBFA22E189F652C6), + UINT64_C(0xBAFB35177C50617B), UINT64_C(0x71965F8636B16A44), + UINT64_C(0x2C20E135E9927605), UINT64_C(0xE74D8BA4A3737D3A), + UINT64_C(0xE2BBF77CAE758C71), UINT64_C(0x29D69DEDE494874E), + UINT64_C(0x7460235E3BB79B0F), UINT64_C(0xBF0D49CF71569030), + UINT64_C(0xCE0C5E3984F0A38D), UINT64_C(0x056134A8CE11A8B2), + UINT64_C(0x58D78A1B1132B4F3), UINT64_C(0x93BAE08A5BD3BFCC), + UINT64_C(0x3FCBAA58D1D00B1B), UINT64_C(0xF4A6C0C99B310024), + UINT64_C(0xA9107E7A44121C65), UINT64_C(0x627D14EB0EF3175A), + UINT64_C(0x137C031DFB5524E7), UINT64_C(0xD811698CB1B42FD8), + UINT64_C(0x85A7D73F6E973399), UINT64_C(0x4ECABDAE247638A6), + UINT64_C(0x585A4D34503F83A4), UINT64_C(0x933727A51ADE889B), + UINT64_C(0xCE819916C5FD94DA), UINT64_C(0x05ECF3878F1C9FE5), + UINT64_C(0x74EDE4717ABAAC58), UINT64_C(0xBF808EE0305BA767), + UINT64_C(0xE2363053EF78BB26), UINT64_C(0x295B5AC2A599B019), + UINT64_C(0x852A10102F9A04CE), UINT64_C(0x4E477A81657B0FF1), + UINT64_C(0x13F1C432BA5813B0), UINT64_C(0xD89CAEA3F0B9188F), + UINT64_C(0xA99DB955051F2B32), UINT64_C(0x62F0D3C44FFE200D), + UINT64_C(0x3F466D7790DD3C4C), UINT64_C(0xF42B07E6DA3C3773), + UINT64_C(0x13668C42794F4A49), UINT64_C(0xD80BE6D333AE4176), + UINT64_C(0x85BD5860EC8D5D37), UINT64_C(0x4ED032F1A66C5608), + UINT64_C(0x3FD1250753CA65B5), UINT64_C(0xF4BC4F96192B6E8A), + UINT64_C(0xA90AF125C60872CB), UINT64_C(0x62679BB48CE979F4), + UINT64_C(0xCE16D16606EACD23), UINT64_C(0x057BBBF74C0BC61C), + UINT64_C(0x58CD05449328DA5D), UINT64_C(0x93A06FD5D9C9D162), + UINT64_C(0xE2A178232C6FE2DF), UINT64_C(0x29CC12B2668EE9E0), + UINT64_C(0x747AAC01B9ADF5A1), UINT64_C(0xBF17C690F34CFE9E), + UINT64_C(0xA987360A8705459C), UINT64_C(0x62EA5C9BCDE44EA3), + UINT64_C(0x3F5CE22812C752E2), UINT64_C(0xF43188B9582659DD), + UINT64_C(0x85309F4FAD806A60), UINT64_C(0x4E5DF5DEE761615F), + UINT64_C(0x13EB4B6D38427D1E), UINT64_C(0xD88621FC72A37621), + UINT64_C(0x74F76B2EF8A0C2F6), UINT64_C(0xBF9A01BFB241C9C9), + UINT64_C(0xE22CBF0C6D62D588), UINT64_C(0x2941D59D2783DEB7), + UINT64_C(0x5840C26BD225ED0A), UINT64_C(0x932DA8FA98C4E635), + UINT64_C(0xCE9B164947E7FA74), UINT64_C(0x05F67CD80D06F14B), + UINT64_C(0xC477EFF95CEB18E3), UINT64_C(0x0F1A8568160A13DC), + UINT64_C(0x52AC3BDBC9290F9D), UINT64_C(0x99C1514A83C804A2), + UINT64_C(0xE8C046BC766E371F), UINT64_C(0x23AD2C2D3C8F3C20), + UINT64_C(0x7E1B929EE3AC2061), UINT64_C(0xB576F80FA94D2B5E), + UINT64_C(0x1907B2DD234E9F89), UINT64_C(0xD26AD84C69AF94B6), + UINT64_C(0x8FDC66FFB68C88F7), UINT64_C(0x44B10C6EFC6D83C8), + UINT64_C(0x35B01B9809CBB075), UINT64_C(0xFEDD7109432ABB4A), + UINT64_C(0xA36BCFBA9C09A70B), UINT64_C(0x6806A52BD6E8AC34), + UINT64_C(0x7E9655B1A2A11736), UINT64_C(0xB5FB3F20E8401C09), + UINT64_C(0xE84D819337630048), UINT64_C(0x2320EB027D820B77), + UINT64_C(0x5221FCF4882438CA), UINT64_C(0x994C9665C2C533F5), + UINT64_C(0xC4FA28D61DE62FB4), UINT64_C(0x0F9742475707248B), + UINT64_C(0xA3E60895DD04905C), UINT64_C(0x688B620497E59B63), + UINT64_C(0x353DDCB748C68722), UINT64_C(0xFE50B62602278C1D), + UINT64_C(0x8F51A1D0F781BFA0), UINT64_C(0x443CCB41BD60B49F), + UINT64_C(0x198A75F26243A8DE), UINT64_C(0xD2E71F6328A2A3E1), + UINT64_C(0x35AA94C78BD1DEDB), UINT64_C(0xFEC7FE56C130D5E4), + UINT64_C(0xA37140E51E13C9A5), UINT64_C(0x681C2A7454F2C29A), + UINT64_C(0x191D3D82A154F127), UINT64_C(0xD2705713EBB5FA18), + UINT64_C(0x8FC6E9A03496E659), UINT64_C(0x44AB83317E77ED66), + UINT64_C(0xE8DAC9E3F47459B1), UINT64_C(0x23B7A372BE95528E), + UINT64_C(0x7E011DC161B64ECF), UINT64_C(0xB56C77502B5745F0), + UINT64_C(0xC46D60A6DEF1764D), UINT64_C(0x0F000A3794107D72), + UINT64_C(0x52B6B4844B336133), UINT64_C(0x99DBDE1501D26A0C), + UINT64_C(0x8F4B2E8F759BD10E), UINT64_C(0x4426441E3F7ADA31), + UINT64_C(0x1990FAADE059C670), UINT64_C(0xD2FD903CAAB8CD4F), + UINT64_C(0xA3FC87CA5F1EFEF2), UINT64_C(0x6891ED5B15FFF5CD), + UINT64_C(0x352753E8CADCE98C), UINT64_C(0xFE4A3979803DE2B3), + UINT64_C(0x523B73AB0A3E5664), UINT64_C(0x9956193A40DF5D5B), + UINT64_C(0xC4E0A7899FFC411A), UINT64_C(0x0F8DCD18D51D4A25), + UINT64_C(0x7E8CDAEE20BB7998), UINT64_C(0xB5E1B07F6A5A72A7), + UINT64_C(0xE8570ECCB5796EE6), UINT64_C(0x233A645DFF9865D9), + UINT64_C(0x26CC1885F29E9492), UINT64_C(0xEDA17214B87F9FAD), + UINT64_C(0xB017CCA7675C83EC), UINT64_C(0x7B7AA6362DBD88D3), + UINT64_C(0x0A7BB1C0D81BBB6E), UINT64_C(0xC116DB5192FAB051), + UINT64_C(0x9CA065E24DD9AC10), UINT64_C(0x57CD0F730738A72F), + UINT64_C(0xFBBC45A18D3B13F8), UINT64_C(0x30D12F30C7DA18C7), + UINT64_C(0x6D67918318F90486), UINT64_C(0xA60AFB1252180FB9), + UINT64_C(0xD70BECE4A7BE3C04), UINT64_C(0x1C668675ED5F373B), + UINT64_C(0x41D038C6327C2B7A), UINT64_C(0x8ABD5257789D2045), + UINT64_C(0x9C2DA2CD0CD49B47), UINT64_C(0x5740C85C46359078), + UINT64_C(0x0AF676EF99168C39), UINT64_C(0xC19B1C7ED3F78706), + UINT64_C(0xB09A0B882651B4BB), UINT64_C(0x7BF761196CB0BF84), + UINT64_C(0x2641DFAAB393A3C5), UINT64_C(0xED2CB53BF972A8FA), + UINT64_C(0x415DFFE973711C2D), UINT64_C(0x8A30957839901712), + UINT64_C(0xD7862BCBE6B30B53), UINT64_C(0x1CEB415AAC52006C), + UINT64_C(0x6DEA56AC59F433D1), UINT64_C(0xA6873C3D131538EE), + UINT64_C(0xFB31828ECC3624AF), UINT64_C(0x305CE81F86D72F90), + UINT64_C(0xD71163BB25A452AA), UINT64_C(0x1C7C092A6F455995), + UINT64_C(0x41CAB799B06645D4), UINT64_C(0x8AA7DD08FA874EEB), + UINT64_C(0xFBA6CAFE0F217D56), UINT64_C(0x30CBA06F45C07669), + UINT64_C(0x6D7D1EDC9AE36A28), UINT64_C(0xA610744DD0026117), + UINT64_C(0x0A613E9F5A01D5C0), UINT64_C(0xC10C540E10E0DEFF), + UINT64_C(0x9CBAEABDCFC3C2BE), UINT64_C(0x57D7802C8522C981), + UINT64_C(0x26D697DA7084FA3C), UINT64_C(0xEDBBFD4B3A65F103), + UINT64_C(0xB00D43F8E546ED42), UINT64_C(0x7B602969AFA7E67D), + UINT64_C(0x6DF0D9F3DBEE5D7F), UINT64_C(0xA69DB362910F5640), + UINT64_C(0xFB2B0DD14E2C4A01), UINT64_C(0x3046674004CD413E), + UINT64_C(0x414770B6F16B7283), UINT64_C(0x8A2A1A27BB8A79BC), + UINT64_C(0xD79CA49464A965FD), UINT64_C(0x1CF1CE052E486EC2), + UINT64_C(0xB08084D7A44BDA15), UINT64_C(0x7BEDEE46EEAAD12A), + UINT64_C(0x265B50F53189CD6B), UINT64_C(0xED363A647B68C654), + UINT64_C(0x9C372D928ECEF5E9), UINT64_C(0x575A4703C42FFED6), + UINT64_C(0x0AECF9B01B0CE297), UINT64_C(0xC181932151EDE9A8) + }, { + UINT64_C(0x0000000000000000), UINT64_C(0xDCA12C225E8AEE1D), + UINT64_C(0xB8435944BC14DD3B), UINT64_C(0x64E27566E29E3326), + UINT64_C(0x7087B2887829BA77), UINT64_C(0xAC269EAA26A3546A), + UINT64_C(0xC8C4EBCCC43D674C), UINT64_C(0x1465C7EE9AB78951), + UINT64_C(0xE00E6511F15274EF), UINT64_C(0x3CAF4933AFD89AF2), + UINT64_C(0x584D3C554D46A9D4), UINT64_C(0x84EC107713CC47C9), + UINT64_C(0x9089D799897BCE98), UINT64_C(0x4C28FBBBD7F12085), + UINT64_C(0x28CA8EDD356F13A3), UINT64_C(0xF46BA2FF6BE5FDBE), + UINT64_C(0x4503C48DC90A304C), UINT64_C(0x99A2E8AF9780DE51), + UINT64_C(0xFD409DC9751EED77), UINT64_C(0x21E1B1EB2B94036A), + UINT64_C(0x35847605B1238A3B), UINT64_C(0xE9255A27EFA96426), + UINT64_C(0x8DC72F410D375700), UINT64_C(0x5166036353BDB91D), + UINT64_C(0xA50DA19C385844A3), UINT64_C(0x79AC8DBE66D2AABE), + UINT64_C(0x1D4EF8D8844C9998), UINT64_C(0xC1EFD4FADAC67785), + UINT64_C(0xD58A13144071FED4), UINT64_C(0x092B3F361EFB10C9), + UINT64_C(0x6DC94A50FC6523EF), UINT64_C(0xB1686672A2EFCDF2), + UINT64_C(0x8A06881B93156098), UINT64_C(0x56A7A439CD9F8E85), + UINT64_C(0x3245D15F2F01BDA3), UINT64_C(0xEEE4FD7D718B53BE), + UINT64_C(0xFA813A93EB3CDAEF), UINT64_C(0x262016B1B5B634F2), + UINT64_C(0x42C263D7572807D4), UINT64_C(0x9E634FF509A2E9C9), + UINT64_C(0x6A08ED0A62471477), UINT64_C(0xB6A9C1283CCDFA6A), + UINT64_C(0xD24BB44EDE53C94C), UINT64_C(0x0EEA986C80D92751), + UINT64_C(0x1A8F5F821A6EAE00), UINT64_C(0xC62E73A044E4401D), + UINT64_C(0xA2CC06C6A67A733B), UINT64_C(0x7E6D2AE4F8F09D26), + UINT64_C(0xCF054C965A1F50D4), UINT64_C(0x13A460B40495BEC9), + UINT64_C(0x774615D2E60B8DEF), UINT64_C(0xABE739F0B88163F2), + UINT64_C(0xBF82FE1E2236EAA3), UINT64_C(0x6323D23C7CBC04BE), + UINT64_C(0x07C1A75A9E223798), UINT64_C(0xDB608B78C0A8D985), + UINT64_C(0x2F0B2987AB4D243B), UINT64_C(0xF3AA05A5F5C7CA26), + UINT64_C(0x974870C31759F900), UINT64_C(0x4BE95CE149D3171D), + UINT64_C(0x5F8C9B0FD3649E4C), UINT64_C(0x832DB72D8DEE7051), + UINT64_C(0xE7CFC24B6F704377), UINT64_C(0x3B6EEE6931FAAD6A), + UINT64_C(0x91131E980D8418A2), UINT64_C(0x4DB232BA530EF6BF), + UINT64_C(0x295047DCB190C599), UINT64_C(0xF5F16BFEEF1A2B84), + UINT64_C(0xE194AC1075ADA2D5), UINT64_C(0x3D3580322B274CC8), + UINT64_C(0x59D7F554C9B97FEE), UINT64_C(0x8576D976973391F3), + UINT64_C(0x711D7B89FCD66C4D), UINT64_C(0xADBC57ABA25C8250), + UINT64_C(0xC95E22CD40C2B176), UINT64_C(0x15FF0EEF1E485F6B), + UINT64_C(0x019AC90184FFD63A), UINT64_C(0xDD3BE523DA753827), + UINT64_C(0xB9D9904538EB0B01), UINT64_C(0x6578BC676661E51C), + UINT64_C(0xD410DA15C48E28EE), UINT64_C(0x08B1F6379A04C6F3), + UINT64_C(0x6C538351789AF5D5), UINT64_C(0xB0F2AF7326101BC8), + UINT64_C(0xA497689DBCA79299), UINT64_C(0x783644BFE22D7C84), + UINT64_C(0x1CD431D900B34FA2), UINT64_C(0xC0751DFB5E39A1BF), + UINT64_C(0x341EBF0435DC5C01), UINT64_C(0xE8BF93266B56B21C), + UINT64_C(0x8C5DE64089C8813A), UINT64_C(0x50FCCA62D7426F27), + UINT64_C(0x44990D8C4DF5E676), UINT64_C(0x983821AE137F086B), + UINT64_C(0xFCDA54C8F1E13B4D), UINT64_C(0x207B78EAAF6BD550), + UINT64_C(0x1B1596839E91783A), UINT64_C(0xC7B4BAA1C01B9627), + UINT64_C(0xA356CFC72285A501), UINT64_C(0x7FF7E3E57C0F4B1C), + UINT64_C(0x6B92240BE6B8C24D), UINT64_C(0xB7330829B8322C50), + UINT64_C(0xD3D17D4F5AAC1F76), UINT64_C(0x0F70516D0426F16B), + UINT64_C(0xFB1BF3926FC30CD5), UINT64_C(0x27BADFB03149E2C8), + UINT64_C(0x4358AAD6D3D7D1EE), UINT64_C(0x9FF986F48D5D3FF3), + UINT64_C(0x8B9C411A17EAB6A2), UINT64_C(0x573D6D38496058BF), + UINT64_C(0x33DF185EABFE6B99), UINT64_C(0xEF7E347CF5748584), + UINT64_C(0x5E16520E579B4876), UINT64_C(0x82B77E2C0911A66B), + UINT64_C(0xE6550B4AEB8F954D), UINT64_C(0x3AF42768B5057B50), + UINT64_C(0x2E91E0862FB2F201), UINT64_C(0xF230CCA471381C1C), + UINT64_C(0x96D2B9C293A62F3A), UINT64_C(0x4A7395E0CD2CC127), + UINT64_C(0xBE18371FA6C93C99), UINT64_C(0x62B91B3DF843D284), + UINT64_C(0x065B6E5B1ADDE1A2), UINT64_C(0xDAFA427944570FBF), + UINT64_C(0xCE9F8597DEE086EE), UINT64_C(0x123EA9B5806A68F3), + UINT64_C(0x76DCDCD362F45BD5), UINT64_C(0xAA7DF0F13C7EB5C8), + UINT64_C(0xA739329F30A7E9D6), UINT64_C(0x7B981EBD6E2D07CB), + UINT64_C(0x1F7A6BDB8CB334ED), UINT64_C(0xC3DB47F9D239DAF0), + UINT64_C(0xD7BE8017488E53A1), UINT64_C(0x0B1FAC351604BDBC), + UINT64_C(0x6FFDD953F49A8E9A), UINT64_C(0xB35CF571AA106087), + UINT64_C(0x4737578EC1F59D39), UINT64_C(0x9B967BAC9F7F7324), + UINT64_C(0xFF740ECA7DE14002), UINT64_C(0x23D522E8236BAE1F), + UINT64_C(0x37B0E506B9DC274E), UINT64_C(0xEB11C924E756C953), + UINT64_C(0x8FF3BC4205C8FA75), UINT64_C(0x535290605B421468), + UINT64_C(0xE23AF612F9ADD99A), UINT64_C(0x3E9BDA30A7273787), + UINT64_C(0x5A79AF5645B904A1), UINT64_C(0x86D883741B33EABC), + UINT64_C(0x92BD449A818463ED), UINT64_C(0x4E1C68B8DF0E8DF0), + UINT64_C(0x2AFE1DDE3D90BED6), UINT64_C(0xF65F31FC631A50CB), + UINT64_C(0x0234930308FFAD75), UINT64_C(0xDE95BF2156754368), + UINT64_C(0xBA77CA47B4EB704E), UINT64_C(0x66D6E665EA619E53), + UINT64_C(0x72B3218B70D61702), UINT64_C(0xAE120DA92E5CF91F), + UINT64_C(0xCAF078CFCCC2CA39), UINT64_C(0x165154ED92482424), + UINT64_C(0x2D3FBA84A3B2894E), UINT64_C(0xF19E96A6FD386753), + UINT64_C(0x957CE3C01FA65475), UINT64_C(0x49DDCFE2412CBA68), + UINT64_C(0x5DB8080CDB9B3339), UINT64_C(0x8119242E8511DD24), + UINT64_C(0xE5FB5148678FEE02), UINT64_C(0x395A7D6A3905001F), + UINT64_C(0xCD31DF9552E0FDA1), UINT64_C(0x1190F3B70C6A13BC), + UINT64_C(0x757286D1EEF4209A), UINT64_C(0xA9D3AAF3B07ECE87), + UINT64_C(0xBDB66D1D2AC947D6), UINT64_C(0x6117413F7443A9CB), + UINT64_C(0x05F5345996DD9AED), UINT64_C(0xD954187BC85774F0), + UINT64_C(0x683C7E096AB8B902), UINT64_C(0xB49D522B3432571F), + UINT64_C(0xD07F274DD6AC6439), UINT64_C(0x0CDE0B6F88268A24), + UINT64_C(0x18BBCC8112910375), UINT64_C(0xC41AE0A34C1BED68), + UINT64_C(0xA0F895C5AE85DE4E), UINT64_C(0x7C59B9E7F00F3053), + UINT64_C(0x88321B189BEACDED), UINT64_C(0x5493373AC56023F0), + UINT64_C(0x3071425C27FE10D6), UINT64_C(0xECD06E7E7974FECB), + UINT64_C(0xF8B5A990E3C3779A), UINT64_C(0x241485B2BD499987), + UINT64_C(0x40F6F0D45FD7AAA1), UINT64_C(0x9C57DCF6015D44BC), + UINT64_C(0x362A2C073D23F174), UINT64_C(0xEA8B002563A91F69), + UINT64_C(0x8E69754381372C4F), UINT64_C(0x52C85961DFBDC252), + UINT64_C(0x46AD9E8F450A4B03), UINT64_C(0x9A0CB2AD1B80A51E), + UINT64_C(0xFEEEC7CBF91E9638), UINT64_C(0x224FEBE9A7947825), + UINT64_C(0xD6244916CC71859B), UINT64_C(0x0A85653492FB6B86), + UINT64_C(0x6E671052706558A0), UINT64_C(0xB2C63C702EEFB6BD), + UINT64_C(0xA6A3FB9EB4583FEC), UINT64_C(0x7A02D7BCEAD2D1F1), + UINT64_C(0x1EE0A2DA084CE2D7), UINT64_C(0xC2418EF856C60CCA), + UINT64_C(0x7329E88AF429C138), UINT64_C(0xAF88C4A8AAA32F25), + UINT64_C(0xCB6AB1CE483D1C03), UINT64_C(0x17CB9DEC16B7F21E), + UINT64_C(0x03AE5A028C007B4F), UINT64_C(0xDF0F7620D28A9552), + UINT64_C(0xBBED03463014A674), UINT64_C(0x674C2F646E9E4869), + UINT64_C(0x93278D9B057BB5D7), UINT64_C(0x4F86A1B95BF15BCA), + UINT64_C(0x2B64D4DFB96F68EC), UINT64_C(0xF7C5F8FDE7E586F1), + UINT64_C(0xE3A03F137D520FA0), UINT64_C(0x3F01133123D8E1BD), + UINT64_C(0x5BE36657C146D29B), UINT64_C(0x87424A759FCC3C86), + UINT64_C(0xBC2CA41CAE3691EC), UINT64_C(0x608D883EF0BC7FF1), + UINT64_C(0x046FFD5812224CD7), UINT64_C(0xD8CED17A4CA8A2CA), + UINT64_C(0xCCAB1694D61F2B9B), UINT64_C(0x100A3AB68895C586), + UINT64_C(0x74E84FD06A0BF6A0), UINT64_C(0xA84963F2348118BD), + UINT64_C(0x5C22C10D5F64E503), UINT64_C(0x8083ED2F01EE0B1E), + UINT64_C(0xE4619849E3703838), UINT64_C(0x38C0B46BBDFAD625), + UINT64_C(0x2CA57385274D5F74), UINT64_C(0xF0045FA779C7B169), + UINT64_C(0x94E62AC19B59824F), UINT64_C(0x484706E3C5D36C52), + UINT64_C(0xF92F6091673CA1A0), UINT64_C(0x258E4CB339B64FBD), + UINT64_C(0x416C39D5DB287C9B), UINT64_C(0x9DCD15F785A29286), + UINT64_C(0x89A8D2191F151BD7), UINT64_C(0x5509FE3B419FF5CA), + UINT64_C(0x31EB8B5DA301C6EC), UINT64_C(0xED4AA77FFD8B28F1), + UINT64_C(0x19210580966ED54F), UINT64_C(0xC58029A2C8E43B52), + UINT64_C(0xA1625CC42A7A0874), UINT64_C(0x7DC370E674F0E669), + UINT64_C(0x69A6B708EE476F38), UINT64_C(0xB5079B2AB0CD8125), + UINT64_C(0xD1E5EE4C5253B203), UINT64_C(0x0D44C26E0CD95C1E) + } +}; diff --git a/src/liblzma/check/crc64_table_le.h b/src/liblzma/check/crc64_table_le.h new file mode 100644 index 0000000000..e40a8c8210 --- /dev/null +++ b/src/liblzma/check/crc64_table_le.h @@ -0,0 +1,523 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by crc64_tablegen.c. + +const uint64_t lzma_crc64_table[4][256] = { + { + UINT64_C(0x0000000000000000), UINT64_C(0xB32E4CBE03A75F6F), + UINT64_C(0xF4843657A840A05B), UINT64_C(0x47AA7AE9ABE7FF34), + UINT64_C(0x7BD0C384FF8F5E33), UINT64_C(0xC8FE8F3AFC28015C), + UINT64_C(0x8F54F5D357CFFE68), UINT64_C(0x3C7AB96D5468A107), + UINT64_C(0xF7A18709FF1EBC66), UINT64_C(0x448FCBB7FCB9E309), + UINT64_C(0x0325B15E575E1C3D), UINT64_C(0xB00BFDE054F94352), + UINT64_C(0x8C71448D0091E255), UINT64_C(0x3F5F08330336BD3A), + UINT64_C(0x78F572DAA8D1420E), UINT64_C(0xCBDB3E64AB761D61), + UINT64_C(0x7D9BA13851336649), UINT64_C(0xCEB5ED8652943926), + UINT64_C(0x891F976FF973C612), UINT64_C(0x3A31DBD1FAD4997D), + UINT64_C(0x064B62BCAEBC387A), UINT64_C(0xB5652E02AD1B6715), + UINT64_C(0xF2CF54EB06FC9821), UINT64_C(0x41E11855055BC74E), + UINT64_C(0x8A3A2631AE2DDA2F), UINT64_C(0x39146A8FAD8A8540), + UINT64_C(0x7EBE1066066D7A74), UINT64_C(0xCD905CD805CA251B), + UINT64_C(0xF1EAE5B551A2841C), UINT64_C(0x42C4A90B5205DB73), + UINT64_C(0x056ED3E2F9E22447), UINT64_C(0xB6409F5CFA457B28), + UINT64_C(0xFB374270A266CC92), UINT64_C(0x48190ECEA1C193FD), + UINT64_C(0x0FB374270A266CC9), UINT64_C(0xBC9D3899098133A6), + UINT64_C(0x80E781F45DE992A1), UINT64_C(0x33C9CD4A5E4ECDCE), + UINT64_C(0x7463B7A3F5A932FA), UINT64_C(0xC74DFB1DF60E6D95), + UINT64_C(0x0C96C5795D7870F4), UINT64_C(0xBFB889C75EDF2F9B), + UINT64_C(0xF812F32EF538D0AF), UINT64_C(0x4B3CBF90F69F8FC0), + UINT64_C(0x774606FDA2F72EC7), UINT64_C(0xC4684A43A15071A8), + UINT64_C(0x83C230AA0AB78E9C), UINT64_C(0x30EC7C140910D1F3), + UINT64_C(0x86ACE348F355AADB), UINT64_C(0x3582AFF6F0F2F5B4), + UINT64_C(0x7228D51F5B150A80), UINT64_C(0xC10699A158B255EF), + UINT64_C(0xFD7C20CC0CDAF4E8), UINT64_C(0x4E526C720F7DAB87), + UINT64_C(0x09F8169BA49A54B3), UINT64_C(0xBAD65A25A73D0BDC), + UINT64_C(0x710D64410C4B16BD), UINT64_C(0xC22328FF0FEC49D2), + UINT64_C(0x85895216A40BB6E6), UINT64_C(0x36A71EA8A7ACE989), + UINT64_C(0x0ADDA7C5F3C4488E), UINT64_C(0xB9F3EB7BF06317E1), + UINT64_C(0xFE5991925B84E8D5), UINT64_C(0x4D77DD2C5823B7BA), + UINT64_C(0x64B62BCAEBC387A1), UINT64_C(0xD7986774E864D8CE), + UINT64_C(0x90321D9D438327FA), UINT64_C(0x231C512340247895), + UINT64_C(0x1F66E84E144CD992), UINT64_C(0xAC48A4F017EB86FD), + UINT64_C(0xEBE2DE19BC0C79C9), UINT64_C(0x58CC92A7BFAB26A6), + UINT64_C(0x9317ACC314DD3BC7), UINT64_C(0x2039E07D177A64A8), + UINT64_C(0x67939A94BC9D9B9C), UINT64_C(0xD4BDD62ABF3AC4F3), + UINT64_C(0xE8C76F47EB5265F4), UINT64_C(0x5BE923F9E8F53A9B), + UINT64_C(0x1C4359104312C5AF), UINT64_C(0xAF6D15AE40B59AC0), + UINT64_C(0x192D8AF2BAF0E1E8), UINT64_C(0xAA03C64CB957BE87), + UINT64_C(0xEDA9BCA512B041B3), UINT64_C(0x5E87F01B11171EDC), + UINT64_C(0x62FD4976457FBFDB), UINT64_C(0xD1D305C846D8E0B4), + UINT64_C(0x96797F21ED3F1F80), UINT64_C(0x2557339FEE9840EF), + UINT64_C(0xEE8C0DFB45EE5D8E), UINT64_C(0x5DA24145464902E1), + UINT64_C(0x1A083BACEDAEFDD5), UINT64_C(0xA9267712EE09A2BA), + UINT64_C(0x955CCE7FBA6103BD), UINT64_C(0x267282C1B9C65CD2), + UINT64_C(0x61D8F8281221A3E6), UINT64_C(0xD2F6B4961186FC89), + UINT64_C(0x9F8169BA49A54B33), UINT64_C(0x2CAF25044A02145C), + UINT64_C(0x6B055FEDE1E5EB68), UINT64_C(0xD82B1353E242B407), + UINT64_C(0xE451AA3EB62A1500), UINT64_C(0x577FE680B58D4A6F), + UINT64_C(0x10D59C691E6AB55B), UINT64_C(0xA3FBD0D71DCDEA34), + UINT64_C(0x6820EEB3B6BBF755), UINT64_C(0xDB0EA20DB51CA83A), + UINT64_C(0x9CA4D8E41EFB570E), UINT64_C(0x2F8A945A1D5C0861), + UINT64_C(0x13F02D374934A966), UINT64_C(0xA0DE61894A93F609), + UINT64_C(0xE7741B60E174093D), UINT64_C(0x545A57DEE2D35652), + UINT64_C(0xE21AC88218962D7A), UINT64_C(0x5134843C1B317215), + UINT64_C(0x169EFED5B0D68D21), UINT64_C(0xA5B0B26BB371D24E), + UINT64_C(0x99CA0B06E7197349), UINT64_C(0x2AE447B8E4BE2C26), + UINT64_C(0x6D4E3D514F59D312), UINT64_C(0xDE6071EF4CFE8C7D), + UINT64_C(0x15BB4F8BE788911C), UINT64_C(0xA6950335E42FCE73), + UINT64_C(0xE13F79DC4FC83147), UINT64_C(0x521135624C6F6E28), + UINT64_C(0x6E6B8C0F1807CF2F), UINT64_C(0xDD45C0B11BA09040), + UINT64_C(0x9AEFBA58B0476F74), UINT64_C(0x29C1F6E6B3E0301B), + UINT64_C(0xC96C5795D7870F42), UINT64_C(0x7A421B2BD420502D), + UINT64_C(0x3DE861C27FC7AF19), UINT64_C(0x8EC62D7C7C60F076), + UINT64_C(0xB2BC941128085171), UINT64_C(0x0192D8AF2BAF0E1E), + UINT64_C(0x4638A2468048F12A), UINT64_C(0xF516EEF883EFAE45), + UINT64_C(0x3ECDD09C2899B324), UINT64_C(0x8DE39C222B3EEC4B), + UINT64_C(0xCA49E6CB80D9137F), UINT64_C(0x7967AA75837E4C10), + UINT64_C(0x451D1318D716ED17), UINT64_C(0xF6335FA6D4B1B278), + UINT64_C(0xB199254F7F564D4C), UINT64_C(0x02B769F17CF11223), + UINT64_C(0xB4F7F6AD86B4690B), UINT64_C(0x07D9BA1385133664), + UINT64_C(0x4073C0FA2EF4C950), UINT64_C(0xF35D8C442D53963F), + UINT64_C(0xCF273529793B3738), UINT64_C(0x7C0979977A9C6857), + UINT64_C(0x3BA3037ED17B9763), UINT64_C(0x888D4FC0D2DCC80C), + UINT64_C(0x435671A479AAD56D), UINT64_C(0xF0783D1A7A0D8A02), + UINT64_C(0xB7D247F3D1EA7536), UINT64_C(0x04FC0B4DD24D2A59), + UINT64_C(0x3886B22086258B5E), UINT64_C(0x8BA8FE9E8582D431), + UINT64_C(0xCC0284772E652B05), UINT64_C(0x7F2CC8C92DC2746A), + UINT64_C(0x325B15E575E1C3D0), UINT64_C(0x8175595B76469CBF), + UINT64_C(0xC6DF23B2DDA1638B), UINT64_C(0x75F16F0CDE063CE4), + UINT64_C(0x498BD6618A6E9DE3), UINT64_C(0xFAA59ADF89C9C28C), + UINT64_C(0xBD0FE036222E3DB8), UINT64_C(0x0E21AC88218962D7), + UINT64_C(0xC5FA92EC8AFF7FB6), UINT64_C(0x76D4DE52895820D9), + UINT64_C(0x317EA4BB22BFDFED), UINT64_C(0x8250E80521188082), + UINT64_C(0xBE2A516875702185), UINT64_C(0x0D041DD676D77EEA), + UINT64_C(0x4AAE673FDD3081DE), UINT64_C(0xF9802B81DE97DEB1), + UINT64_C(0x4FC0B4DD24D2A599), UINT64_C(0xFCEEF8632775FAF6), + UINT64_C(0xBB44828A8C9205C2), UINT64_C(0x086ACE348F355AAD), + UINT64_C(0x34107759DB5DFBAA), UINT64_C(0x873E3BE7D8FAA4C5), + UINT64_C(0xC094410E731D5BF1), UINT64_C(0x73BA0DB070BA049E), + UINT64_C(0xB86133D4DBCC19FF), UINT64_C(0x0B4F7F6AD86B4690), + UINT64_C(0x4CE50583738CB9A4), UINT64_C(0xFFCB493D702BE6CB), + UINT64_C(0xC3B1F050244347CC), UINT64_C(0x709FBCEE27E418A3), + UINT64_C(0x3735C6078C03E797), UINT64_C(0x841B8AB98FA4B8F8), + UINT64_C(0xADDA7C5F3C4488E3), UINT64_C(0x1EF430E13FE3D78C), + UINT64_C(0x595E4A08940428B8), UINT64_C(0xEA7006B697A377D7), + UINT64_C(0xD60ABFDBC3CBD6D0), UINT64_C(0x6524F365C06C89BF), + UINT64_C(0x228E898C6B8B768B), UINT64_C(0x91A0C532682C29E4), + UINT64_C(0x5A7BFB56C35A3485), UINT64_C(0xE955B7E8C0FD6BEA), + UINT64_C(0xAEFFCD016B1A94DE), UINT64_C(0x1DD181BF68BDCBB1), + UINT64_C(0x21AB38D23CD56AB6), UINT64_C(0x9285746C3F7235D9), + UINT64_C(0xD52F0E859495CAED), UINT64_C(0x6601423B97329582), + UINT64_C(0xD041DD676D77EEAA), UINT64_C(0x636F91D96ED0B1C5), + UINT64_C(0x24C5EB30C5374EF1), UINT64_C(0x97EBA78EC690119E), + UINT64_C(0xAB911EE392F8B099), UINT64_C(0x18BF525D915FEFF6), + UINT64_C(0x5F1528B43AB810C2), UINT64_C(0xEC3B640A391F4FAD), + UINT64_C(0x27E05A6E926952CC), UINT64_C(0x94CE16D091CE0DA3), + UINT64_C(0xD3646C393A29F297), UINT64_C(0x604A2087398EADF8), + UINT64_C(0x5C3099EA6DE60CFF), UINT64_C(0xEF1ED5546E415390), + UINT64_C(0xA8B4AFBDC5A6ACA4), UINT64_C(0x1B9AE303C601F3CB), + UINT64_C(0x56ED3E2F9E224471), UINT64_C(0xE5C372919D851B1E), + UINT64_C(0xA26908783662E42A), UINT64_C(0x114744C635C5BB45), + UINT64_C(0x2D3DFDAB61AD1A42), UINT64_C(0x9E13B115620A452D), + UINT64_C(0xD9B9CBFCC9EDBA19), UINT64_C(0x6A978742CA4AE576), + UINT64_C(0xA14CB926613CF817), UINT64_C(0x1262F598629BA778), + UINT64_C(0x55C88F71C97C584C), UINT64_C(0xE6E6C3CFCADB0723), + UINT64_C(0xDA9C7AA29EB3A624), UINT64_C(0x69B2361C9D14F94B), + UINT64_C(0x2E184CF536F3067F), UINT64_C(0x9D36004B35545910), + UINT64_C(0x2B769F17CF112238), UINT64_C(0x9858D3A9CCB67D57), + UINT64_C(0xDFF2A94067518263), UINT64_C(0x6CDCE5FE64F6DD0C), + UINT64_C(0x50A65C93309E7C0B), UINT64_C(0xE388102D33392364), + UINT64_C(0xA4226AC498DEDC50), UINT64_C(0x170C267A9B79833F), + UINT64_C(0xDCD7181E300F9E5E), UINT64_C(0x6FF954A033A8C131), + UINT64_C(0x28532E49984F3E05), UINT64_C(0x9B7D62F79BE8616A), + UINT64_C(0xA707DB9ACF80C06D), UINT64_C(0x14299724CC279F02), + UINT64_C(0x5383EDCD67C06036), UINT64_C(0xE0ADA17364673F59) + }, { + UINT64_C(0x0000000000000000), UINT64_C(0x54E979925CD0F10D), + UINT64_C(0xA9D2F324B9A1E21A), UINT64_C(0xFD3B8AB6E5711317), + UINT64_C(0xC17D4962DC4DDAB1), UINT64_C(0x959430F0809D2BBC), + UINT64_C(0x68AFBA4665EC38AB), UINT64_C(0x3C46C3D4393CC9A6), + UINT64_C(0x10223DEE1795ABE7), UINT64_C(0x44CB447C4B455AEA), + UINT64_C(0xB9F0CECAAE3449FD), UINT64_C(0xED19B758F2E4B8F0), + UINT64_C(0xD15F748CCBD87156), UINT64_C(0x85B60D1E9708805B), + UINT64_C(0x788D87A87279934C), UINT64_C(0x2C64FE3A2EA96241), + UINT64_C(0x20447BDC2F2B57CE), UINT64_C(0x74AD024E73FBA6C3), + UINT64_C(0x899688F8968AB5D4), UINT64_C(0xDD7FF16ACA5A44D9), + UINT64_C(0xE13932BEF3668D7F), UINT64_C(0xB5D04B2CAFB67C72), + UINT64_C(0x48EBC19A4AC76F65), UINT64_C(0x1C02B80816179E68), + UINT64_C(0x3066463238BEFC29), UINT64_C(0x648F3FA0646E0D24), + UINT64_C(0x99B4B516811F1E33), UINT64_C(0xCD5DCC84DDCFEF3E), + UINT64_C(0xF11B0F50E4F32698), UINT64_C(0xA5F276C2B823D795), + UINT64_C(0x58C9FC745D52C482), UINT64_C(0x0C2085E60182358F), + UINT64_C(0x4088F7B85E56AF9C), UINT64_C(0x14618E2A02865E91), + UINT64_C(0xE95A049CE7F74D86), UINT64_C(0xBDB37D0EBB27BC8B), + UINT64_C(0x81F5BEDA821B752D), UINT64_C(0xD51CC748DECB8420), + UINT64_C(0x28274DFE3BBA9737), UINT64_C(0x7CCE346C676A663A), + UINT64_C(0x50AACA5649C3047B), UINT64_C(0x0443B3C41513F576), + UINT64_C(0xF9783972F062E661), UINT64_C(0xAD9140E0ACB2176C), + UINT64_C(0x91D78334958EDECA), UINT64_C(0xC53EFAA6C95E2FC7), + UINT64_C(0x380570102C2F3CD0), UINT64_C(0x6CEC098270FFCDDD), + UINT64_C(0x60CC8C64717DF852), UINT64_C(0x3425F5F62DAD095F), + UINT64_C(0xC91E7F40C8DC1A48), UINT64_C(0x9DF706D2940CEB45), + UINT64_C(0xA1B1C506AD3022E3), UINT64_C(0xF558BC94F1E0D3EE), + UINT64_C(0x086336221491C0F9), UINT64_C(0x5C8A4FB0484131F4), + UINT64_C(0x70EEB18A66E853B5), UINT64_C(0x2407C8183A38A2B8), + UINT64_C(0xD93C42AEDF49B1AF), UINT64_C(0x8DD53B3C839940A2), + UINT64_C(0xB193F8E8BAA58904), UINT64_C(0xE57A817AE6757809), + UINT64_C(0x18410BCC03046B1E), UINT64_C(0x4CA8725E5FD49A13), + UINT64_C(0x8111EF70BCAD5F38), UINT64_C(0xD5F896E2E07DAE35), + UINT64_C(0x28C31C54050CBD22), UINT64_C(0x7C2A65C659DC4C2F), + UINT64_C(0x406CA61260E08589), UINT64_C(0x1485DF803C307484), + UINT64_C(0xE9BE5536D9416793), UINT64_C(0xBD572CA48591969E), + UINT64_C(0x9133D29EAB38F4DF), UINT64_C(0xC5DAAB0CF7E805D2), + UINT64_C(0x38E121BA129916C5), UINT64_C(0x6C0858284E49E7C8), + UINT64_C(0x504E9BFC77752E6E), UINT64_C(0x04A7E26E2BA5DF63), + UINT64_C(0xF99C68D8CED4CC74), UINT64_C(0xAD75114A92043D79), + UINT64_C(0xA15594AC938608F6), UINT64_C(0xF5BCED3ECF56F9FB), + UINT64_C(0x088767882A27EAEC), UINT64_C(0x5C6E1E1A76F71BE1), + UINT64_C(0x6028DDCE4FCBD247), UINT64_C(0x34C1A45C131B234A), + UINT64_C(0xC9FA2EEAF66A305D), UINT64_C(0x9D135778AABAC150), + UINT64_C(0xB177A9428413A311), UINT64_C(0xE59ED0D0D8C3521C), + UINT64_C(0x18A55A663DB2410B), UINT64_C(0x4C4C23F46162B006), + UINT64_C(0x700AE020585E79A0), UINT64_C(0x24E399B2048E88AD), + UINT64_C(0xD9D81304E1FF9BBA), UINT64_C(0x8D316A96BD2F6AB7), + UINT64_C(0xC19918C8E2FBF0A4), UINT64_C(0x9570615ABE2B01A9), + UINT64_C(0x684BEBEC5B5A12BE), UINT64_C(0x3CA2927E078AE3B3), + UINT64_C(0x00E451AA3EB62A15), UINT64_C(0x540D28386266DB18), + UINT64_C(0xA936A28E8717C80F), UINT64_C(0xFDDFDB1CDBC73902), + UINT64_C(0xD1BB2526F56E5B43), UINT64_C(0x85525CB4A9BEAA4E), + UINT64_C(0x7869D6024CCFB959), UINT64_C(0x2C80AF90101F4854), + UINT64_C(0x10C66C44292381F2), UINT64_C(0x442F15D675F370FF), + UINT64_C(0xB9149F60908263E8), UINT64_C(0xEDFDE6F2CC5292E5), + UINT64_C(0xE1DD6314CDD0A76A), UINT64_C(0xB5341A8691005667), + UINT64_C(0x480F903074714570), UINT64_C(0x1CE6E9A228A1B47D), + UINT64_C(0x20A02A76119D7DDB), UINT64_C(0x744953E44D4D8CD6), + UINT64_C(0x8972D952A83C9FC1), UINT64_C(0xDD9BA0C0F4EC6ECC), + UINT64_C(0xF1FF5EFADA450C8D), UINT64_C(0xA51627688695FD80), + UINT64_C(0x582DADDE63E4EE97), UINT64_C(0x0CC4D44C3F341F9A), + UINT64_C(0x308217980608D63C), UINT64_C(0x646B6E0A5AD82731), + UINT64_C(0x9950E4BCBFA93426), UINT64_C(0xCDB99D2EE379C52B), + UINT64_C(0x90FB71CAD654A0F5), UINT64_C(0xC41208588A8451F8), + UINT64_C(0x392982EE6FF542EF), UINT64_C(0x6DC0FB7C3325B3E2), + UINT64_C(0x518638A80A197A44), UINT64_C(0x056F413A56C98B49), + UINT64_C(0xF854CB8CB3B8985E), UINT64_C(0xACBDB21EEF686953), + UINT64_C(0x80D94C24C1C10B12), UINT64_C(0xD43035B69D11FA1F), + UINT64_C(0x290BBF007860E908), UINT64_C(0x7DE2C69224B01805), + UINT64_C(0x41A405461D8CD1A3), UINT64_C(0x154D7CD4415C20AE), + UINT64_C(0xE876F662A42D33B9), UINT64_C(0xBC9F8FF0F8FDC2B4), + UINT64_C(0xB0BF0A16F97FF73B), UINT64_C(0xE4567384A5AF0636), + UINT64_C(0x196DF93240DE1521), UINT64_C(0x4D8480A01C0EE42C), + UINT64_C(0x71C2437425322D8A), UINT64_C(0x252B3AE679E2DC87), + UINT64_C(0xD810B0509C93CF90), UINT64_C(0x8CF9C9C2C0433E9D), + UINT64_C(0xA09D37F8EEEA5CDC), UINT64_C(0xF4744E6AB23AADD1), + UINT64_C(0x094FC4DC574BBEC6), UINT64_C(0x5DA6BD4E0B9B4FCB), + UINT64_C(0x61E07E9A32A7866D), UINT64_C(0x350907086E777760), + UINT64_C(0xC8328DBE8B066477), UINT64_C(0x9CDBF42CD7D6957A), + UINT64_C(0xD073867288020F69), UINT64_C(0x849AFFE0D4D2FE64), + UINT64_C(0x79A1755631A3ED73), UINT64_C(0x2D480CC46D731C7E), + UINT64_C(0x110ECF10544FD5D8), UINT64_C(0x45E7B682089F24D5), + UINT64_C(0xB8DC3C34EDEE37C2), UINT64_C(0xEC3545A6B13EC6CF), + UINT64_C(0xC051BB9C9F97A48E), UINT64_C(0x94B8C20EC3475583), + UINT64_C(0x698348B826364694), UINT64_C(0x3D6A312A7AE6B799), + UINT64_C(0x012CF2FE43DA7E3F), UINT64_C(0x55C58B6C1F0A8F32), + UINT64_C(0xA8FE01DAFA7B9C25), UINT64_C(0xFC177848A6AB6D28), + UINT64_C(0xF037FDAEA72958A7), UINT64_C(0xA4DE843CFBF9A9AA), + UINT64_C(0x59E50E8A1E88BABD), UINT64_C(0x0D0C771842584BB0), + UINT64_C(0x314AB4CC7B648216), UINT64_C(0x65A3CD5E27B4731B), + UINT64_C(0x989847E8C2C5600C), UINT64_C(0xCC713E7A9E159101), + UINT64_C(0xE015C040B0BCF340), UINT64_C(0xB4FCB9D2EC6C024D), + UINT64_C(0x49C73364091D115A), UINT64_C(0x1D2E4AF655CDE057), + UINT64_C(0x216889226CF129F1), UINT64_C(0x7581F0B03021D8FC), + UINT64_C(0x88BA7A06D550CBEB), UINT64_C(0xDC53039489803AE6), + UINT64_C(0x11EA9EBA6AF9FFCD), UINT64_C(0x4503E72836290EC0), + UINT64_C(0xB8386D9ED3581DD7), UINT64_C(0xECD1140C8F88ECDA), + UINT64_C(0xD097D7D8B6B4257C), UINT64_C(0x847EAE4AEA64D471), + UINT64_C(0x794524FC0F15C766), UINT64_C(0x2DAC5D6E53C5366B), + UINT64_C(0x01C8A3547D6C542A), UINT64_C(0x5521DAC621BCA527), + UINT64_C(0xA81A5070C4CDB630), UINT64_C(0xFCF329E2981D473D), + UINT64_C(0xC0B5EA36A1218E9B), UINT64_C(0x945C93A4FDF17F96), + UINT64_C(0x6967191218806C81), UINT64_C(0x3D8E608044509D8C), + UINT64_C(0x31AEE56645D2A803), UINT64_C(0x65479CF41902590E), + UINT64_C(0x987C1642FC734A19), UINT64_C(0xCC956FD0A0A3BB14), + UINT64_C(0xF0D3AC04999F72B2), UINT64_C(0xA43AD596C54F83BF), + UINT64_C(0x59015F20203E90A8), UINT64_C(0x0DE826B27CEE61A5), + UINT64_C(0x218CD888524703E4), UINT64_C(0x7565A11A0E97F2E9), + UINT64_C(0x885E2BACEBE6E1FE), UINT64_C(0xDCB7523EB73610F3), + UINT64_C(0xE0F191EA8E0AD955), UINT64_C(0xB418E878D2DA2858), + UINT64_C(0x492362CE37AB3B4F), UINT64_C(0x1DCA1B5C6B7BCA42), + UINT64_C(0x5162690234AF5051), UINT64_C(0x058B1090687FA15C), + UINT64_C(0xF8B09A268D0EB24B), UINT64_C(0xAC59E3B4D1DE4346), + UINT64_C(0x901F2060E8E28AE0), UINT64_C(0xC4F659F2B4327BED), + UINT64_C(0x39CDD344514368FA), UINT64_C(0x6D24AAD60D9399F7), + UINT64_C(0x414054EC233AFBB6), UINT64_C(0x15A92D7E7FEA0ABB), + UINT64_C(0xE892A7C89A9B19AC), UINT64_C(0xBC7BDE5AC64BE8A1), + UINT64_C(0x803D1D8EFF772107), UINT64_C(0xD4D4641CA3A7D00A), + UINT64_C(0x29EFEEAA46D6C31D), UINT64_C(0x7D0697381A063210), + UINT64_C(0x712612DE1B84079F), UINT64_C(0x25CF6B4C4754F692), + UINT64_C(0xD8F4E1FAA225E585), UINT64_C(0x8C1D9868FEF51488), + UINT64_C(0xB05B5BBCC7C9DD2E), UINT64_C(0xE4B2222E9B192C23), + UINT64_C(0x1989A8987E683F34), UINT64_C(0x4D60D10A22B8CE39), + UINT64_C(0x61042F300C11AC78), UINT64_C(0x35ED56A250C15D75), + UINT64_C(0xC8D6DC14B5B04E62), UINT64_C(0x9C3FA586E960BF6F), + UINT64_C(0xA0796652D05C76C9), UINT64_C(0xF4901FC08C8C87C4), + UINT64_C(0x09AB957669FD94D3), UINT64_C(0x5D42ECE4352D65DE) + }, { + UINT64_C(0x0000000000000000), UINT64_C(0x3F0BE14A916A6DCB), + UINT64_C(0x7E17C29522D4DB96), UINT64_C(0x411C23DFB3BEB65D), + UINT64_C(0xFC2F852A45A9B72C), UINT64_C(0xC3246460D4C3DAE7), + UINT64_C(0x823847BF677D6CBA), UINT64_C(0xBD33A6F5F6170171), + UINT64_C(0x6A87A57F245D70DD), UINT64_C(0x558C4435B5371D16), + UINT64_C(0x149067EA0689AB4B), UINT64_C(0x2B9B86A097E3C680), + UINT64_C(0x96A8205561F4C7F1), UINT64_C(0xA9A3C11FF09EAA3A), + UINT64_C(0xE8BFE2C043201C67), UINT64_C(0xD7B4038AD24A71AC), + UINT64_C(0xD50F4AFE48BAE1BA), UINT64_C(0xEA04ABB4D9D08C71), + UINT64_C(0xAB18886B6A6E3A2C), UINT64_C(0x94136921FB0457E7), + UINT64_C(0x2920CFD40D135696), UINT64_C(0x162B2E9E9C793B5D), + UINT64_C(0x57370D412FC78D00), UINT64_C(0x683CEC0BBEADE0CB), + UINT64_C(0xBF88EF816CE79167), UINT64_C(0x80830ECBFD8DFCAC), + UINT64_C(0xC19F2D144E334AF1), UINT64_C(0xFE94CC5EDF59273A), + UINT64_C(0x43A76AAB294E264B), UINT64_C(0x7CAC8BE1B8244B80), + UINT64_C(0x3DB0A83E0B9AFDDD), UINT64_C(0x02BB49749AF09016), + UINT64_C(0x38C63AD73E7BDDF1), UINT64_C(0x07CDDB9DAF11B03A), + UINT64_C(0x46D1F8421CAF0667), UINT64_C(0x79DA19088DC56BAC), + UINT64_C(0xC4E9BFFD7BD26ADD), UINT64_C(0xFBE25EB7EAB80716), + UINT64_C(0xBAFE7D685906B14B), UINT64_C(0x85F59C22C86CDC80), + UINT64_C(0x52419FA81A26AD2C), UINT64_C(0x6D4A7EE28B4CC0E7), + UINT64_C(0x2C565D3D38F276BA), UINT64_C(0x135DBC77A9981B71), + UINT64_C(0xAE6E1A825F8F1A00), UINT64_C(0x9165FBC8CEE577CB), + UINT64_C(0xD079D8177D5BC196), UINT64_C(0xEF72395DEC31AC5D), + UINT64_C(0xEDC9702976C13C4B), UINT64_C(0xD2C29163E7AB5180), + UINT64_C(0x93DEB2BC5415E7DD), UINT64_C(0xACD553F6C57F8A16), + UINT64_C(0x11E6F50333688B67), UINT64_C(0x2EED1449A202E6AC), + UINT64_C(0x6FF1379611BC50F1), UINT64_C(0x50FAD6DC80D63D3A), + UINT64_C(0x874ED556529C4C96), UINT64_C(0xB845341CC3F6215D), + UINT64_C(0xF95917C370489700), UINT64_C(0xC652F689E122FACB), + UINT64_C(0x7B61507C1735FBBA), UINT64_C(0x446AB136865F9671), + UINT64_C(0x057692E935E1202C), UINT64_C(0x3A7D73A3A48B4DE7), + UINT64_C(0x718C75AE7CF7BBE2), UINT64_C(0x4E8794E4ED9DD629), + UINT64_C(0x0F9BB73B5E236074), UINT64_C(0x30905671CF490DBF), + UINT64_C(0x8DA3F084395E0CCE), UINT64_C(0xB2A811CEA8346105), + UINT64_C(0xF3B432111B8AD758), UINT64_C(0xCCBFD35B8AE0BA93), + UINT64_C(0x1B0BD0D158AACB3F), UINT64_C(0x2400319BC9C0A6F4), + UINT64_C(0x651C12447A7E10A9), UINT64_C(0x5A17F30EEB147D62), + UINT64_C(0xE72455FB1D037C13), UINT64_C(0xD82FB4B18C6911D8), + UINT64_C(0x9933976E3FD7A785), UINT64_C(0xA6387624AEBDCA4E), + UINT64_C(0xA4833F50344D5A58), UINT64_C(0x9B88DE1AA5273793), + UINT64_C(0xDA94FDC5169981CE), UINT64_C(0xE59F1C8F87F3EC05), + UINT64_C(0x58ACBA7A71E4ED74), UINT64_C(0x67A75B30E08E80BF), + UINT64_C(0x26BB78EF533036E2), UINT64_C(0x19B099A5C25A5B29), + UINT64_C(0xCE049A2F10102A85), UINT64_C(0xF10F7B65817A474E), + UINT64_C(0xB01358BA32C4F113), UINT64_C(0x8F18B9F0A3AE9CD8), + UINT64_C(0x322B1F0555B99DA9), UINT64_C(0x0D20FE4FC4D3F062), + UINT64_C(0x4C3CDD90776D463F), UINT64_C(0x73373CDAE6072BF4), + UINT64_C(0x494A4F79428C6613), UINT64_C(0x7641AE33D3E60BD8), + UINT64_C(0x375D8DEC6058BD85), UINT64_C(0x08566CA6F132D04E), + UINT64_C(0xB565CA530725D13F), UINT64_C(0x8A6E2B19964FBCF4), + UINT64_C(0xCB7208C625F10AA9), UINT64_C(0xF479E98CB49B6762), + UINT64_C(0x23CDEA0666D116CE), UINT64_C(0x1CC60B4CF7BB7B05), + UINT64_C(0x5DDA28934405CD58), UINT64_C(0x62D1C9D9D56FA093), + UINT64_C(0xDFE26F2C2378A1E2), UINT64_C(0xE0E98E66B212CC29), + UINT64_C(0xA1F5ADB901AC7A74), UINT64_C(0x9EFE4CF390C617BF), + UINT64_C(0x9C4505870A3687A9), UINT64_C(0xA34EE4CD9B5CEA62), + UINT64_C(0xE252C71228E25C3F), UINT64_C(0xDD592658B98831F4), + UINT64_C(0x606A80AD4F9F3085), UINT64_C(0x5F6161E7DEF55D4E), + UINT64_C(0x1E7D42386D4BEB13), UINT64_C(0x2176A372FC2186D8), + UINT64_C(0xF6C2A0F82E6BF774), UINT64_C(0xC9C941B2BF019ABF), + UINT64_C(0x88D5626D0CBF2CE2), UINT64_C(0xB7DE83279DD54129), + UINT64_C(0x0AED25D26BC24058), UINT64_C(0x35E6C498FAA82D93), + UINT64_C(0x74FAE74749169BCE), UINT64_C(0x4BF1060DD87CF605), + UINT64_C(0xE318EB5CF9EF77C4), UINT64_C(0xDC130A1668851A0F), + UINT64_C(0x9D0F29C9DB3BAC52), UINT64_C(0xA204C8834A51C199), + UINT64_C(0x1F376E76BC46C0E8), UINT64_C(0x203C8F3C2D2CAD23), + UINT64_C(0x6120ACE39E921B7E), UINT64_C(0x5E2B4DA90FF876B5), + UINT64_C(0x899F4E23DDB20719), UINT64_C(0xB694AF694CD86AD2), + UINT64_C(0xF7888CB6FF66DC8F), UINT64_C(0xC8836DFC6E0CB144), + UINT64_C(0x75B0CB09981BB035), UINT64_C(0x4ABB2A430971DDFE), + UINT64_C(0x0BA7099CBACF6BA3), UINT64_C(0x34ACE8D62BA50668), + UINT64_C(0x3617A1A2B155967E), UINT64_C(0x091C40E8203FFBB5), + UINT64_C(0x4800633793814DE8), UINT64_C(0x770B827D02EB2023), + UINT64_C(0xCA382488F4FC2152), UINT64_C(0xF533C5C265964C99), + UINT64_C(0xB42FE61DD628FAC4), UINT64_C(0x8B2407574742970F), + UINT64_C(0x5C9004DD9508E6A3), UINT64_C(0x639BE59704628B68), + UINT64_C(0x2287C648B7DC3D35), UINT64_C(0x1D8C270226B650FE), + UINT64_C(0xA0BF81F7D0A1518F), UINT64_C(0x9FB460BD41CB3C44), + UINT64_C(0xDEA84362F2758A19), UINT64_C(0xE1A3A228631FE7D2), + UINT64_C(0xDBDED18BC794AA35), UINT64_C(0xE4D530C156FEC7FE), + UINT64_C(0xA5C9131EE54071A3), UINT64_C(0x9AC2F254742A1C68), + UINT64_C(0x27F154A1823D1D19), UINT64_C(0x18FAB5EB135770D2), + UINT64_C(0x59E69634A0E9C68F), UINT64_C(0x66ED777E3183AB44), + UINT64_C(0xB15974F4E3C9DAE8), UINT64_C(0x8E5295BE72A3B723), + UINT64_C(0xCF4EB661C11D017E), UINT64_C(0xF045572B50776CB5), + UINT64_C(0x4D76F1DEA6606DC4), UINT64_C(0x727D1094370A000F), + UINT64_C(0x3361334B84B4B652), UINT64_C(0x0C6AD20115DEDB99), + UINT64_C(0x0ED19B758F2E4B8F), UINT64_C(0x31DA7A3F1E442644), + UINT64_C(0x70C659E0ADFA9019), UINT64_C(0x4FCDB8AA3C90FDD2), + UINT64_C(0xF2FE1E5FCA87FCA3), UINT64_C(0xCDF5FF155BED9168), + UINT64_C(0x8CE9DCCAE8532735), UINT64_C(0xB3E23D8079394AFE), + UINT64_C(0x64563E0AAB733B52), UINT64_C(0x5B5DDF403A195699), + UINT64_C(0x1A41FC9F89A7E0C4), UINT64_C(0x254A1DD518CD8D0F), + UINT64_C(0x9879BB20EEDA8C7E), UINT64_C(0xA7725A6A7FB0E1B5), + UINT64_C(0xE66E79B5CC0E57E8), UINT64_C(0xD96598FF5D643A23), + UINT64_C(0x92949EF28518CC26), UINT64_C(0xAD9F7FB81472A1ED), + UINT64_C(0xEC835C67A7CC17B0), UINT64_C(0xD388BD2D36A67A7B), + UINT64_C(0x6EBB1BD8C0B17B0A), UINT64_C(0x51B0FA9251DB16C1), + UINT64_C(0x10ACD94DE265A09C), UINT64_C(0x2FA73807730FCD57), + UINT64_C(0xF8133B8DA145BCFB), UINT64_C(0xC718DAC7302FD130), + UINT64_C(0x8604F9188391676D), UINT64_C(0xB90F185212FB0AA6), + UINT64_C(0x043CBEA7E4EC0BD7), UINT64_C(0x3B375FED7586661C), + UINT64_C(0x7A2B7C32C638D041), UINT64_C(0x45209D785752BD8A), + UINT64_C(0x479BD40CCDA22D9C), UINT64_C(0x789035465CC84057), + UINT64_C(0x398C1699EF76F60A), UINT64_C(0x0687F7D37E1C9BC1), + UINT64_C(0xBBB45126880B9AB0), UINT64_C(0x84BFB06C1961F77B), + UINT64_C(0xC5A393B3AADF4126), UINT64_C(0xFAA872F93BB52CED), + UINT64_C(0x2D1C7173E9FF5D41), UINT64_C(0x121790397895308A), + UINT64_C(0x530BB3E6CB2B86D7), UINT64_C(0x6C0052AC5A41EB1C), + UINT64_C(0xD133F459AC56EA6D), UINT64_C(0xEE3815133D3C87A6), + UINT64_C(0xAF2436CC8E8231FB), UINT64_C(0x902FD7861FE85C30), + UINT64_C(0xAA52A425BB6311D7), UINT64_C(0x9559456F2A097C1C), + UINT64_C(0xD44566B099B7CA41), UINT64_C(0xEB4E87FA08DDA78A), + UINT64_C(0x567D210FFECAA6FB), UINT64_C(0x6976C0456FA0CB30), + UINT64_C(0x286AE39ADC1E7D6D), UINT64_C(0x176102D04D7410A6), + UINT64_C(0xC0D5015A9F3E610A), UINT64_C(0xFFDEE0100E540CC1), + UINT64_C(0xBEC2C3CFBDEABA9C), UINT64_C(0x81C922852C80D757), + UINT64_C(0x3CFA8470DA97D626), UINT64_C(0x03F1653A4BFDBBED), + UINT64_C(0x42ED46E5F8430DB0), UINT64_C(0x7DE6A7AF6929607B), + UINT64_C(0x7F5DEEDBF3D9F06D), UINT64_C(0x40560F9162B39DA6), + UINT64_C(0x014A2C4ED10D2BFB), UINT64_C(0x3E41CD0440674630), + UINT64_C(0x83726BF1B6704741), UINT64_C(0xBC798ABB271A2A8A), + UINT64_C(0xFD65A96494A49CD7), UINT64_C(0xC26E482E05CEF11C), + UINT64_C(0x15DA4BA4D78480B0), UINT64_C(0x2AD1AAEE46EEED7B), + UINT64_C(0x6BCD8931F5505B26), UINT64_C(0x54C6687B643A36ED), + UINT64_C(0xE9F5CE8E922D379C), UINT64_C(0xD6FE2FC403475A57), + UINT64_C(0x97E20C1BB0F9EC0A), UINT64_C(0xA8E9ED51219381C1) + }, { + UINT64_C(0x0000000000000000), UINT64_C(0x1DEE8A5E222CA1DC), + UINT64_C(0x3BDD14BC445943B8), UINT64_C(0x26339EE26675E264), + UINT64_C(0x77BA297888B28770), UINT64_C(0x6A54A326AA9E26AC), + UINT64_C(0x4C673DC4CCEBC4C8), UINT64_C(0x5189B79AEEC76514), + UINT64_C(0xEF7452F111650EE0), UINT64_C(0xF29AD8AF3349AF3C), + UINT64_C(0xD4A9464D553C4D58), UINT64_C(0xC947CC137710EC84), + UINT64_C(0x98CE7B8999D78990), UINT64_C(0x8520F1D7BBFB284C), + UINT64_C(0xA3136F35DD8ECA28), UINT64_C(0xBEFDE56BFFA26BF4), + UINT64_C(0x4C300AC98DC40345), UINT64_C(0x51DE8097AFE8A299), + UINT64_C(0x77ED1E75C99D40FD), UINT64_C(0x6A03942BEBB1E121), + UINT64_C(0x3B8A23B105768435), UINT64_C(0x2664A9EF275A25E9), + UINT64_C(0x0057370D412FC78D), UINT64_C(0x1DB9BD5363036651), + UINT64_C(0xA34458389CA10DA5), UINT64_C(0xBEAAD266BE8DAC79), + UINT64_C(0x98994C84D8F84E1D), UINT64_C(0x8577C6DAFAD4EFC1), + UINT64_C(0xD4FE714014138AD5), UINT64_C(0xC910FB1E363F2B09), + UINT64_C(0xEF2365FC504AC96D), UINT64_C(0xF2CDEFA2726668B1), + UINT64_C(0x986015931B88068A), UINT64_C(0x858E9FCD39A4A756), + UINT64_C(0xA3BD012F5FD14532), UINT64_C(0xBE538B717DFDE4EE), + UINT64_C(0xEFDA3CEB933A81FA), UINT64_C(0xF234B6B5B1162026), + UINT64_C(0xD4072857D763C242), UINT64_C(0xC9E9A209F54F639E), + UINT64_C(0x771447620AED086A), UINT64_C(0x6AFACD3C28C1A9B6), + UINT64_C(0x4CC953DE4EB44BD2), UINT64_C(0x5127D9806C98EA0E), + UINT64_C(0x00AE6E1A825F8F1A), UINT64_C(0x1D40E444A0732EC6), + UINT64_C(0x3B737AA6C606CCA2), UINT64_C(0x269DF0F8E42A6D7E), + UINT64_C(0xD4501F5A964C05CF), UINT64_C(0xC9BE9504B460A413), + UINT64_C(0xEF8D0BE6D2154677), UINT64_C(0xF26381B8F039E7AB), + UINT64_C(0xA3EA36221EFE82BF), UINT64_C(0xBE04BC7C3CD22363), + UINT64_C(0x9837229E5AA7C107), UINT64_C(0x85D9A8C0788B60DB), + UINT64_C(0x3B244DAB87290B2F), UINT64_C(0x26CAC7F5A505AAF3), + UINT64_C(0x00F95917C3704897), UINT64_C(0x1D17D349E15CE94B), + UINT64_C(0x4C9E64D30F9B8C5F), UINT64_C(0x5170EE8D2DB72D83), + UINT64_C(0x7743706F4BC2CFE7), UINT64_C(0x6AADFA3169EE6E3B), + UINT64_C(0xA218840D981E1391), UINT64_C(0xBFF60E53BA32B24D), + UINT64_C(0x99C590B1DC475029), UINT64_C(0x842B1AEFFE6BF1F5), + UINT64_C(0xD5A2AD7510AC94E1), UINT64_C(0xC84C272B3280353D), + UINT64_C(0xEE7FB9C954F5D759), UINT64_C(0xF391339776D97685), + UINT64_C(0x4D6CD6FC897B1D71), UINT64_C(0x50825CA2AB57BCAD), + UINT64_C(0x76B1C240CD225EC9), UINT64_C(0x6B5F481EEF0EFF15), + UINT64_C(0x3AD6FF8401C99A01), UINT64_C(0x273875DA23E53BDD), + UINT64_C(0x010BEB384590D9B9), UINT64_C(0x1CE5616667BC7865), + UINT64_C(0xEE288EC415DA10D4), UINT64_C(0xF3C6049A37F6B108), + UINT64_C(0xD5F59A785183536C), UINT64_C(0xC81B102673AFF2B0), + UINT64_C(0x9992A7BC9D6897A4), UINT64_C(0x847C2DE2BF443678), + UINT64_C(0xA24FB300D931D41C), UINT64_C(0xBFA1395EFB1D75C0), + UINT64_C(0x015CDC3504BF1E34), UINT64_C(0x1CB2566B2693BFE8), + UINT64_C(0x3A81C88940E65D8C), UINT64_C(0x276F42D762CAFC50), + UINT64_C(0x76E6F54D8C0D9944), UINT64_C(0x6B087F13AE213898), + UINT64_C(0x4D3BE1F1C854DAFC), UINT64_C(0x50D56BAFEA787B20), + UINT64_C(0x3A78919E8396151B), UINT64_C(0x27961BC0A1BAB4C7), + UINT64_C(0x01A58522C7CF56A3), UINT64_C(0x1C4B0F7CE5E3F77F), + UINT64_C(0x4DC2B8E60B24926B), UINT64_C(0x502C32B8290833B7), + UINT64_C(0x761FAC5A4F7DD1D3), UINT64_C(0x6BF126046D51700F), + UINT64_C(0xD50CC36F92F31BFB), UINT64_C(0xC8E24931B0DFBA27), + UINT64_C(0xEED1D7D3D6AA5843), UINT64_C(0xF33F5D8DF486F99F), + UINT64_C(0xA2B6EA171A419C8B), UINT64_C(0xBF586049386D3D57), + UINT64_C(0x996BFEAB5E18DF33), UINT64_C(0x848574F57C347EEF), + UINT64_C(0x76489B570E52165E), UINT64_C(0x6BA611092C7EB782), + UINT64_C(0x4D958FEB4A0B55E6), UINT64_C(0x507B05B56827F43A), + UINT64_C(0x01F2B22F86E0912E), UINT64_C(0x1C1C3871A4CC30F2), + UINT64_C(0x3A2FA693C2B9D296), UINT64_C(0x27C12CCDE095734A), + UINT64_C(0x993CC9A61F3718BE), UINT64_C(0x84D243F83D1BB962), + UINT64_C(0xA2E1DD1A5B6E5B06), UINT64_C(0xBF0F57447942FADA), + UINT64_C(0xEE86E0DE97859FCE), UINT64_C(0xF3686A80B5A93E12), + UINT64_C(0xD55BF462D3DCDC76), UINT64_C(0xC8B57E3CF1F07DAA), + UINT64_C(0xD6E9A7309F3239A7), UINT64_C(0xCB072D6EBD1E987B), + UINT64_C(0xED34B38CDB6B7A1F), UINT64_C(0xF0DA39D2F947DBC3), + UINT64_C(0xA1538E481780BED7), UINT64_C(0xBCBD041635AC1F0B), + UINT64_C(0x9A8E9AF453D9FD6F), UINT64_C(0x876010AA71F55CB3), + UINT64_C(0x399DF5C18E573747), UINT64_C(0x24737F9FAC7B969B), + UINT64_C(0x0240E17DCA0E74FF), UINT64_C(0x1FAE6B23E822D523), + UINT64_C(0x4E27DCB906E5B037), UINT64_C(0x53C956E724C911EB), + UINT64_C(0x75FAC80542BCF38F), UINT64_C(0x6814425B60905253), + UINT64_C(0x9AD9ADF912F63AE2), UINT64_C(0x873727A730DA9B3E), + UINT64_C(0xA104B94556AF795A), UINT64_C(0xBCEA331B7483D886), + UINT64_C(0xED6384819A44BD92), UINT64_C(0xF08D0EDFB8681C4E), + UINT64_C(0xD6BE903DDE1DFE2A), UINT64_C(0xCB501A63FC315FF6), + UINT64_C(0x75ADFF0803933402), UINT64_C(0x6843755621BF95DE), + UINT64_C(0x4E70EBB447CA77BA), UINT64_C(0x539E61EA65E6D666), + UINT64_C(0x0217D6708B21B372), UINT64_C(0x1FF95C2EA90D12AE), + UINT64_C(0x39CAC2CCCF78F0CA), UINT64_C(0x24244892ED545116), + UINT64_C(0x4E89B2A384BA3F2D), UINT64_C(0x536738FDA6969EF1), + UINT64_C(0x7554A61FC0E37C95), UINT64_C(0x68BA2C41E2CFDD49), + UINT64_C(0x39339BDB0C08B85D), UINT64_C(0x24DD11852E241981), + UINT64_C(0x02EE8F674851FBE5), UINT64_C(0x1F0005396A7D5A39), + UINT64_C(0xA1FDE05295DF31CD), UINT64_C(0xBC136A0CB7F39011), + UINT64_C(0x9A20F4EED1867275), UINT64_C(0x87CE7EB0F3AAD3A9), + UINT64_C(0xD647C92A1D6DB6BD), UINT64_C(0xCBA943743F411761), + UINT64_C(0xED9ADD965934F505), UINT64_C(0xF07457C87B1854D9), + UINT64_C(0x02B9B86A097E3C68), UINT64_C(0x1F5732342B529DB4), + UINT64_C(0x3964ACD64D277FD0), UINT64_C(0x248A26886F0BDE0C), + UINT64_C(0x7503911281CCBB18), UINT64_C(0x68ED1B4CA3E01AC4), + UINT64_C(0x4EDE85AEC595F8A0), UINT64_C(0x53300FF0E7B9597C), + UINT64_C(0xEDCDEA9B181B3288), UINT64_C(0xF02360C53A379354), + UINT64_C(0xD610FE275C427130), UINT64_C(0xCBFE74797E6ED0EC), + UINT64_C(0x9A77C3E390A9B5F8), UINT64_C(0x879949BDB2851424), + UINT64_C(0xA1AAD75FD4F0F640), UINT64_C(0xBC445D01F6DC579C), + UINT64_C(0x74F1233D072C2A36), UINT64_C(0x691FA96325008BEA), + UINT64_C(0x4F2C37814375698E), UINT64_C(0x52C2BDDF6159C852), + UINT64_C(0x034B0A458F9EAD46), UINT64_C(0x1EA5801BADB20C9A), + UINT64_C(0x38961EF9CBC7EEFE), UINT64_C(0x257894A7E9EB4F22), + UINT64_C(0x9B8571CC164924D6), UINT64_C(0x866BFB923465850A), + UINT64_C(0xA05865705210676E), UINT64_C(0xBDB6EF2E703CC6B2), + UINT64_C(0xEC3F58B49EFBA3A6), UINT64_C(0xF1D1D2EABCD7027A), + UINT64_C(0xD7E24C08DAA2E01E), UINT64_C(0xCA0CC656F88E41C2), + UINT64_C(0x38C129F48AE82973), UINT64_C(0x252FA3AAA8C488AF), + UINT64_C(0x031C3D48CEB16ACB), UINT64_C(0x1EF2B716EC9DCB17), + UINT64_C(0x4F7B008C025AAE03), UINT64_C(0x52958AD220760FDF), + UINT64_C(0x74A614304603EDBB), UINT64_C(0x69489E6E642F4C67), + UINT64_C(0xD7B57B059B8D2793), UINT64_C(0xCA5BF15BB9A1864F), + UINT64_C(0xEC686FB9DFD4642B), UINT64_C(0xF186E5E7FDF8C5F7), + UINT64_C(0xA00F527D133FA0E3), UINT64_C(0xBDE1D8233113013F), + UINT64_C(0x9BD246C15766E35B), UINT64_C(0x863CCC9F754A4287), + UINT64_C(0xEC9136AE1CA42CBC), UINT64_C(0xF17FBCF03E888D60), + UINT64_C(0xD74C221258FD6F04), UINT64_C(0xCAA2A84C7AD1CED8), + UINT64_C(0x9B2B1FD69416ABCC), UINT64_C(0x86C59588B63A0A10), + UINT64_C(0xA0F60B6AD04FE874), UINT64_C(0xBD188134F26349A8), + UINT64_C(0x03E5645F0DC1225C), UINT64_C(0x1E0BEE012FED8380), + UINT64_C(0x383870E3499861E4), UINT64_C(0x25D6FABD6BB4C038), + UINT64_C(0x745F4D278573A52C), UINT64_C(0x69B1C779A75F04F0), + UINT64_C(0x4F82599BC12AE694), UINT64_C(0x526CD3C5E3064748), + UINT64_C(0xA0A13C6791602FF9), UINT64_C(0xBD4FB639B34C8E25), + UINT64_C(0x9B7C28DBD5396C41), UINT64_C(0x8692A285F715CD9D), + UINT64_C(0xD71B151F19D2A889), UINT64_C(0xCAF59F413BFE0955), + UINT64_C(0xECC601A35D8BEB31), UINT64_C(0xF1288BFD7FA74AED), + UINT64_C(0x4FD56E9680052119), UINT64_C(0x523BE4C8A22980C5), + UINT64_C(0x74087A2AC45C62A1), UINT64_C(0x69E6F074E670C37D), + UINT64_C(0x386F47EE08B7A669), UINT64_C(0x2581CDB02A9B07B5), + UINT64_C(0x03B253524CEEE5D1), UINT64_C(0x1E5CD90C6EC2440D) + } +}; diff --git a/src/liblzma/check/crc64_tablegen.c b/src/liblzma/check/crc64_tablegen.c new file mode 100644 index 0000000000..2035127a11 --- /dev/null +++ b/src/liblzma/check/crc64_tablegen.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc64_tablegen.c +/// \brief Generate crc64_table_le.h and crc64_table_be.h +/// +/// Compiling: gcc -std=c99 -o crc64_tablegen crc64_tablegen.c +/// Add -DWORDS_BIGENDIAN to generate big endian table. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include +#include "../../common/tuklib_integer.h" + + +static uint64_t crc64_table[4][256]; + + +extern void +init_crc64_table(void) +{ + static const uint64_t poly64 = UINT64_C(0xC96C5795D7870F42); + + for (size_t s = 0; s < 4; ++s) { + for (size_t b = 0; b < 256; ++b) { + uint64_t r = s == 0 ? b : crc64_table[s - 1][b]; + + for (size_t i = 0; i < 8; ++i) { + if (r & 1) + r = (r >> 1) ^ poly64; + else + r >>= 1; + } + + crc64_table[s][b] = r; + } + } + +#ifdef WORDS_BIGENDIAN + for (size_t s = 0; s < 4; ++s) + for (size_t b = 0; b < 256; ++b) + crc64_table[s][b] = byteswap64(crc64_table[s][b]); +#endif + + return; +} + + +static void +print_crc64_table(void) +{ + // Split the SPDX string so that it won't accidentally match + // when tools search for the string. + printf("// SPDX" "-License-Identifier" ": 0BSD\n\n" + "// This file has been generated by crc64_tablegen.c.\n\n" + "const uint64_t lzma_crc64_table[4][256] = {\n\t{"); + + for (size_t s = 0; s < 4; ++s) { + for (size_t b = 0; b < 256; ++b) { + if ((b % 2) == 0) + printf("\n\t\t"); + + printf("UINT64_C(0x%016" PRIX64 ")", + crc64_table[s][b]); + + if (b != 255) + printf(",%s", (b+1) % 2 == 0 ? "" : " "); + } + + if (s == 3) + printf("\n\t}\n};\n"); + else + printf("\n\t}, {"); + } + + return; +} + + +int +main(void) +{ + init_crc64_table(); + print_crc64_table(); + return 0; +} diff --git a/src/liblzma/check/crc64_x86.S b/src/liblzma/check/crc64_x86.S new file mode 100644 index 0000000000..df50018653 --- /dev/null +++ b/src/liblzma/check/crc64_x86.S @@ -0,0 +1,291 @@ +/* SPDX-License-Identifier: 0BSD */ + +/* + * Speed-optimized CRC64 using slicing-by-four algorithm + * + * This uses only i386 instructions, but it is optimized for i686 and later + * (including e.g. Pentium II/III/IV, Athlon XP, and Core 2). + * + * Authors: Igor Pavlov (original CRC32 assembly code) + * Lasse Collin (CRC64 adaptation of the modified CRC32 code) + * + * This code needs lzma_crc64_table, which can be created using the + * following C code: + +uint64_t lzma_crc64_table[4][256]; + +void +init_table(void) +{ + // ECMA-182 + static const uint64_t poly64 = UINT64_C(0xC96C5795D7870F42); + + for (size_t s = 0; s < 4; ++s) { + for (size_t b = 0; b < 256; ++b) { + uint64_t r = s == 0 ? b : lzma_crc64_table[s - 1][b]; + + for (size_t i = 0; i < 8; ++i) { + if (r & 1) + r = (r >> 1) ^ poly64; + else + r >>= 1; + } + + lzma_crc64_table[s][b] = r; + } + } +} + + * The prototype of the CRC64 function: + * extern uint64_t lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc); + */ + +/* When Intel CET is enabled, include in assembly code to mark + Intel CET support. */ +#ifdef __CET__ +# include +#else +# define _CET_ENDBR +#endif + +/* + * On some systems, the functions need to be prefixed. The prefix is + * usually an underscore. + */ +#ifndef __USER_LABEL_PREFIX__ +# define __USER_LABEL_PREFIX__ +#endif +#define MAKE_SYM_CAT(prefix, sym) prefix ## sym +#define MAKE_SYM(prefix, sym) MAKE_SYM_CAT(prefix, sym) +#define LZMA_CRC64 MAKE_SYM(__USER_LABEL_PREFIX__, lzma_crc64_generic) +#define LZMA_CRC64_TABLE MAKE_SYM(__USER_LABEL_PREFIX__, lzma_crc64_table) + +/* + * Solaris assembler doesn't have .p2align, and Darwin uses .align + * differently than GNU/Linux and Solaris. + */ +#if defined(__APPLE__) || defined(__MSDOS__) +# define ALIGN(pow2, abs) .align pow2 +#else +# define ALIGN(pow2, abs) .align abs +#endif + + .text + .globl LZMA_CRC64 +#ifdef __ELF__ + .hidden LZMA_CRC64 +#endif + +#if !defined(__APPLE__) && !defined(_WIN32) && !defined(__CYGWIN__) \ + && !defined(__MSDOS__) + .type LZMA_CRC64, @function +#endif + + ALIGN(4, 16) +LZMA_CRC64: + _CET_ENDBR + /* + * Register usage: + * %eax crc LSB + * %edx crc MSB + * %esi buf + * %edi size or buf + size + * %ebx lzma_crc64_table + * %ebp Table index + * %ecx Temporary + */ + pushl %ebx + pushl %esi + pushl %edi + pushl %ebp + movl 0x14(%esp), %esi /* buf */ + movl 0x18(%esp), %edi /* size */ + movl 0x1C(%esp), %eax /* crc LSB */ + movl 0x20(%esp), %edx /* crc MSB */ + + /* + * Store the address of lzma_crc64_table to %ebx. This is needed to + * get position-independent code (PIC). + * + * The PIC macro is defined by libtool, while __PIC__ is defined + * by GCC but only on some systems. Testing for both makes it simpler + * to test this code without libtool, and keeps the code working also + * when built with libtool but using something else than GCC. + * + * I understood that libtool may define PIC on Windows even though + * the code in Windows DLLs is not PIC in sense that it is in ELF + * binaries, so we need a separate check to always use the non-PIC + * code on Windows. + */ +#if (!defined(PIC) && !defined(__PIC__)) \ + || (defined(_WIN32) || defined(__CYGWIN__)) + /* Not PIC */ + movl $ LZMA_CRC64_TABLE, %ebx +#elif defined(__APPLE__) + /* Mach-O */ + call .L_get_pc +.L_pic: + leal .L_lzma_crc64_table$non_lazy_ptr-.L_pic(%ebx), %ebx + movl (%ebx), %ebx +#else + /* ELF */ + call .L_get_pc + addl $_GLOBAL_OFFSET_TABLE_, %ebx + movl LZMA_CRC64_TABLE@GOT(%ebx), %ebx +#endif + + /* Complement the initial value. */ + notl %eax + notl %edx + +.L_align: + /* + * Check if there is enough input to use slicing-by-four. + * We need eight bytes, because the loop pre-reads four bytes. + */ + cmpl $8, %edi + jb .L_rest + + /* Check if we have reached alignment of four bytes. */ + testl $3, %esi + jz .L_slice + + /* Calculate CRC of the next input byte. */ + movzbl (%esi), %ebp + incl %esi + movzbl %al, %ecx + xorl %ecx, %ebp + shrdl $8, %edx, %eax + xorl (%ebx, %ebp, 8), %eax + shrl $8, %edx + xorl 4(%ebx, %ebp, 8), %edx + decl %edi + jmp .L_align + +.L_slice: + /* + * If we get here, there's at least eight bytes of aligned input + * available. Make %edi multiple of four bytes. Store the possible + * remainder over the "size" variable in the argument stack. + */ + movl %edi, 0x18(%esp) + andl $-4, %edi + subl %edi, 0x18(%esp) + + /* + * Let %edi be buf + size - 4 while running the main loop. This way + * we can compare for equality to determine when exit the loop. + */ + addl %esi, %edi + subl $4, %edi + + /* Read in the first four aligned bytes. */ + movl (%esi), %ecx + +.L_loop: + xorl %eax, %ecx + movzbl %cl, %ebp + movl 0x1800(%ebx, %ebp, 8), %eax + xorl %edx, %eax + movl 0x1804(%ebx, %ebp, 8), %edx + movzbl %ch, %ebp + xorl 0x1000(%ebx, %ebp, 8), %eax + xorl 0x1004(%ebx, %ebp, 8), %edx + shrl $16, %ecx + movzbl %cl, %ebp + xorl 0x0800(%ebx, %ebp, 8), %eax + xorl 0x0804(%ebx, %ebp, 8), %edx + movzbl %ch, %ebp + addl $4, %esi + xorl (%ebx, %ebp, 8), %eax + xorl 4(%ebx, %ebp, 8), %edx + + /* Check for end of aligned input. */ + cmpl %edi, %esi + + /* + * Copy the next input byte to %ecx. It is slightly faster to + * read it here than at the top of the loop. + */ + movl (%esi), %ecx + jb .L_loop + + /* + * Process the remaining four bytes, which we have already + * copied to %ecx. + */ + xorl %eax, %ecx + movzbl %cl, %ebp + movl 0x1800(%ebx, %ebp, 8), %eax + xorl %edx, %eax + movl 0x1804(%ebx, %ebp, 8), %edx + movzbl %ch, %ebp + xorl 0x1000(%ebx, %ebp, 8), %eax + xorl 0x1004(%ebx, %ebp, 8), %edx + shrl $16, %ecx + movzbl %cl, %ebp + xorl 0x0800(%ebx, %ebp, 8), %eax + xorl 0x0804(%ebx, %ebp, 8), %edx + movzbl %ch, %ebp + addl $4, %esi + xorl (%ebx, %ebp, 8), %eax + xorl 4(%ebx, %ebp, 8), %edx + + /* Copy the number of remaining bytes to %edi. */ + movl 0x18(%esp), %edi + +.L_rest: + /* Check for end of input. */ + testl %edi, %edi + jz .L_return + + /* Calculate CRC of the next input byte. */ + movzbl (%esi), %ebp + incl %esi + movzbl %al, %ecx + xorl %ecx, %ebp + shrdl $8, %edx, %eax + xorl (%ebx, %ebp, 8), %eax + shrl $8, %edx + xorl 4(%ebx, %ebp, 8), %edx + decl %edi + jmp .L_rest + +.L_return: + /* Complement the final value. */ + notl %eax + notl %edx + + popl %ebp + popl %edi + popl %esi + popl %ebx + ret + +#if defined(PIC) || defined(__PIC__) + ALIGN(4, 16) +.L_get_pc: + movl (%esp), %ebx + ret +#endif + +#if defined(__APPLE__) && (defined(PIC) || defined(__PIC__)) + /* Mach-O PIC */ + .section __IMPORT,__pointers,non_lazy_symbol_pointers +.L_lzma_crc64_table$non_lazy_ptr: + .indirect_symbol LZMA_CRC64_TABLE + .long 0 + +#elif !defined(_WIN32) && !defined(__CYGWIN__) && !defined(__MSDOS__) + /* ELF */ + .size LZMA_CRC64, .-LZMA_CRC64 +#endif + +/* + * This is needed to support non-executable stack. It's ugly to + * use __FreeBSD__ and __linux__ here, but I don't know a way to detect when + * we are using GNU assembler. + */ +#if defined(__ELF__) && (defined(__FreeBSD__) || defined(__linux__)) + .section .note.GNU-stack,"",@progbits +#endif diff --git a/src/liblzma/check/crc_clmul_consts_gen.c b/src/liblzma/check/crc_clmul_consts_gen.c new file mode 100644 index 0000000000..5fe14bd6f0 --- /dev/null +++ b/src/liblzma/check/crc_clmul_consts_gen.c @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc_clmul_consts_gen.c +/// \brief Generate constants for CLMUL CRC code +/// +/// Compiling: gcc -std=c99 -o crc_clmul_consts_gen crc_clmul_consts_gen.c +/// +/// This is for CRCs that use reversed bit order (bit reflection). +/// The same CLMUL CRC code can be used with CRC64 and smaller ones like +/// CRC32 apart from one special case: CRC64 needs an extra step in the +/// Barrett reduction to handle the 65th bit; the smaller ones don't. +/// Otherwise it's enough to just change the polynomial and the derived +/// constants and use the same code. +/// +/// See the Intel white paper "Fast CRC Computation for Generic Polynomials +/// Using PCLMULQDQ Instruction" from 2009. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include +#include + + +/// CRC32 (Ethernet) polynomial in reversed representation +static const uint64_t p32 = 0xedb88320; + +// CRC64 (ECMA-182) polynomial in reversed representation +static const uint64_t p64 = 0xc96c5795d7870f42; + + +/// Calculates floor(x^128 / p) where p is a CRC64 polynomial in +/// reversed representation. The result is in reversed representation too. +static uint64_t +calc_cldiv(uint64_t p) +{ + // Quotient + uint64_t q = 0; + + // Align the x^64 term with the x^128 (the implied high bits of the + // divisor and the dividend) and do the first step of polynomial long + // division, calculating the first remainder. The variable q remains + // zero because the highest bit of the quotient is an implied bit 1 + // (we kind of set q = 1 << -1). + uint64_t r = p; + + // Then process the remaining 64 terms. Note that r has no implied + // high bit, only q and p do. (And remember that a high bit in the + // polynomial is stored at a low bit in the variable due to the + // reversed bit order.) + for (unsigned i = 0; i < 64; ++i) { + q |= (r & 1) << i; + r = (r >> 1) ^ (r & 1 ? p : 0); + } + + return q; +} + + +/// Calculate the remainder of carryless division: +/// +/// x^(bits + n - 1) % p, where n=64 (for CRC64) +/// +/// p must be in reversed representation which omits the bit of +/// the highest term of the polynomial. Instead, it is an implied bit +/// at kind of like "1 << -1" position, as if it had just been shifted out. +/// +/// The return value is in the reversed bit order. (There are no implied bits.) +static uint64_t +calc_clrem(uint64_t p, unsigned bits) +{ + // Do the first step of polynomial long division. + uint64_t r = p; + + // Then process the remaining terms. Start with i = 1 instead of i = 0 + // to account for the -1 in x^(bits + n - 1). This -1 is convenient + // with the reversed bit order. See the "Bit-Reflection" section in + // the Intel white paper. + for (unsigned i = 1; i < bits; ++i) + r = (r >> 1) ^ (r & 1 ? p : 0); + + return r; +} + + +extern int +main(void) +{ + puts("// CRC64"); + + // The order of the two 64-bit constants in a vector don't matter. + // It feels logical to put them in this order as it matches the + // order in which the input bytes are read. + printf("const __m128i fold512 = _mm_set_epi64x(" + "0x%016" PRIx64 ", 0x%016" PRIx64 ");\n", + calc_clrem(p64, 4 * 128 - 64), + calc_clrem(p64, 4 * 128)); + + printf("const __m128i fold128 = _mm_set_epi64x(" + "0x%016" PRIx64 ", 0x%016" PRIx64 ");\n", + calc_clrem(p64, 128 - 64), + calc_clrem(p64, 128)); + + // When we multiply by mu, we care about the high bits of the result + // (in reversed bit order!). It doesn't matter that the low bit gets + // shifted out because the affected output bits will be ignored. + // Below we add the implied high bit with "| 1" after the shifting + // so that the high bits of the multiplication will be correct. + // + // p64 is shifted left by one so that the final multiplication + // in Barrett reduction won't be misaligned by one bit. We could + // use "(p64 << 1) | 1" instead of "p64 << 1" too but it makes + // no difference as that bit won't affect the relevant output bits + // (we only care about the lowest 64 bits of the result, that is, + // lowest in the reversed bit order). + // + // NOTE: The 65rd bit of p64 gets shifted out. It needs to be + // compensated with 64-bit shift and xor in the CRC64 code. + printf("const __m128i mu_p = _mm_set_epi64x(" + "0x%016" PRIx64 ", 0x%016" PRIx64 ");\n", + (calc_cldiv(p64) << 1) | 1, + p64 << 1); + + puts(""); + + puts("// CRC32"); + + printf("const __m128i fold512 = _mm_set_epi64x(" + "0x%08" PRIx64 ", 0x%08" PRIx64 ");\n", + calc_clrem(p32, 4 * 128 - 64), + calc_clrem(p32, 4 * 128)); + + printf("const __m128i fold128 = _mm_set_epi64x(" + "0x%08" PRIx64 ", 0x%08" PRIx64 ");\n", + calc_clrem(p32, 128 - 64), + calc_clrem(p32, 128)); + + // CRC32 calculation is done by modulus scaling it to a CRC64. + // Since the CRC is in reversed representation, only the mu + // constant changes with the modulus scaling. This method avoids + // one additional constant and one additional clmul in the final + // reduction steps, making the code both simpler and faster. + // + // p32 is shifted left by one so that the final multiplication + // in Barrett reduction won't be misaligned by one bit. We could + // use "(p32 << 1) | 1" instead of "p32 << 1" too but it makes + // no difference as that bit won't affect the relevant output bits. + // + // NOTE: The 33-bit value fits in 64 bits so, unlike with CRC64, + // there is no need to compensate for any missing bits in the code. + printf("const __m128i mu_p = _mm_set_epi64x(" + "0x%016" PRIx64 ", 0x%" PRIx64 ");\n", + (calc_cldiv(p32) << 1) | 1, + p32 << 1); + + return 0; +} diff --git a/src/liblzma/check/crc_common.h b/src/liblzma/check/crc_common.h new file mode 100644 index 0000000000..7ea1e60b04 --- /dev/null +++ b/src/liblzma/check/crc_common.h @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc_common.h +/// \brief Macros and declarations for CRC32 and CRC64 +// +// Authors: Lasse Collin +// Ilya Kurdyukov +// Jia Tan +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_CRC_COMMON_H +#define LZMA_CRC_COMMON_H + +#include "common.h" + + +///////////// +// Generic // +///////////// + +#ifdef WORDS_BIGENDIAN +# define A(x) ((x) >> 24) +# define B(x) (((x) >> 16) & 0xFF) +# define C(x) (((x) >> 8) & 0xFF) +# define D(x) ((x) & 0xFF) + +# define S8(x) ((x) << 8) +# define S32(x) ((x) << 32) + +#else +# define A(x) ((x) & 0xFF) +# define B(x) (((x) >> 8) & 0xFF) +# define C(x) (((x) >> 16) & 0xFF) +# define D(x) ((x) >> 24) + +# define S8(x) ((x) >> 8) +# define S32(x) ((x) >> 32) +#endif + + +/// lzma_crc32_table[0] is needed by LZ encoder so we need to keep +/// the array two-dimensional. +#ifdef HAVE_SMALL +lzma_attr_visibility_hidden +extern uint32_t lzma_crc32_table[1][256]; + +extern void lzma_crc32_init(void); + +#else + +lzma_attr_visibility_hidden +extern const uint32_t lzma_crc32_table[8][256]; + +lzma_attr_visibility_hidden +extern const uint64_t lzma_crc64_table[4][256]; +#endif + + +/////////////////// +// Configuration // +/////////////////// + +// NOTE: This config isn't used if HAVE_SMALL is defined! + +// These are defined if the generic slicing-by-n implementations and their +// lookup tables are built. +#undef CRC32_GENERIC +#undef CRC64_GENERIC + +// These are defined if an arch-specific version is built. If both this +// and matching _GENERIC is defined then runtime detection must be used. +#undef CRC32_ARCH_OPTIMIZED +#undef CRC64_ARCH_OPTIMIZED + +// The x86 CLMUL is used for both CRC32 and CRC64. +#undef CRC_X86_CLMUL + +// Many ARM64 processor have CRC32 instructions. +// CRC64 could be done with CLMUL but it's not implemented yet. +#undef CRC32_ARM64 + +// 64-bit LoongArch has CRC32 instructions. +#undef CRC32_LOONGARCH + + +// ARM64 +// +// Keep this in sync with changes to crc32_arm64.h +#if defined(_WIN32) || defined(HAVE_GETAUXVAL) \ + || defined(HAVE_ELF_AUX_INFO) \ + || (defined(__APPLE__) && defined(HAVE_SYSCTLBYNAME)) +# define CRC_ARM64_RUNTIME_DETECTION 1 +#endif + +// ARM64 CRC32 instruction is only useful for CRC32. Currently, only +// little endian is supported since we were unable to test on a big +// endian machine. +#if defined(HAVE_ARM64_CRC32) && !defined(WORDS_BIGENDIAN) + // Allow ARM64 CRC32 instruction without a runtime check if + // __ARM_FEATURE_CRC32 is defined. GCC and Clang only define + // this if the proper compiler options are used. +# if defined(__ARM_FEATURE_CRC32) +# define CRC32_ARCH_OPTIMIZED 1 +# define CRC32_ARM64 1 +# elif defined(CRC_ARM64_RUNTIME_DETECTION) +# define CRC32_ARCH_OPTIMIZED 1 +# define CRC32_ARM64 1 +# define CRC32_GENERIC 1 +# endif +#endif + + +// LoongArch +// +// Only 64-bit LoongArch is supported for now. No runtime detection +// is needed because the LoongArch specification says that the CRC32 +// instructions are a part of the Basic Integer Instructions and +// they shall be implemented by 64-bit LoongArch implementations. +#ifdef HAVE_LOONGARCH_CRC32 +# define CRC32_ARCH_OPTIMIZED 1 +# define CRC32_LOONGARCH 1 +#endif + + +// x86 and E2K +#if defined(HAVE_USABLE_CLMUL) + // If CLMUL is allowed unconditionally in the compiler options then + // the generic version and the tables can be omitted. Exceptions: + // + // - If 32-bit x86 assembly files are enabled then those are always + // built and runtime detection is used even if compiler flags + // were set to allow CLMUL unconditionally. + // + // - This doesn't work with MSVC as I don't know how to detect + // the features here. + // +# if (defined(__SSSE3__) && defined(__SSE4_1__) && defined(__PCLMUL__) \ + && !defined(HAVE_CRC_X86_ASM)) \ + || (defined(__e2k__) && __iset__ >= 6) +# define CRC32_ARCH_OPTIMIZED 1 +# define CRC64_ARCH_OPTIMIZED 1 +# define CRC_X86_CLMUL 1 +# else +# define CRC32_GENERIC 1 +# define CRC64_GENERIC 1 +# define CRC32_ARCH_OPTIMIZED 1 +# define CRC64_ARCH_OPTIMIZED 1 +# define CRC_X86_CLMUL 1 +# endif +#endif + + +// Fallback configuration +// +// For CRC32 use the generic slice-by-eight implementation if no optimized +// version is available. +#if !defined(CRC32_ARCH_OPTIMIZED) && !defined(CRC32_GENERIC) +# define CRC32_GENERIC 1 +#endif + +// For CRC64 use the generic slice-by-four implementation if no optimized +// version is available. +#if !defined(CRC64_ARCH_OPTIMIZED) && !defined(CRC64_GENERIC) +# define CRC64_GENERIC 1 +#endif + +#endif diff --git a/src/liblzma/check/crc_x86_clmul.h b/src/liblzma/check/crc_x86_clmul.h new file mode 100644 index 0000000000..b302d6cf7f --- /dev/null +++ b/src/liblzma/check/crc_x86_clmul.h @@ -0,0 +1,377 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file crc_x86_clmul.h +/// \brief CRC32 and CRC64 implementations using CLMUL instructions. +/// +/// The CRC32 and CRC64 implementations use 32/64-bit x86 SSSE3, SSE4.1, and +/// CLMUL instructions. This is compatible with Elbrus 2000 (E2K) too. +/// +/// See the Intel white paper "Fast CRC Computation for Generic Polynomials +/// Using PCLMULQDQ Instruction" from 2009. The original file seems to be +/// gone from Intel's website but a version is available here: +/// https://www.researchgate.net/publication/263424619_Fast_CRC_computation +/// (The link was checked on 2024-06-11.) +/// +/// While this file has both CRC32 and CRC64 implementations, only one +/// can be built at a time. The version to build is selected by defining +/// BUILDING_CRC_CLMUL to 32 or 64 before including this file. +/// +/// NOTE: The x86 CLMUL CRC implementation was rewritten for XZ Utils 5.8.0. +// +// Authors: Lasse Collin +// Ilya Kurdyukov +// +/////////////////////////////////////////////////////////////////////////////// + +// This file must not be included more than once. +#ifdef LZMA_CRC_X86_CLMUL_H +# error crc_x86_clmul.h was included twice. +#endif +#define LZMA_CRC_X86_CLMUL_H + +#if BUILDING_CRC_CLMUL != 32 && BUILDING_CRC_CLMUL != 64 +# error BUILDING_CRC_CLMUL is undefined or has an invalid value +#endif + +#include + +#if defined(_MSC_VER) +# include +#elif defined(HAVE_CPUID_H) +# include +#endif + + +// EDG-based compilers (Intel's classic compiler and compiler for E2K) can +// define __GNUC__ but the attribute must not be used with them. +// The new Clang-based ICX needs the attribute. +// +// NOTE: Build systems check for this too, keep them in sync with this. +#if (defined(__GNUC__) || defined(__clang__)) && !defined(__EDG__) +# define crc_attr_target \ + __attribute__((__target__("ssse3,sse4.1,pclmul"))) +#else +# define crc_attr_target +#endif + + +// GCC and Clang would produce good code with _mm_set_epi64x +// but MSVC needs _mm_cvtsi64_si128 on x86-64. +#if defined(__i386__) || defined(_M_IX86) +# define my_set_low64(a) _mm_set_epi64x(0, (a)) +#else +# define my_set_low64(a) _mm_cvtsi64_si128(a) +#endif + + +// Align it so that the whole array is within the same cache line. +// More than one unaligned load can be done from this during the +// same CRC function call. +// +// The bytes [0] to [31] are used with AND to clear the low bytes. (With ANDN +// those could be used to clear the high bytes too but it's not needed here.) +// +// The bytes [16] to [47] are for left shifts. +// The bytes [32] to [63] are for right shifts. +alignas(64) +static uint8_t vmasks[64] = { + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, +}; + + +// *Unaligned* 128-bit load +crc_attr_target +static inline __m128i +my_load128(const uint8_t *p) +{ + return _mm_loadu_si128((const __m128i *)p); +} + + +// Keep the highest "count" bytes as is and clear the remaining low bytes. +crc_attr_target +static inline __m128i +keep_high_bytes(__m128i v, size_t count) +{ + return _mm_and_si128(my_load128((vmasks + count)), v); +} + + +// Shift the 128-bit value left by "amount" bytes (not bits). +crc_attr_target +static inline __m128i +shift_left(__m128i v, size_t amount) +{ + return _mm_shuffle_epi8(v, my_load128((vmasks + 32 - amount))); +} + + +// Shift the 128-bit value right by "amount" bytes (not bits). +crc_attr_target +static inline __m128i +shift_right(__m128i v, size_t amount) +{ + return _mm_shuffle_epi8(v, my_load128((vmasks + 32 + amount))); +} + + +crc_attr_target +static inline __m128i +fold(__m128i v, __m128i k) +{ + __m128i a = _mm_clmulepi64_si128(v, k, 0x00); + __m128i b = _mm_clmulepi64_si128(v, k, 0x11); + return _mm_xor_si128(a, b); +} + + +crc_attr_target +static inline __m128i +fold_xor(__m128i v, __m128i k, const uint8_t *buf) +{ + return _mm_xor_si128(my_load128(buf), fold(v, k)); +} + + +#if BUILDING_CRC_CLMUL == 32 +crc_attr_target +static uint32_t +crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc) +#else +crc_attr_target +static uint64_t +crc64_arch_optimized(const uint8_t *buf, size_t size, uint64_t crc) +#endif +{ + // We will assume that there is at least one byte of input. + if (size == 0) + return crc; + + // See crc_clmul_consts_gen.c. +#if BUILDING_CRC_CLMUL == 32 + const __m128i fold512 = _mm_set_epi64x(0x1d9513d7, 0x8f352d95); + const __m128i fold128 = _mm_set_epi64x(0xccaa009e, 0xae689191); + const __m128i mu_p = _mm_set_epi64x( + (int64_t)0xb4e5b025f7011641, 0x1db710640); +#else + const __m128i fold512 = _mm_set_epi64x( + (int64_t)0x081f6054a7842df4, (int64_t)0x6ae3efbb9dd441f3); + + const __m128i fold128 = _mm_set_epi64x( + (int64_t)0xdabe95afc7875f40, (int64_t)0xe05dd497ca393ae4); + + const __m128i mu_p = _mm_set_epi64x( + (int64_t)0x9c3e466c172963d5, (int64_t)0x92d8af2baf0e1e84); +#endif + + __m128i v0, v1, v2, v3; + + crc = ~crc; + + if (size < 8) { + uint64_t x = crc; + size_t i = 0; + + // Checking the bit instead of comparing the size means + // that we don't need to update the size between the steps. + if (size & 4) { + x ^= read32le(buf); + buf += 4; + i = 32; + } + + if (size & 2) { + x ^= (uint64_t)read16le(buf) << i; + buf += 2; + i += 16; + } + + if (size & 1) + x ^= (uint64_t)*buf << i; + + v0 = my_set_low64((int64_t)x); + v0 = shift_left(v0, 8 - size); + + } else if (size < 16) { + v0 = my_set_low64((int64_t)(crc ^ read64le(buf))); + + // NOTE: buf is intentionally left 8 bytes behind so that + // we can read the last 1-7 bytes with read64le(buf + size). + size -= 8; + + // Handling 8-byte input specially is a speed optimization + // as the clmul can be skipped. A branch is also needed to + // avoid a too high shift amount. + if (size > 0) { + const size_t padding = 8 - size; + uint64_t high = read64le(buf + size) >> (padding * 8); + +#if defined(__i386__) || defined(_M_IX86) + // Simple but likely not the best code for 32-bit x86. + v0 = _mm_insert_epi32(v0, (int32_t)high, 2); + v0 = _mm_insert_epi32(v0, (int32_t)(high >> 32), 3); +#else + v0 = _mm_insert_epi64(v0, (int64_t)high, 1); +#endif + + v0 = shift_left(v0, padding); + + v1 = _mm_srli_si128(v0, 8); + v0 = _mm_clmulepi64_si128(v0, fold128, 0x10); + v0 = _mm_xor_si128(v0, v1); + } + } else { + v0 = my_set_low64((int64_t)crc); + + // To align or not to align the buf pointer? If the end of + // the buffer isn't aligned, aligning the pointer here would + // make us do an extra folding step with the associated byte + // shuffling overhead. The cost of that would need to be + // lower than the benefit of aligned reads. Testing on an old + // Intel Ivy Bridge processor suggested that aligning isn't + // worth the cost but it likely depends on the processor and + // buffer size. Unaligned loads (MOVDQU) should be fast on + // x86 processors that support PCLMULQDQ, so we don't align + // the buf pointer here. + + // Read the first (and possibly the only) full 16 bytes. + v0 = _mm_xor_si128(v0, my_load128(buf)); + buf += 16; + size -= 16; + + if (size >= 48) { + v1 = my_load128(buf); + v2 = my_load128(buf + 16); + v3 = my_load128(buf + 32); + buf += 48; + size -= 48; + + while (size >= 64) { + v0 = fold_xor(v0, fold512, buf); + v1 = fold_xor(v1, fold512, buf + 16); + v2 = fold_xor(v2, fold512, buf + 32); + v3 = fold_xor(v3, fold512, buf + 48); + buf += 64; + size -= 64; + } + + v0 = _mm_xor_si128(v1, fold(v0, fold128)); + v0 = _mm_xor_si128(v2, fold(v0, fold128)); + v0 = _mm_xor_si128(v3, fold(v0, fold128)); + } + + while (size >= 16) { + v0 = fold_xor(v0, fold128, buf); + buf += 16; + size -= 16; + } + + if (size > 0) { + // We want the last "size" number of input bytes to + // be at the high bits of v1. First do a full 16-byte + // load and then mask the low bytes to zeros. + v1 = my_load128(buf + size - 16); + v1 = keep_high_bytes(v1, size); + + // Shift high bytes from v0 to the low bytes of v1. + // + // Alternatively we could replace the combination + // keep_high_bytes + shift_right + _mm_or_si128 with + // _mm_shuffle_epi8 + _mm_blendv_epi8 but that would + // require larger tables for the masks. Now there are + // three loads (instead of two) from the mask tables + // but they all are from the same cache line. + v1 = _mm_or_si128(v1, shift_right(v0, size)); + + // Shift high bytes of v0 away, padding the + // low bytes with zeros. + v0 = shift_left(v0, 16 - size); + + v0 = _mm_xor_si128(v1, fold(v0, fold128)); + } + + v1 = _mm_srli_si128(v0, 8); + v0 = _mm_clmulepi64_si128(v0, fold128, 0x10); + v0 = _mm_xor_si128(v0, v1); + } + + // Barrett reduction + +#if BUILDING_CRC_CLMUL == 32 + v1 = _mm_clmulepi64_si128(v0, mu_p, 0x10); // v0 * mu + v1 = _mm_clmulepi64_si128(v1, mu_p, 0x00); // v1 * p + v0 = _mm_xor_si128(v0, v1); + return ~(uint32_t)_mm_extract_epi32(v0, 2); +#else + // Because p is 65 bits but one bit doesn't fit into the 64-bit + // half of __m128i, finish the second clmul by shifting v1 left + // by 64 bits and xorring it to the final result. + v1 = _mm_clmulepi64_si128(v0, mu_p, 0x10); // v0 * mu + v2 = _mm_slli_si128(v1, 8); + v1 = _mm_clmulepi64_si128(v1, mu_p, 0x00); // v1 * p + v0 = _mm_xor_si128(v0, v2); + v0 = _mm_xor_si128(v0, v1); +#if defined(__i386__) || defined(_M_IX86) + return ~(((uint64_t)(uint32_t)_mm_extract_epi32(v0, 3) << 32) | + (uint64_t)(uint32_t)_mm_extract_epi32(v0, 2)); +#else + return ~(uint64_t)_mm_extract_epi64(v0, 1); +#endif +#endif +} + + +// Even though this is an inline function, compile it only when needed. +// This way it won't appear in E2K builds at all. +#if defined(CRC32_GENERIC) || defined(CRC64_GENERIC) +// Inlining this function duplicates the function body in crc32_resolve() and +// crc64_resolve(), but this is acceptable because this is a tiny function. +static inline bool +is_arch_extension_supported(void) +{ + int success = 1; + uint32_t r[4]; // eax, ebx, ecx, edx + +#if defined(_MSC_VER) + // This needs with MSVC. ICC has it as a built-in + // on all platforms. + __cpuid(r, 1); +#elif defined(HAVE_CPUID_H) + // Compared to just using __asm__ to run CPUID, this also checks + // that CPUID is supported and saves and restores ebx as that is + // needed with GCC < 5 with position-independent code (PIC). + success = __get_cpuid(1, &r[0], &r[1], &r[2], &r[3]); +#else + // Just a fallback that shouldn't be needed. + __asm__("cpuid\n\t" + : "=a"(r[0]), "=b"(r[1]), "=c"(r[2]), "=d"(r[3]) + : "a"(1), "c"(0)); +#endif + + // Returns true if these are supported: + // CLMUL (bit 1 in ecx) + // SSSE3 (bit 9 in ecx) + // SSE4.1 (bit 19 in ecx) + const uint32_t ecx_mask = (1 << 1) | (1 << 9) | (1 << 19); + return success && (r[2] & ecx_mask) == ecx_mask; + + // Alternative methods that weren't used: + // - ICC's _may_i_use_cpu_feature: the other methods should work too. + // - GCC >= 6 / Clang / ICX __builtin_cpu_supports("pclmul") + // + // CPUID decoding is needed with MSVC anyway and older GCC. This keeps + // the feature checks in the build system simpler too. The nice thing + // about __builtin_cpu_supports would be that it generates very short + // code as is it only reads a variable set at startup but a few bytes + // doesn't matter here. +} +#endif diff --git a/src/liblzma/check/sha256.c b/src/liblzma/check/sha256.c new file mode 100644 index 0000000000..c067a3a693 --- /dev/null +++ b/src/liblzma/check/sha256.c @@ -0,0 +1,189 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file sha256.c +/// \brief SHA-256 +// +// The C code is based on the public domain SHA-256 code found from +// Crypto++ Library 5.5.1 released in 2007: https://www.cryptopp.com/ +// A few minor tweaks have been made in liblzma. +// +// Authors: Wei Dai +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "check.h" + +// Rotate a uint32_t. GCC can optimize this to a rotate instruction +// at least on x86. +static inline uint32_t +rotr_32(uint32_t num, unsigned amount) +{ + return (num >> amount) | (num << (32 - amount)); +} + +#define blk0(i) (W[i] = conv32be(data[i])) +#define blk2(i) (W[i & 15] += s1(W[(i - 2) & 15]) + W[(i - 7) & 15] \ + + s0(W[(i - 15) & 15])) + +#define Ch(x, y, z) (z ^ (x & (y ^ z))) +#define Maj(x, y, z) ((x & (y ^ z)) + (y & z)) + +#define a(i) T[(0 - i) & 7] +#define b(i) T[(1 - i) & 7] +#define c(i) T[(2 - i) & 7] +#define d(i) T[(3 - i) & 7] +#define e(i) T[(4 - i) & 7] +#define f(i) T[(5 - i) & 7] +#define g(i) T[(6 - i) & 7] +#define h(i) T[(7 - i) & 7] + +#define R(i, j, blk) \ + h(i) += S1(e(i)) + Ch(e(i), f(i), g(i)) + SHA256_K[i + j] + blk; \ + d(i) += h(i); \ + h(i) += S0(a(i)) + Maj(a(i), b(i), c(i)) +#define R0(i) R(i, 0, blk0(i)) +#define R2(i) R(i, j, blk2(i)) + +#define S0(x) rotr_32(x ^ rotr_32(x ^ rotr_32(x, 9), 11), 2) +#define S1(x) rotr_32(x ^ rotr_32(x ^ rotr_32(x, 14), 5), 6) +#define s0(x) (rotr_32(x ^ rotr_32(x, 11), 7) ^ (x >> 3)) +#define s1(x) (rotr_32(x ^ rotr_32(x, 2), 17) ^ (x >> 10)) + + +static const uint32_t SHA256_K[64] = { + 0x428A2F98, 0x71374491, 0xB5C0FBCF, 0xE9B5DBA5, + 0x3956C25B, 0x59F111F1, 0x923F82A4, 0xAB1C5ED5, + 0xD807AA98, 0x12835B01, 0x243185BE, 0x550C7DC3, + 0x72BE5D74, 0x80DEB1FE, 0x9BDC06A7, 0xC19BF174, + 0xE49B69C1, 0xEFBE4786, 0x0FC19DC6, 0x240CA1CC, + 0x2DE92C6F, 0x4A7484AA, 0x5CB0A9DC, 0x76F988DA, + 0x983E5152, 0xA831C66D, 0xB00327C8, 0xBF597FC7, + 0xC6E00BF3, 0xD5A79147, 0x06CA6351, 0x14292967, + 0x27B70A85, 0x2E1B2138, 0x4D2C6DFC, 0x53380D13, + 0x650A7354, 0x766A0ABB, 0x81C2C92E, 0x92722C85, + 0xA2BFE8A1, 0xA81A664B, 0xC24B8B70, 0xC76C51A3, + 0xD192E819, 0xD6990624, 0xF40E3585, 0x106AA070, + 0x19A4C116, 0x1E376C08, 0x2748774C, 0x34B0BCB5, + 0x391C0CB3, 0x4ED8AA4A, 0x5B9CCA4F, 0x682E6FF3, + 0x748F82EE, 0x78A5636F, 0x84C87814, 0x8CC70208, + 0x90BEFFFA, 0xA4506CEB, 0xBEF9A3F7, 0xC67178F2, +}; + + +static void +transform(uint32_t state[8], const uint32_t data[16]) +{ + uint32_t W[16]; + uint32_t T[8]; + + // Copy state[] to working vars. + memcpy(T, state, sizeof(T)); + + // The first 16 operations unrolled + R0( 0); R0( 1); R0( 2); R0( 3); + R0( 4); R0( 5); R0( 6); R0( 7); + R0( 8); R0( 9); R0(10); R0(11); + R0(12); R0(13); R0(14); R0(15); + + // The remaining 48 operations partially unrolled + for (unsigned int j = 16; j < 64; j += 16) { + R2( 0); R2( 1); R2( 2); R2( 3); + R2( 4); R2( 5); R2( 6); R2( 7); + R2( 8); R2( 9); R2(10); R2(11); + R2(12); R2(13); R2(14); R2(15); + } + + // Add the working vars back into state[]. + state[0] += a(0); + state[1] += b(0); + state[2] += c(0); + state[3] += d(0); + state[4] += e(0); + state[5] += f(0); + state[6] += g(0); + state[7] += h(0); +} + + +static void +process(lzma_check_state *check) +{ + transform(check->state.sha256.state, check->buffer.u32); + return; +} + + +extern void +lzma_sha256_init(lzma_check_state *check) +{ + static const uint32_t s[8] = { + 0x6A09E667, 0xBB67AE85, 0x3C6EF372, 0xA54FF53A, + 0x510E527F, 0x9B05688C, 0x1F83D9AB, 0x5BE0CD19, + }; + + memcpy(check->state.sha256.state, s, sizeof(s)); + check->state.sha256.size = 0; + + return; +} + + +extern void +lzma_sha256_update(const uint8_t *buf, size_t size, lzma_check_state *check) +{ + // Copy the input data into a properly aligned temporary buffer. + // This way we can be called with arbitrarily sized buffers + // (no need to be multiple of 64 bytes), and the code works also + // on architectures that don't allow unaligned memory access. + while (size > 0) { + const size_t copy_start = check->state.sha256.size & 0x3F; + size_t copy_size = 64 - copy_start; + if (copy_size > size) + copy_size = size; + + memcpy(check->buffer.u8 + copy_start, buf, copy_size); + + buf += copy_size; + size -= copy_size; + check->state.sha256.size += copy_size; + + if ((check->state.sha256.size & 0x3F) == 0) + process(check); + } + + return; +} + + +extern void +lzma_sha256_finish(lzma_check_state *check) +{ + // Add padding as described in RFC 3174 (it describes SHA-1 but + // the same padding style is used for SHA-256 too). + size_t pos = check->state.sha256.size & 0x3F; + check->buffer.u8[pos++] = 0x80; + + while (pos != 64 - 8) { + if (pos == 64) { + process(check); + pos = 0; + } + + check->buffer.u8[pos++] = 0x00; + } + + // Convert the message size from bytes to bits. + check->state.sha256.size *= 8; + + check->buffer.u64[(64 - 8) / 8] = conv64be(check->state.sha256.size); + + process(check); + + for (size_t i = 0; i < 8; ++i) + check->buffer.u32[i] = conv32be(check->state.sha256.state[i]); + + return; +} diff --git a/src/liblzma/common/Makefile.inc b/src/liblzma/common/Makefile.inc new file mode 100644 index 0000000000..51b1aae86e --- /dev/null +++ b/src/liblzma/common/Makefile.inc @@ -0,0 +1,100 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +liblzma_la_SOURCES += \ + common/common.c \ + common/common.h \ + common/memcmplen.h \ + common/block_util.c \ + common/easy_preset.c \ + common/easy_preset.h \ + common/filter_common.c \ + common/filter_common.h \ + common/hardware_physmem.c \ + common/index.c \ + common/index.h \ + common/stream_flags_common.c \ + common/stream_flags_common.h \ + common/string_conversion.c \ + common/vli_size.c + +if COND_THREADS +liblzma_la_SOURCES += \ + common/hardware_cputhreads.c \ + common/outqueue.c \ + common/outqueue.h +endif + +if COND_MAIN_ENCODER +liblzma_la_SOURCES += \ + common/alone_encoder.c \ + common/block_buffer_encoder.c \ + common/block_buffer_encoder.h \ + common/block_encoder.c \ + common/block_encoder.h \ + common/block_header_encoder.c \ + common/easy_buffer_encoder.c \ + common/easy_encoder.c \ + common/easy_encoder_memusage.c \ + common/filter_buffer_encoder.c \ + common/filter_encoder.c \ + common/filter_encoder.h \ + common/filter_flags_encoder.c \ + common/index_encoder.c \ + common/index_encoder.h \ + common/stream_buffer_encoder.c \ + common/stream_encoder.c \ + common/stream_flags_encoder.c \ + common/vli_encoder.c + +if COND_THREADS +liblzma_la_SOURCES += \ + common/stream_encoder_mt.c +endif + +if COND_MICROLZMA +liblzma_la_SOURCES += \ + common/microlzma_encoder.c +endif +endif + +if COND_MAIN_DECODER +liblzma_la_SOURCES += \ + common/alone_decoder.c \ + common/alone_decoder.h \ + common/auto_decoder.c \ + common/block_buffer_decoder.c \ + common/block_decoder.c \ + common/block_decoder.h \ + common/block_header_decoder.c \ + common/easy_decoder_memusage.c \ + common/file_info.c \ + common/filter_buffer_decoder.c \ + common/filter_decoder.c \ + common/filter_decoder.h \ + common/filter_flags_decoder.c \ + common/index_decoder.c \ + common/index_decoder.h \ + common/index_hash.c \ + common/stream_buffer_decoder.c \ + common/stream_decoder.c \ + common/stream_decoder.h \ + common/stream_flags_decoder.c \ + common/vli_decoder.c + +if COND_THREADS +liblzma_la_SOURCES += \ + common/stream_decoder_mt.c +endif + +if COND_MICROLZMA +liblzma_la_SOURCES += \ + common/microlzma_decoder.c +endif + +if COND_LZIP_DECODER +liblzma_la_SOURCES += \ + common/lzip_decoder.c \ + common/lzip_decoder.h +endif +endif diff --git a/src/liblzma/common/alone_decoder.c b/src/liblzma/common/alone_decoder.c new file mode 100644 index 0000000000..e2b58e1f37 --- /dev/null +++ b/src/liblzma/common/alone_decoder.c @@ -0,0 +1,247 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file alone_decoder.c +/// \brief Decoder for LZMA_Alone files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "alone_decoder.h" +#include "lzma_decoder.h" +#include "lz_decoder.h" + + +typedef struct { + lzma_next_coder next; + + enum { + SEQ_PROPERTIES, + SEQ_DICTIONARY_SIZE, + SEQ_UNCOMPRESSED_SIZE, + SEQ_CODER_INIT, + SEQ_CODE, + } sequence; + + /// If true, reject files that are unlikely to be .lzma files. + /// If false, more non-.lzma files get accepted and will give + /// LZMA_DATA_ERROR either immediately or after a few output bytes. + bool picky; + + /// Position in the header fields + size_t pos; + + /// Uncompressed size decoded from the header + lzma_vli uncompressed_size; + + /// Memory usage limit + uint64_t memlimit; + + /// Amount of memory actually needed (only an estimate) + uint64_t memusage; + + /// Options decoded from the header needed to initialize + /// the LZMA decoder + lzma_options_lzma options; +} lzma_alone_coder; + + +static lzma_ret +alone_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, + lzma_action action) +{ + lzma_alone_coder *coder = coder_ptr; + + while (*out_pos < out_size + && (coder->sequence == SEQ_CODE || *in_pos < in_size)) + switch (coder->sequence) { + case SEQ_PROPERTIES: + if (lzma_lzma_lclppb_decode(&coder->options, in[*in_pos])) + return LZMA_FORMAT_ERROR; + + coder->sequence = SEQ_DICTIONARY_SIZE; + ++*in_pos; + break; + + case SEQ_DICTIONARY_SIZE: + coder->options.dict_size + |= (size_t)(in[*in_pos]) << (coder->pos * 8); + + if (++coder->pos == 4) { + if (coder->picky && coder->options.dict_size + != UINT32_MAX) { + // A hack to ditch tons of false positives: + // We allow only dictionary sizes that are + // 2^n or 2^n + 2^(n-1). LZMA_Alone created + // only files with 2^n, but accepts any + // dictionary size. + uint32_t d = coder->options.dict_size - 1; + d |= d >> 2; + d |= d >> 3; + d |= d >> 4; + d |= d >> 8; + d |= d >> 16; + ++d; + + if (d != coder->options.dict_size) + return LZMA_FORMAT_ERROR; + } + + coder->pos = 0; + coder->sequence = SEQ_UNCOMPRESSED_SIZE; + } + + ++*in_pos; + break; + + case SEQ_UNCOMPRESSED_SIZE: + coder->uncompressed_size + |= (lzma_vli)(in[*in_pos]) << (coder->pos * 8); + ++*in_pos; + if (++coder->pos < 8) + break; + + // Another hack to ditch false positives: Assume that + // if the uncompressed size is known, it must be less + // than 256 GiB. + // + // FIXME? Without picky we allow > LZMA_VLI_MAX which doesn't + // really matter in this specific situation (> LZMA_VLI_MAX is + // safe in the LZMA decoder) but it's somewhat weird still. + if (coder->picky + && coder->uncompressed_size != LZMA_VLI_UNKNOWN + && coder->uncompressed_size + >= (LZMA_VLI_C(1) << 38)) + return LZMA_FORMAT_ERROR; + + // Use LZMA_FILTER_LZMA1EXT features to specify the + // uncompressed size and that the end marker is allowed + // even when the uncompressed size is known. Both .lzma + // header and LZMA1EXT use UINT64_MAX indicate that size + // is unknown. + coder->options.ext_flags = LZMA_LZMA1EXT_ALLOW_EOPM; + lzma_set_ext_size(coder->options, coder->uncompressed_size); + + // Calculate the memory usage so that it is ready + // for SEQ_CODER_INIT. + coder->memusage = lzma_lzma_decoder_memusage(&coder->options) + + LZMA_MEMUSAGE_BASE; + + coder->pos = 0; + coder->sequence = SEQ_CODER_INIT; + FALLTHROUGH; + + case SEQ_CODER_INIT: { + if (coder->memusage > coder->memlimit) + return LZMA_MEMLIMIT_ERROR; + + lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_decoder_init, + .options = &coder->options, + }, { + .init = NULL, + } + }; + + return_if_error(lzma_next_filter_init(&coder->next, + allocator, filters)); + + coder->sequence = SEQ_CODE; + break; + } + + case SEQ_CODE: { + return coder->next.code(coder->next.coder, + allocator, in, in_pos, in_size, + out, out_pos, out_size, action); + } + + default: + return LZMA_PROG_ERROR; + } + + return LZMA_OK; +} + + +static void +alone_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_alone_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +alone_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_alone_coder *coder = coder_ptr; + + *memusage = coder->memusage; + *old_memlimit = coder->memlimit; + + if (new_memlimit != 0) { + if (new_memlimit < coder->memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +extern lzma_ret +lzma_alone_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, bool picky) +{ + lzma_next_coder_init(&lzma_alone_decoder_init, next, allocator); + + lzma_alone_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_alone_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &alone_decode; + next->end = &alone_decoder_end; + next->memconfig = &alone_decoder_memconfig; + coder->next = LZMA_NEXT_CODER_INIT; + } + + coder->sequence = SEQ_PROPERTIES; + coder->picky = picky; + coder->pos = 0; + coder->options.dict_size = 0; + coder->options.preset_dict = NULL; + coder->options.preset_dict_size = 0; + coder->uncompressed_size = 0; + coder->memlimit = my_max(1, memlimit); + coder->memusage = LZMA_MEMUSAGE_BASE; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_alone_decoder(lzma_stream *strm, uint64_t memlimit) +{ + lzma_next_strm_init(lzma_alone_decoder_init, strm, memlimit, false); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/alone_decoder.h b/src/liblzma/common/alone_decoder.h new file mode 100644 index 0000000000..61ee24d97f --- /dev/null +++ b/src/liblzma/common/alone_decoder.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file alone_decoder.h +/// \brief Decoder for LZMA_Alone files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_ALONE_DECODER_H +#define LZMA_ALONE_DECODER_H + +#include "common.h" + + +extern lzma_ret lzma_alone_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, bool picky); + +#endif diff --git a/src/liblzma/common/alone_encoder.c b/src/liblzma/common/alone_encoder.c new file mode 100644 index 0000000000..21b039509a --- /dev/null +++ b/src/liblzma/common/alone_encoder.c @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file alone_encoder.c +/// \brief Encoder for LZMA_Alone files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "lzma_encoder.h" + + +#define ALONE_HEADER_SIZE (1 + 4 + 8) + + +typedef struct { + lzma_next_coder next; + + enum { + SEQ_HEADER, + SEQ_CODE, + } sequence; + + size_t header_pos; + uint8_t header[ALONE_HEADER_SIZE]; +} lzma_alone_coder; + + +static lzma_ret +alone_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, + lzma_action action) +{ + lzma_alone_coder *coder = coder_ptr; + + while (*out_pos < out_size) + switch (coder->sequence) { + case SEQ_HEADER: + lzma_bufcpy(coder->header, &coder->header_pos, + ALONE_HEADER_SIZE, + out, out_pos, out_size); + if (coder->header_pos < ALONE_HEADER_SIZE) + return LZMA_OK; + + coder->sequence = SEQ_CODE; + break; + + case SEQ_CODE: + return coder->next.code(coder->next.coder, + allocator, in, in_pos, in_size, + out, out_pos, out_size, action); + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + return LZMA_OK; +} + + +static void +alone_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_alone_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +alone_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_options_lzma *options) +{ + lzma_next_coder_init(&alone_encoder_init, next, allocator); + + lzma_alone_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_alone_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &alone_encode; + next->end = &alone_encoder_end; + coder->next = LZMA_NEXT_CODER_INIT; + } + + // Basic initializations + coder->sequence = SEQ_HEADER; + coder->header_pos = 0; + + // Encode the header: + // - Properties (1 byte) + if (lzma_lzma_lclppb_encode(options, coder->header)) + return LZMA_OPTIONS_ERROR; + + // - Dictionary size (4 bytes) + if (options->dict_size < LZMA_DICT_SIZE_MIN) + return LZMA_OPTIONS_ERROR; + + // Round up to the next 2^n or 2^n + 2^(n - 1) depending on which + // one is the next unless it is UINT32_MAX. While the header would + // allow any 32-bit integer, we do this to keep the decoder of liblzma + // accepting the resulting files. + uint32_t d = options->dict_size - 1; + d |= d >> 2; + d |= d >> 3; + d |= d >> 4; + d |= d >> 8; + d |= d >> 16; + if (d != UINT32_MAX) + ++d; + + write32le(coder->header + 1, d); + + // - Uncompressed size (always unknown and using EOPM) + memset(coder->header + 1 + 4, 0xFF, 8); + + // Initialize the LZMA encoder. + const lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_encoder_init, + .options = (void *)(options), + }, { + .init = NULL, + } + }; + + return lzma_next_filter_init(&coder->next, allocator, filters); +} + + +extern LZMA_API(lzma_ret) +lzma_alone_encoder(lzma_stream *strm, const lzma_options_lzma *options) +{ + lzma_next_strm_init(alone_encoder_init, strm, options); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/auto_decoder.c b/src/liblzma/common/auto_decoder.c new file mode 100644 index 0000000000..da49345f90 --- /dev/null +++ b/src/liblzma/common/auto_decoder.c @@ -0,0 +1,204 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file auto_decoder.c +/// \brief Autodetect between .xz, .lzma (LZMA_Alone), and .lz (lzip) +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "stream_decoder.h" +#include "alone_decoder.h" +#ifdef HAVE_LZIP_DECODER +# include "lzip_decoder.h" +#endif + + +typedef struct { + /// .xz Stream decoder, LZMA_Alone decoder, or lzip decoder + lzma_next_coder next; + + uint64_t memlimit; + uint32_t flags; + + enum { + SEQ_INIT, + SEQ_CODE, + SEQ_FINISH, + } sequence; +} lzma_auto_coder; + + +static lzma_ret +auto_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_auto_coder *coder = coder_ptr; + + switch (coder->sequence) { + case SEQ_INIT: + if (*in_pos >= in_size) + return LZMA_OK; + + // Update the sequence now, because we want to continue from + // SEQ_CODE even if we return some LZMA_*_CHECK. + coder->sequence = SEQ_CODE; + + // Detect the file format. .xz files start with 0xFD which + // cannot be the first byte of .lzma (LZMA_Alone) format. + // The .lz format starts with 0x4C which could be the + // first byte of a .lzma file but luckily it would mean + // lc/lp/pb being 4/3/1 which liblzma doesn't support because + // lc + lp > 4. So using just 0x4C to detect .lz is OK here. + if (in[*in_pos] == 0xFD) { + return_if_error(lzma_stream_decoder_init( + &coder->next, allocator, + coder->memlimit, coder->flags)); +#ifdef HAVE_LZIP_DECODER + } else if (in[*in_pos] == 0x4C) { + return_if_error(lzma_lzip_decoder_init( + &coder->next, allocator, + coder->memlimit, coder->flags)); +#endif + } else { + return_if_error(lzma_alone_decoder_init(&coder->next, + allocator, coder->memlimit, true)); + + // If the application wants to know about missing + // integrity check or about the check in general, we + // need to handle it here, because LZMA_Alone decoder + // doesn't accept any flags. + if (coder->flags & LZMA_TELL_NO_CHECK) + return LZMA_NO_CHECK; + + if (coder->flags & LZMA_TELL_ANY_CHECK) + return LZMA_GET_CHECK; + } + + FALLTHROUGH; + + case SEQ_CODE: { + const lzma_ret ret = coder->next.code( + coder->next.coder, allocator, + in, in_pos, in_size, + out, out_pos, out_size, action); + if (ret != LZMA_STREAM_END + || (coder->flags & LZMA_CONCATENATED) == 0) + return ret; + + coder->sequence = SEQ_FINISH; + FALLTHROUGH; + } + + case SEQ_FINISH: + // When LZMA_CONCATENATED was used and we were decoding + // a LZMA_Alone file, we need to check that there is no + // trailing garbage and wait for LZMA_FINISH. + if (*in_pos < in_size) + return LZMA_DATA_ERROR; + + return action == LZMA_FINISH ? LZMA_STREAM_END : LZMA_OK; + + default: + assert(0); + return LZMA_PROG_ERROR; + } +} + + +static void +auto_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_auto_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_check +auto_decoder_get_check(const void *coder_ptr) +{ + const lzma_auto_coder *coder = coder_ptr; + + // It is LZMA_Alone if get_check is NULL. + return coder->next.get_check == NULL ? LZMA_CHECK_NONE + : coder->next.get_check(coder->next.coder); +} + + +static lzma_ret +auto_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_auto_coder *coder = coder_ptr; + + lzma_ret ret; + + if (coder->next.memconfig != NULL) { + ret = coder->next.memconfig(coder->next.coder, + memusage, old_memlimit, new_memlimit); + assert(*old_memlimit == coder->memlimit); + } else { + // No coder is configured yet. Use the base value as + // the current memory usage. + *memusage = LZMA_MEMUSAGE_BASE; + *old_memlimit = coder->memlimit; + + ret = LZMA_OK; + if (new_memlimit != 0 && new_memlimit < *memusage) + ret = LZMA_MEMLIMIT_ERROR; + } + + if (ret == LZMA_OK && new_memlimit != 0) + coder->memlimit = new_memlimit; + + return ret; +} + + +static lzma_ret +auto_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags) +{ + lzma_next_coder_init(&auto_decoder_init, next, allocator); + + if (flags & ~LZMA_SUPPORTED_FLAGS) + return LZMA_OPTIONS_ERROR; + + lzma_auto_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_auto_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &auto_decode; + next->end = &auto_decoder_end; + next->get_check = &auto_decoder_get_check; + next->memconfig = &auto_decoder_memconfig; + coder->next = LZMA_NEXT_CODER_INIT; + } + + coder->memlimit = my_max(1, memlimit); + coder->flags = flags; + coder->sequence = SEQ_INIT; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_auto_decoder(lzma_stream *strm, uint64_t memlimit, uint32_t flags) +{ + lzma_next_strm_init(auto_decoder_init, strm, memlimit, flags); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/block_buffer_decoder.c b/src/liblzma/common/block_buffer_decoder.c new file mode 100644 index 0000000000..55566cd2f2 --- /dev/null +++ b/src/liblzma/common/block_buffer_decoder.c @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_buffer_decoder.c +/// \brief Single-call .xz Block decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "block_decoder.h" + + +extern LZMA_API(lzma_ret) +lzma_block_buffer_decode(lzma_block *block, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + if (in_pos == NULL || (in == NULL && *in_pos != in_size) + || *in_pos > in_size || out_pos == NULL + || (out == NULL && *out_pos != out_size) + || *out_pos > out_size) + return LZMA_PROG_ERROR; + + // Initialize the Block decoder. + lzma_next_coder block_decoder = LZMA_NEXT_CODER_INIT; + lzma_ret ret = lzma_block_decoder_init( + &block_decoder, allocator, block); + + if (ret == LZMA_OK) { + // Save the positions so that we can restore them in case + // an error occurs. + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + // Do the actual decoding. + ret = block_decoder.code(block_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + LZMA_FINISH); + + if (ret == LZMA_STREAM_END) { + ret = LZMA_OK; + } else { + if (ret == LZMA_OK) { + // Either the input was truncated or the + // output buffer was too small. + assert(*in_pos == in_size + || *out_pos == out_size); + + // If all the input was consumed, then the + // input is truncated, even if the output + // buffer is also full. This is because + // processing the last byte of the Block + // never produces output. + // + // NOTE: This assumption may break when new + // filters are added, if the end marker of + // the filter doesn't consume at least one + // complete byte. + if (*in_pos == in_size) + ret = LZMA_DATA_ERROR; + else + ret = LZMA_BUF_ERROR; + } + + // Restore the positions. + *in_pos = in_start; + *out_pos = out_start; + } + } + + // Free the decoder memory. This needs to be done even if + // initialization fails, because the internal API doesn't + // require the initialization function to free its memory on error. + lzma_next_end(&block_decoder, allocator); + + return ret; +} diff --git a/src/liblzma/common/block_buffer_encoder.c b/src/liblzma/common/block_buffer_encoder.c new file mode 100644 index 0000000000..df3b90e8a1 --- /dev/null +++ b/src/liblzma/common/block_buffer_encoder.c @@ -0,0 +1,354 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_buffer_encoder.c +/// \brief Single-call .xz Block encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "block_buffer_encoder.h" +#include "block_encoder.h" +#include "filter_encoder.h" +#include "lzma2_encoder.h" +#include "check.h" + + +/// Estimate the maximum size of the Block Header and Check fields for +/// a Block that uses LZMA2 uncompressed chunks. We could use +/// lzma_block_header_size() but this is simpler. +/// +/// Block Header Size + Block Flags + Compressed Size +/// + Uncompressed Size + Filter Flags for LZMA2 + CRC32 + Check +/// and round up to the next multiple of four to take Header Padding +/// into account. +#define HEADERS_BOUND ((1 + 1 + 2 * LZMA_VLI_BYTES_MAX + 3 + 4 \ + + LZMA_CHECK_SIZE_MAX + 3) & ~3) + + +static uint64_t +lzma2_bound(uint64_t uncompressed_size) +{ + // Prevent integer overflow in overhead calculation. + if (uncompressed_size > COMPRESSED_SIZE_MAX) + return 0; + + // Calculate the exact overhead of the LZMA2 headers: Round + // uncompressed_size up to the next multiple of LZMA2_CHUNK_MAX, + // multiply by the size of per-chunk header, and add one byte for + // the end marker. + const uint64_t overhead = ((uncompressed_size + LZMA2_CHUNK_MAX - 1) + / LZMA2_CHUNK_MAX) + * LZMA2_HEADER_UNCOMPRESSED + 1; + + // Catch the possible integer overflow. + if (COMPRESSED_SIZE_MAX - overhead < uncompressed_size) + return 0; + + return uncompressed_size + overhead; +} + + +extern uint64_t +lzma_block_buffer_bound64(uint64_t uncompressed_size) +{ + // If the data doesn't compress, we always use uncompressed + // LZMA2 chunks. + uint64_t lzma2_size = lzma2_bound(uncompressed_size); + if (lzma2_size == 0) + return 0; + + // Take Block Padding into account. + lzma2_size = (lzma2_size + 3) & ~UINT64_C(3); + + // No risk of integer overflow because lzma2_bound() already takes + // into account the size of the headers in the Block. + return HEADERS_BOUND + lzma2_size; +} + + +extern LZMA_API(size_t) +lzma_block_buffer_bound(size_t uncompressed_size) +{ + uint64_t ret = lzma_block_buffer_bound64(uncompressed_size); + +#if SIZE_MAX < UINT64_MAX + // Catch the possible integer overflow on 32-bit systems. + if (ret > SIZE_MAX) + return 0; +#endif + + return ret; +} + + +static lzma_ret +block_encode_uncompressed(lzma_block *block, const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Use LZMA2 uncompressed chunks. We wouldn't need a dictionary at + // all, but LZMA2 always requires a dictionary, so use the minimum + // value to minimize memory usage of the decoder. + lzma_options_lzma lzma2 = { + .dict_size = LZMA_DICT_SIZE_MIN, + }; + + lzma_filter filters[2]; + filters[0].id = LZMA_FILTER_LZMA2; + filters[0].options = &lzma2; + filters[1].id = LZMA_VLI_UNKNOWN; + + // Set the above filter options to *block temporarily so that we can + // encode the Block Header. + lzma_filter *filters_orig = block->filters; + block->filters = filters; + + if (lzma_block_header_size(block) != LZMA_OK) { + block->filters = filters_orig; + return LZMA_PROG_ERROR; + } + + // Check that there's enough output space. The caller has already + // set block->compressed_size to what lzma2_bound() has returned, + // so we can reuse that value. We know that compressed_size is a + // known valid VLI and header_size is a small value so their sum + // will never overflow. + assert(block->compressed_size == lzma2_bound(in_size)); + if (out_size - *out_pos + < block->header_size + block->compressed_size) { + block->filters = filters_orig; + return LZMA_BUF_ERROR; + } + + if (lzma_block_header_encode(block, out + *out_pos) != LZMA_OK) { + block->filters = filters_orig; + return LZMA_PROG_ERROR; + } + + block->filters = filters_orig; + *out_pos += block->header_size; + + // Encode the data using LZMA2 uncompressed chunks. + size_t in_pos = 0; + uint8_t control = 0x01; // Dictionary reset + + while (in_pos < in_size) { + // Control byte: Indicate uncompressed chunk, of which + // the first resets the dictionary. + out[(*out_pos)++] = control; + control = 0x02; // No dictionary reset + + // Size of the uncompressed chunk + const size_t copy_size + = my_min(in_size - in_pos, LZMA2_CHUNK_MAX); + out[(*out_pos)++] = (copy_size - 1) >> 8; + out[(*out_pos)++] = (copy_size - 1) & 0xFF; + + // The actual data + assert(*out_pos + copy_size <= out_size); + memcpy(out + *out_pos, in + in_pos, copy_size); + + in_pos += copy_size; + *out_pos += copy_size; + } + + // End marker + out[(*out_pos)++] = 0x00; + assert(*out_pos <= out_size); + + return LZMA_OK; +} + + +static lzma_ret +block_encode_normal(lzma_block *block, const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Find out the size of the Block Header. + return_if_error(lzma_block_header_size(block)); + + // Reserve space for the Block Header and skip it for now. + if (out_size - *out_pos <= block->header_size) + return LZMA_BUF_ERROR; + + const size_t out_start = *out_pos; + *out_pos += block->header_size; + + // Limit out_size so that we stop encoding if the output would grow + // bigger than what uncompressed Block would be. + if (out_size - *out_pos > block->compressed_size) + out_size = *out_pos + block->compressed_size; + + // TODO: In many common cases this could be optimized to use + // significantly less memory. + lzma_next_coder raw_encoder = LZMA_NEXT_CODER_INIT; + lzma_ret ret = lzma_raw_encoder_init( + &raw_encoder, allocator, block->filters); + + if (ret == LZMA_OK) { + size_t in_pos = 0; + ret = raw_encoder.code(raw_encoder.coder, allocator, + in, &in_pos, in_size, out, out_pos, out_size, + LZMA_FINISH); + } + + // NOTE: This needs to be run even if lzma_raw_encoder_init() failed. + lzma_next_end(&raw_encoder, allocator); + + if (ret == LZMA_STREAM_END) { + // Compression was successful. Write the Block Header. + block->compressed_size + = *out_pos - (out_start + block->header_size); + ret = lzma_block_header_encode(block, out + out_start); + if (ret != LZMA_OK) + ret = LZMA_PROG_ERROR; + + } else if (ret == LZMA_OK) { + // Output buffer became full. + ret = LZMA_BUF_ERROR; + } + + // Reset *out_pos if something went wrong. + if (ret != LZMA_OK) + *out_pos = out_start; + + return ret; +} + + +static lzma_ret +block_buffer_encode(lzma_block *block, const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size, + bool try_to_compress) +{ + // Validate the arguments. + if (block == NULL || (in == NULL && in_size != 0) || out == NULL + || out_pos == NULL || *out_pos > out_size) + return LZMA_PROG_ERROR; + + // The contents of the structure may depend on the version so + // check the version before validating the contents of *block. + if (block->version > 1) + return LZMA_OPTIONS_ERROR; + + if ((unsigned int)(block->check) > LZMA_CHECK_ID_MAX + || (try_to_compress && block->filters == NULL)) + return LZMA_PROG_ERROR; + + if (!lzma_check_is_supported(block->check)) + return LZMA_UNSUPPORTED_CHECK; + + // Size of a Block has to be a multiple of four, so limit the size + // here already. This way we don't need to check it again when adding + // Block Padding. + out_size -= (out_size - *out_pos) & 3; + + // Get the size of the Check field. + const size_t check_size = lzma_check_size(block->check); + assert(check_size != UINT32_MAX); + + // Reserve space for the Check field. + if (out_size - *out_pos <= check_size) + return LZMA_BUF_ERROR; + + out_size -= check_size; + + // Initialize block->uncompressed_size and calculate the worst-case + // value for block->compressed_size. + block->uncompressed_size = in_size; + block->compressed_size = lzma2_bound(in_size); + if (block->compressed_size == 0) + return LZMA_DATA_ERROR; + + // Do the actual compression. + lzma_ret ret = LZMA_BUF_ERROR; + if (try_to_compress) + ret = block_encode_normal(block, allocator, + in, in_size, out, out_pos, out_size); + + if (ret != LZMA_OK) { + // If the error was something else than output buffer + // becoming full, return the error now. + if (ret != LZMA_BUF_ERROR) + return ret; + + // The data was incompressible (at least with the options + // given to us) or the output buffer was too small. Use the + // uncompressed chunks of LZMA2 to wrap the data into a valid + // Block. If we haven't been given enough output space, even + // this may fail. + return_if_error(block_encode_uncompressed(block, in, in_size, + out, out_pos, out_size)); + } + + assert(*out_pos <= out_size); + + // Block Padding. No buffer overflow here, because we already adjusted + // out_size so that (out_size - out_start) is a multiple of four. + // Thus, if the buffer is full, the loop body can never run. + for (size_t i = (size_t)(block->compressed_size); i & 3; ++i) { + assert(*out_pos < out_size); + out[(*out_pos)++] = 0x00; + } + + // If there's no Check field, we are done now. + if (check_size > 0) { + // Calculate the integrity check. We reserved space for + // the Check field earlier so we don't need to check for + // available output space here. + lzma_check_state check; + lzma_check_init(&check, block->check); + lzma_check_update(&check, block->check, in, in_size); + lzma_check_finish(&check, block->check); + + memcpy(block->raw_check, check.buffer.u8, check_size); + memcpy(out + *out_pos, check.buffer.u8, check_size); + *out_pos += check_size; + } + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_block_buffer_encode(lzma_block *block, const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + return block_buffer_encode(block, allocator, + in, in_size, out, out_pos, out_size, true); +} + + +#ifdef HAVE_SYMBOL_VERSIONS_LINUX +// This is for compatibility with binaries linked against liblzma that +// has been patched with xz-5.2.2-compat-libs.patch from RHEL/CentOS 7. +LZMA_SYMVER_API("lzma_block_uncomp_encode@XZ_5.2.2", + lzma_ret, lzma_block_uncomp_encode_522)(lzma_block *block, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result + __attribute__((__alias__("lzma_block_uncomp_encode_52"))); + +LZMA_SYMVER_API("lzma_block_uncomp_encode@@XZ_5.2", + lzma_ret, lzma_block_uncomp_encode_52)(lzma_block *block, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) + lzma_nothrow lzma_attr_warn_unused_result; + +#define lzma_block_uncomp_encode lzma_block_uncomp_encode_52 +#endif +extern LZMA_API(lzma_ret) +lzma_block_uncomp_encode(lzma_block *block, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // It won't allocate any memory from heap so no need + // for lzma_allocator. + return block_buffer_encode(block, NULL, + in, in_size, out, out_pos, out_size, false); +} diff --git a/src/liblzma/common/block_buffer_encoder.h b/src/liblzma/common/block_buffer_encoder.h new file mode 100644 index 0000000000..5274ac40d3 --- /dev/null +++ b/src/liblzma/common/block_buffer_encoder.h @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_buffer_encoder.h +/// \brief Single-call .xz Block encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_BLOCK_BUFFER_ENCODER_H +#define LZMA_BLOCK_BUFFER_ENCODER_H + +#include "common.h" + + +/// uint64_t version of lzma_block_buffer_bound(). It is used by +/// stream_encoder_mt.c. Probably the original lzma_block_buffer_bound() +/// should have been 64-bit, but fixing it would break the ABI. +extern uint64_t lzma_block_buffer_bound64(uint64_t uncompressed_size); + +#endif diff --git a/src/liblzma/common/block_decoder.c b/src/liblzma/common/block_decoder.c new file mode 100644 index 0000000000..bbc9f5566c --- /dev/null +++ b/src/liblzma/common/block_decoder.c @@ -0,0 +1,286 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_decoder.c +/// \brief Decodes .xz Blocks +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "block_decoder.h" +#include "filter_decoder.h" +#include "check.h" + + +typedef struct { + enum { + SEQ_CODE, + SEQ_PADDING, + SEQ_CHECK, + } sequence; + + /// The filters in the chain; initialized with lzma_raw_decoder_init(). + lzma_next_coder next; + + /// Decoding options; we also write Compressed Size and Uncompressed + /// Size back to this structure when the decoding has been finished. + lzma_block *block; + + /// Compressed Size calculated while decoding + lzma_vli compressed_size; + + /// Uncompressed Size calculated while decoding + lzma_vli uncompressed_size; + + /// Maximum allowed Compressed Size; this takes into account the + /// size of the Block Header and Check fields when Compressed Size + /// is unknown. + lzma_vli compressed_limit; + + /// Maximum allowed Uncompressed Size. + lzma_vli uncompressed_limit; + + /// Position when reading the Check field + size_t check_pos; + + /// Check of the uncompressed data + lzma_check_state check; + + /// True if the integrity check won't be calculated and verified. + bool ignore_check; +} lzma_block_coder; + + +static inline bool +is_size_valid(lzma_vli size, lzma_vli reference) +{ + return reference == LZMA_VLI_UNKNOWN || reference == size; +} + + +static lzma_ret +block_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_block_coder *coder = coder_ptr; + + switch (coder->sequence) { + case SEQ_CODE: { + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + // Limit the amount of input and output space that we give + // to the raw decoder based on the information we have + // (or don't have) from Block Header. + const size_t in_stop = *in_pos + (size_t)my_min( + in_size - *in_pos, + coder->compressed_limit - coder->compressed_size); + const size_t out_stop = *out_pos + (size_t)my_min( + out_size - *out_pos, + coder->uncompressed_limit - coder->uncompressed_size); + + const lzma_ret ret = coder->next.code(coder->next.coder, + allocator, in, in_pos, in_stop, + out, out_pos, out_stop, action); + + const size_t in_used = *in_pos - in_start; + const size_t out_used = *out_pos - out_start; + + // Because we have limited the input and output sizes, + // we know that these cannot grow too big or overflow. + coder->compressed_size += in_used; + coder->uncompressed_size += out_used; + + if (ret == LZMA_OK) { + const bool comp_done = coder->compressed_size + == coder->block->compressed_size; + const bool uncomp_done = coder->uncompressed_size + == coder->block->uncompressed_size; + + // If both input and output amounts match the sizes + // in Block Header but we still got LZMA_OK instead + // of LZMA_STREAM_END, the file is broken. + if (comp_done && uncomp_done) + return LZMA_DATA_ERROR; + + // If the decoder has consumed all the input that it + // needs but it still couldn't fill the output buffer + // or return LZMA_STREAM_END, the file is broken. + if (comp_done && *out_pos < out_size) + return LZMA_DATA_ERROR; + + // If the decoder has produced all the output but + // it still didn't return LZMA_STREAM_END or consume + // more input (for example, detecting an end of + // payload marker may need more input but produce + // no output) the file is broken. + if (uncomp_done && *in_pos < in_size) + return LZMA_DATA_ERROR; + } + + // Don't waste time updating the integrity check if it will be + // ignored. Also skip it if no new output was produced. This + // avoids null pointer + 0 (undefined behavior) when out == 0. + if (!coder->ignore_check && out_used > 0) + lzma_check_update(&coder->check, coder->block->check, + out + out_start, out_used); + + if (ret != LZMA_STREAM_END) + return ret; + + // Compressed and Uncompressed Sizes are now at their final + // values. Verify that they match the values given to us. + if (!is_size_valid(coder->compressed_size, + coder->block->compressed_size) + || !is_size_valid(coder->uncompressed_size, + coder->block->uncompressed_size)) + return LZMA_DATA_ERROR; + + // Copy the values into coder->block. The caller + // may use this information to construct Index. + coder->block->compressed_size = coder->compressed_size; + coder->block->uncompressed_size = coder->uncompressed_size; + + coder->sequence = SEQ_PADDING; + FALLTHROUGH; + } + + case SEQ_PADDING: + // Compressed Data is padded to a multiple of four bytes. + while (coder->compressed_size & 3) { + if (*in_pos >= in_size) + return LZMA_OK; + + // We use compressed_size here just get the Padding + // right. The actual Compressed Size was stored to + // coder->block already, and won't be modified by + // us anymore. + ++coder->compressed_size; + + if (in[(*in_pos)++] != 0x00) + return LZMA_DATA_ERROR; + } + + if (coder->block->check == LZMA_CHECK_NONE) + return LZMA_STREAM_END; + + if (!coder->ignore_check) + lzma_check_finish(&coder->check, coder->block->check); + + coder->sequence = SEQ_CHECK; + FALLTHROUGH; + + case SEQ_CHECK: { + const size_t check_size = lzma_check_size(coder->block->check); + lzma_bufcpy(in, in_pos, in_size, coder->block->raw_check, + &coder->check_pos, check_size); + if (coder->check_pos < check_size) + return LZMA_OK; + + // Validate the Check only if we support it. + // coder->check.buffer may be uninitialized + // when the Check ID is not supported. + if (!coder->ignore_check + && lzma_check_is_supported(coder->block->check) + && memcmp(coder->block->raw_check, + coder->check.buffer.u8, + check_size) != 0) + return LZMA_DATA_ERROR; + + return LZMA_STREAM_END; + } + } + + return LZMA_PROG_ERROR; +} + + +static void +block_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_block_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder, allocator); + return; +} + + +extern lzma_ret +lzma_block_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + lzma_block *block) +{ + lzma_next_coder_init(&lzma_block_decoder_init, next, allocator); + + // Validate the options. lzma_block_unpadded_size() does that for us + // except for Uncompressed Size and filters. Filters are validated + // by the raw decoder. + if (lzma_block_unpadded_size(block) == 0 + || !lzma_vli_is_valid(block->uncompressed_size)) + return LZMA_PROG_ERROR; + + // Allocate *next->coder if needed. + lzma_block_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_block_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &block_decode; + next->end = &block_decoder_end; + coder->next = LZMA_NEXT_CODER_INIT; + } + + // Basic initializations + coder->sequence = SEQ_CODE; + coder->block = block; + coder->compressed_size = 0; + coder->uncompressed_size = 0; + + // If Compressed Size is not known, we calculate the maximum allowed + // value so that encoded size of the Block (including Block Padding) + // is still a valid VLI and a multiple of four. + coder->compressed_limit + = block->compressed_size == LZMA_VLI_UNKNOWN + ? (LZMA_VLI_MAX & ~LZMA_VLI_C(3)) + - block->header_size + - lzma_check_size(block->check) + : block->compressed_size; + + // With Uncompressed Size this is simpler. If Block Header lacks + // the size info, then LZMA_VLI_MAX is the maximum possible + // Uncompressed Size. + coder->uncompressed_limit + = block->uncompressed_size == LZMA_VLI_UNKNOWN + ? LZMA_VLI_MAX + : block->uncompressed_size; + + // Initialize the check. It's caller's problem if the Check ID is not + // supported, and the Block decoder cannot verify the Check field. + // Caller can test lzma_check_is_supported(block->check). + coder->check_pos = 0; + lzma_check_init(&coder->check, block->check); + + coder->ignore_check = block->version >= 1 + ? block->ignore_check : false; + + // Initialize the filter chain. + return lzma_raw_decoder_init(&coder->next, allocator, + block->filters); +} + + +extern LZMA_API(lzma_ret) +lzma_block_decoder(lzma_stream *strm, lzma_block *block) +{ + lzma_next_strm_init(lzma_block_decoder_init, strm, block); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/block_decoder.h b/src/liblzma/common/block_decoder.h new file mode 100644 index 0000000000..2cbf9ba6db --- /dev/null +++ b/src/liblzma/common/block_decoder.h @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_decoder.h +/// \brief Decodes .xz Blocks +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_BLOCK_DECODER_H +#define LZMA_BLOCK_DECODER_H + +#include "common.h" + + +extern lzma_ret lzma_block_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, lzma_block *block); + +#endif diff --git a/src/liblzma/common/block_encoder.c b/src/liblzma/common/block_encoder.c new file mode 100644 index 0000000000..eb7997a72a --- /dev/null +++ b/src/liblzma/common/block_encoder.c @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_encoder.c +/// \brief Encodes .xz Blocks +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "block_encoder.h" +#include "filter_encoder.h" +#include "check.h" + + +typedef struct { + /// The filters in the chain; initialized with lzma_raw_decoder_init(). + lzma_next_coder next; + + /// Encoding options; we also write Unpadded Size, Compressed Size, + /// and Uncompressed Size back to this structure when the encoding + /// has been finished. + lzma_block *block; + + enum { + SEQ_CODE, + SEQ_PADDING, + SEQ_CHECK, + } sequence; + + /// Compressed Size calculated while encoding + lzma_vli compressed_size; + + /// Uncompressed Size calculated while encoding + lzma_vli uncompressed_size; + + /// Position in the Check field + size_t pos; + + /// Check of the uncompressed data + lzma_check_state check; +} lzma_block_coder; + + +static lzma_ret +block_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_block_coder *coder = coder_ptr; + + // Check that our amount of input stays in proper limits. + if (LZMA_VLI_MAX - coder->uncompressed_size < in_size - *in_pos) + return LZMA_DATA_ERROR; + + switch (coder->sequence) { + case SEQ_CODE: { + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + const lzma_ret ret = coder->next.code(coder->next.coder, + allocator, in, in_pos, in_size, + out, out_pos, out_size, action); + + const size_t in_used = *in_pos - in_start; + const size_t out_used = *out_pos - out_start; + + if (COMPRESSED_SIZE_MAX - coder->compressed_size < out_used) + return LZMA_DATA_ERROR; + + coder->compressed_size += out_used; + + // No need to check for overflow because we have already + // checked it at the beginning of this function. + coder->uncompressed_size += in_used; + + // Call lzma_check_update() only if input was consumed. This + // avoids null pointer + 0 (undefined behavior) when in == 0. + if (in_used > 0) + lzma_check_update(&coder->check, coder->block->check, + in + in_start, in_used); + + if (ret != LZMA_STREAM_END || action == LZMA_SYNC_FLUSH) + return ret; + + assert(*in_pos == in_size); + assert(action == LZMA_FINISH); + + // Copy the values into coder->block. The caller + // may use this information to construct Index. + coder->block->compressed_size = coder->compressed_size; + coder->block->uncompressed_size = coder->uncompressed_size; + + coder->sequence = SEQ_PADDING; + FALLTHROUGH; + } + + case SEQ_PADDING: + // Pad Compressed Data to a multiple of four bytes. We can + // use coder->compressed_size for this since we don't need + // it for anything else anymore. + while (coder->compressed_size & 3) { + if (*out_pos >= out_size) + return LZMA_OK; + + out[*out_pos] = 0x00; + ++*out_pos; + ++coder->compressed_size; + } + + if (coder->block->check == LZMA_CHECK_NONE) + return LZMA_STREAM_END; + + lzma_check_finish(&coder->check, coder->block->check); + + coder->sequence = SEQ_CHECK; + FALLTHROUGH; + + case SEQ_CHECK: { + const size_t check_size = lzma_check_size(coder->block->check); + lzma_bufcpy(coder->check.buffer.u8, &coder->pos, check_size, + out, out_pos, out_size); + if (coder->pos < check_size) + return LZMA_OK; + + memcpy(coder->block->raw_check, coder->check.buffer.u8, + check_size); + return LZMA_STREAM_END; + } + } + + return LZMA_PROG_ERROR; +} + + +static void +block_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_block_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +block_encoder_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters lzma_attribute((__unused__)), + const lzma_filter *reversed_filters) +{ + lzma_block_coder *coder = coder_ptr; + + if (coder->sequence != SEQ_CODE) + return LZMA_PROG_ERROR; + + return lzma_next_filter_update( + &coder->next, allocator, reversed_filters); +} + + +extern lzma_ret +lzma_block_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + lzma_block *block) +{ + lzma_next_coder_init(&lzma_block_encoder_init, next, allocator); + + if (block == NULL) + return LZMA_PROG_ERROR; + + // The contents of the structure may depend on the version so + // check the version first. + if (block->version > 1) + return LZMA_OPTIONS_ERROR; + + // If the Check ID is not supported, we cannot calculate the check and + // thus not create a proper Block. + if ((unsigned int)(block->check) > LZMA_CHECK_ID_MAX) + return LZMA_PROG_ERROR; + + if (!lzma_check_is_supported(block->check)) + return LZMA_UNSUPPORTED_CHECK; + + // Allocate and initialize *next->coder if needed. + lzma_block_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_block_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &block_encode; + next->end = &block_encoder_end; + next->update = &block_encoder_update; + coder->next = LZMA_NEXT_CODER_INIT; + } + + // Basic initializations + coder->sequence = SEQ_CODE; + coder->block = block; + coder->compressed_size = 0; + coder->uncompressed_size = 0; + coder->pos = 0; + + // Initialize the check + lzma_check_init(&coder->check, block->check); + + // Initialize the requested filters. + return lzma_raw_encoder_init(&coder->next, allocator, block->filters); +} + + +extern LZMA_API(lzma_ret) +lzma_block_encoder(lzma_stream *strm, lzma_block *block) +{ + lzma_next_strm_init(lzma_block_encoder_init, strm, block); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_SYNC_FLUSH] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/block_encoder.h b/src/liblzma/common/block_encoder.h new file mode 100644 index 0000000000..b7dfe9a084 --- /dev/null +++ b/src/liblzma/common/block_encoder.h @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_encoder.h +/// \brief Encodes .xz Blocks +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_BLOCK_ENCODER_H +#define LZMA_BLOCK_ENCODER_H + +#include "common.h" + + +/// \brief Biggest Compressed Size value that the Block encoder supports +/// +/// The maximum size of a single Block is limited by the maximum size of +/// a Stream, which in theory is 2^63 - 3 bytes (i.e. LZMA_VLI_MAX - 3). +/// While the size is really big and no one should hit it in practice, we +/// take it into account in some places anyway to catch some errors e.g. if +/// application passes insanely big value to some function. +/// +/// We could take into account the headers etc. to determine the exact +/// maximum size of the Compressed Data field, but the complexity would give +/// us nothing useful. Instead, limit the size of Compressed Data so that +/// even with biggest possible Block Header and Check fields the total +/// encoded size of the Block stays as a valid VLI. This doesn't guarantee +/// that the size of the Stream doesn't grow too big, but that problem is +/// taken care outside the Block handling code. +/// +/// ~LZMA_VLI_C(3) is to guarantee that if we need padding at the end of +/// the Compressed Data field, it will still stay in the proper limit. +/// +/// This constant is in this file because it is needed in both +/// block_encoder.c and block_buffer_encoder.c. +#define COMPRESSED_SIZE_MAX ((LZMA_VLI_MAX - LZMA_BLOCK_HEADER_SIZE_MAX \ + - LZMA_CHECK_SIZE_MAX) & ~LZMA_VLI_C(3)) + + +extern lzma_ret lzma_block_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, lzma_block *block); + +#endif diff --git a/src/liblzma/common/block_header_decoder.c b/src/liblzma/common/block_header_decoder.c new file mode 100644 index 0000000000..f0b2fbe54d --- /dev/null +++ b/src/liblzma/common/block_header_decoder.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_header_decoder.c +/// \brief Decodes Block Header from .xz files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "check.h" + + +extern LZMA_API(lzma_ret) +lzma_block_header_decode(lzma_block *block, + const lzma_allocator *allocator, const uint8_t *in) +{ + // NOTE: We consider the header to be corrupt not only when the + // CRC32 doesn't match, but also when variable-length integers + // are invalid or over 63 bits, or if the header is too small + // to contain the claimed information. + + // Catch unexpected NULL pointers. + if (block == NULL || block->filters == NULL || in == NULL) + return LZMA_PROG_ERROR; + + // Initialize the filter options array. This way the caller can + // safely free() the options even if an error occurs in this function. + for (size_t i = 0; i <= LZMA_FILTERS_MAX; ++i) { + block->filters[i].id = LZMA_VLI_UNKNOWN; + block->filters[i].options = NULL; + } + + // Versions 0 and 1 are supported. If a newer version was specified, + // we need to downgrade it. + if (block->version > 1) + block->version = 1; + + // This isn't a Block Header option, but since the decompressor will + // read it if version >= 1, it's better to initialize it here than + // to expect the caller to do it since in almost all cases this + // should be false. + block->ignore_check = false; + + // Validate Block Header Size and Check type. The caller must have + // already set these, so it is a programming error if this test fails. + if (lzma_block_header_size_decode(in[0]) != block->header_size + || (unsigned int)(block->check) > LZMA_CHECK_ID_MAX) + return LZMA_PROG_ERROR; + + // Exclude the CRC32 field. + const size_t in_size = block->header_size - 4; + + // Verify CRC32 + if (lzma_crc32(in, in_size, 0) != read32le(in + in_size)) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + return LZMA_DATA_ERROR; +#endif + } + + // Check for unsupported flags. + if (in[1] & 0x3C) + return LZMA_OPTIONS_ERROR; + + // Start after the Block Header Size and Block Flags fields. + size_t in_pos = 2; + + // Compressed Size + if (in[1] & 0x40) { + return_if_error(lzma_vli_decode(&block->compressed_size, + NULL, in, &in_pos, in_size)); + + // Validate Compressed Size. This checks that it isn't zero + // and that the total size of the Block is a valid VLI. + if (lzma_block_unpadded_size(block) == 0) + return LZMA_DATA_ERROR; + } else { + block->compressed_size = LZMA_VLI_UNKNOWN; + } + + // Uncompressed Size + if (in[1] & 0x80) + return_if_error(lzma_vli_decode(&block->uncompressed_size, + NULL, in, &in_pos, in_size)); + else + block->uncompressed_size = LZMA_VLI_UNKNOWN; + + // Filter Flags + const size_t filter_count = (in[1] & 3U) + 1; + for (size_t i = 0; i < filter_count; ++i) { + const lzma_ret ret = lzma_filter_flags_decode( + &block->filters[i], allocator, + in, &in_pos, in_size); + if (ret != LZMA_OK) { + lzma_filters_free(block->filters, allocator); + return ret; + } + } + + // Padding + while (in_pos < in_size) { + if (in[in_pos++] != 0x00) { + lzma_filters_free(block->filters, allocator); + + // Possibly some new field present so use + // LZMA_OPTIONS_ERROR instead of LZMA_DATA_ERROR. + return LZMA_OPTIONS_ERROR; + } + } + + return LZMA_OK; +} diff --git a/src/liblzma/common/block_header_encoder.c b/src/liblzma/common/block_header_encoder.c new file mode 100644 index 0000000000..45e57a26ab --- /dev/null +++ b/src/liblzma/common/block_header_encoder.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_header_encoder.c +/// \brief Encodes Block Header for .xz files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "check.h" + + +extern LZMA_API(lzma_ret) +lzma_block_header_size(lzma_block *block) +{ + if (block->version > 1) + return LZMA_OPTIONS_ERROR; + + // Block Header Size + Block Flags + CRC32. + uint32_t size = 1 + 1 + 4; + + // Compressed Size + if (block->compressed_size != LZMA_VLI_UNKNOWN) { + const uint32_t add = lzma_vli_size(block->compressed_size); + if (add == 0 || block->compressed_size == 0) + return LZMA_PROG_ERROR; + + size += add; + } + + // Uncompressed Size + if (block->uncompressed_size != LZMA_VLI_UNKNOWN) { + const uint32_t add = lzma_vli_size(block->uncompressed_size); + if (add == 0) + return LZMA_PROG_ERROR; + + size += add; + } + + // List of Filter Flags + if (block->filters == NULL || block->filters[0].id == LZMA_VLI_UNKNOWN) + return LZMA_PROG_ERROR; + + for (size_t i = 0; block->filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + // Don't allow too many filters. + if (i == LZMA_FILTERS_MAX) + return LZMA_PROG_ERROR; + + uint32_t add; + return_if_error(lzma_filter_flags_size(&add, + block->filters + i)); + + size += add; + } + + // Pad to a multiple of four bytes. + block->header_size = (size + 3) & ~UINT32_C(3); + + // NOTE: We don't verify that the encoded size of the Block stays + // within limits. This is because it is possible that we are called + // with exaggerated Compressed Size (e.g. LZMA_VLI_MAX) to reserve + // space for Block Header, and later called again with lower, + // real values. + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_block_header_encode(const lzma_block *block, uint8_t *out) +{ + // Validate everything but filters. + if (lzma_block_unpadded_size(block) == 0 + || !lzma_vli_is_valid(block->uncompressed_size)) + return LZMA_PROG_ERROR; + + // Indicate the size of the buffer _excluding_ the CRC32 field. + const size_t out_size = block->header_size - 4; + + // Store the Block Header Size. + out[0] = out_size / 4; + + // We write Block Flags in pieces. + out[1] = 0x00; + size_t out_pos = 2; + + // Compressed Size + if (block->compressed_size != LZMA_VLI_UNKNOWN) { + return_if_error(lzma_vli_encode(block->compressed_size, NULL, + out, &out_pos, out_size)); + + out[1] |= 0x40; + } + + // Uncompressed Size + if (block->uncompressed_size != LZMA_VLI_UNKNOWN) { + return_if_error(lzma_vli_encode(block->uncompressed_size, NULL, + out, &out_pos, out_size)); + + out[1] |= 0x80; + } + + // Filter Flags + if (block->filters == NULL || block->filters[0].id == LZMA_VLI_UNKNOWN) + return LZMA_PROG_ERROR; + + size_t filter_count = 0; + do { + // There can be a maximum of four filters. + if (filter_count == LZMA_FILTERS_MAX) + return LZMA_PROG_ERROR; + + return_if_error(lzma_filter_flags_encode( + block->filters + filter_count, + out, &out_pos, out_size)); + + } while (block->filters[++filter_count].id != LZMA_VLI_UNKNOWN); + + out[1] |= filter_count - 1; + + // Padding + memzero(out + out_pos, out_size - out_pos); + + // CRC32 + write32le(out + out_size, lzma_crc32(out, out_size, 0)); + + return LZMA_OK; +} diff --git a/src/liblzma/common/block_util.c b/src/liblzma/common/block_util.c new file mode 100644 index 0000000000..191f6d444a --- /dev/null +++ b/src/liblzma/common/block_util.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file block_util.c +/// \brief Utility functions to handle lzma_block +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "index.h" + + +extern LZMA_API(lzma_ret) +lzma_block_compressed_size(lzma_block *block, lzma_vli unpadded_size) +{ + // Validate everything but Uncompressed Size and filters. + if (lzma_block_unpadded_size(block) == 0) + return LZMA_PROG_ERROR; + + const uint32_t container_size = block->header_size + + lzma_check_size(block->check); + + // Validate that Compressed Size will be greater than zero. + if (unpadded_size <= container_size) + return LZMA_DATA_ERROR; + + // Calculate what Compressed Size is supposed to be. + // If Compressed Size was present in Block Header, + // compare that the new value matches it. + const lzma_vli compressed_size = unpadded_size - container_size; + if (block->compressed_size != LZMA_VLI_UNKNOWN + && block->compressed_size != compressed_size) + return LZMA_DATA_ERROR; + + block->compressed_size = compressed_size; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_vli) +lzma_block_unpadded_size(const lzma_block *block) +{ + // Validate the values that we are interested in i.e. all but + // Uncompressed Size and the filters. + // + // NOTE: This function is used for validation too, so it is + // essential that these checks are always done even if + // Compressed Size is unknown. + if (block == NULL || block->version > 1 + || block->header_size < LZMA_BLOCK_HEADER_SIZE_MIN + || block->header_size > LZMA_BLOCK_HEADER_SIZE_MAX + || (block->header_size & 3) + || !lzma_vli_is_valid(block->compressed_size) + || block->compressed_size == 0 + || (unsigned int)(block->check) > LZMA_CHECK_ID_MAX) + return 0; + + // If Compressed Size is unknown, return that we cannot know + // size of the Block either. + if (block->compressed_size == LZMA_VLI_UNKNOWN) + return LZMA_VLI_UNKNOWN; + + // Calculate Unpadded Size and validate it. + const lzma_vli unpadded_size = block->compressed_size + + block->header_size + + lzma_check_size(block->check); + + assert(unpadded_size >= UNPADDED_SIZE_MIN); + if (unpadded_size > UNPADDED_SIZE_MAX) + return 0; + + return unpadded_size; +} + + +extern LZMA_API(lzma_vli) +lzma_block_total_size(const lzma_block *block) +{ + lzma_vli unpadded_size = lzma_block_unpadded_size(block); + + if (unpadded_size != LZMA_VLI_UNKNOWN) + unpadded_size = vli_ceil4(unpadded_size); + + return unpadded_size; +} diff --git a/src/liblzma/common/common.c b/src/liblzma/common/common.c new file mode 100644 index 0000000000..6e031a56c8 --- /dev/null +++ b/src/liblzma/common/common.c @@ -0,0 +1,486 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file common.c +/// \brief Common functions needed in many places in liblzma +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + + +///////////// +// Version // +///////////// + +extern LZMA_API(uint32_t) +lzma_version_number(void) +{ + return LZMA_VERSION; +} + + +extern LZMA_API(const char *) +lzma_version_string(void) +{ + return LZMA_VERSION_STRING; +} + + +/////////////////////// +// Memory allocation // +/////////////////////// + +lzma_attr_alloc_size(1) +extern void * +lzma_alloc(size_t size, const lzma_allocator *allocator) +{ + // Some malloc() variants return NULL if called with size == 0. + if (size == 0) + size = 1; + + void *ptr; + + if (allocator != NULL && allocator->alloc != NULL) + ptr = allocator->alloc(allocator->opaque, 1, size); + else + ptr = malloc(size); + + return ptr; +} + + +lzma_attr_alloc_size(1) +extern void * +lzma_alloc_zero(size_t size, const lzma_allocator *allocator) +{ + // Some calloc() variants return NULL if called with size == 0. + if (size == 0) + size = 1; + + void *ptr; + + if (allocator != NULL && allocator->alloc != NULL) { + ptr = allocator->alloc(allocator->opaque, 1, size); + if (ptr != NULL) + memzero(ptr, size); + } else { + ptr = calloc(1, size); + } + + return ptr; +} + + +extern void +lzma_free(void *ptr, const lzma_allocator *allocator) +{ + if (allocator != NULL && allocator->free != NULL) + allocator->free(allocator->opaque, ptr); + else + free(ptr); + + return; +} + + +////////// +// Misc // +////////// + +extern size_t +lzma_bufcpy(const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size) +{ + assert(in != NULL || *in_pos == in_size); + assert(out != NULL || *out_pos == out_size); + + assert(*in_pos <= in_size); + assert(*out_pos <= out_size); + + const size_t in_avail = in_size - *in_pos; + const size_t out_avail = out_size - *out_pos; + const size_t copy_size = my_min(in_avail, out_avail); + + // Call memcpy() only if there is something to copy. If there is + // nothing to copy, in or out might be NULL and then the memcpy() + // call would trigger undefined behavior. + if (copy_size > 0) + memcpy(out + *out_pos, in + *in_pos, copy_size); + + *in_pos += copy_size; + *out_pos += copy_size; + + return copy_size; +} + + +extern lzma_ret +lzma_next_filter_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + lzma_next_coder_init(filters[0].init, next, allocator); + next->id = filters[0].id; + return filters[0].init == NULL + ? LZMA_OK : filters[0].init(next, allocator, filters); +} + + +extern lzma_ret +lzma_next_filter_update(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *reversed_filters) +{ + // Check that the application isn't trying to change the Filter ID. + // End of filters is indicated with LZMA_VLI_UNKNOWN in both + // reversed_filters[0].id and next->id. + if (reversed_filters[0].id != next->id) + return LZMA_PROG_ERROR; + + if (reversed_filters[0].id == LZMA_VLI_UNKNOWN) + return LZMA_OK; + + assert(next->update != NULL); + return next->update(next->coder, allocator, NULL, reversed_filters); +} + + +extern void +lzma_next_end(lzma_next_coder *next, const lzma_allocator *allocator) +{ + if (next->init != (uintptr_t)(NULL)) { + // To avoid tiny end functions that simply call + // lzma_free(coder, allocator), we allow leaving next->end + // NULL and call lzma_free() here. + if (next->end != NULL) + next->end(next->coder, allocator); + else + lzma_free(next->coder, allocator); + + // Reset the variables so the we don't accidentally think + // that it is an already initialized coder. + *next = LZMA_NEXT_CODER_INIT; + } + + return; +} + + +////////////////////////////////////// +// External to internal API wrapper // +////////////////////////////////////// + +extern lzma_ret +lzma_strm_init(lzma_stream *strm) +{ + if (strm == NULL) + return LZMA_PROG_ERROR; + + if (strm->internal == NULL) { + strm->internal = lzma_alloc(sizeof(lzma_internal), + strm->allocator); + if (strm->internal == NULL) + return LZMA_MEM_ERROR; + + strm->internal->next = LZMA_NEXT_CODER_INIT; + } + + memzero(strm->internal->supported_actions, + sizeof(strm->internal->supported_actions)); + strm->internal->sequence = ISEQ_RUN; + strm->internal->allow_buf_error = false; + + strm->total_in = 0; + strm->total_out = 0; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_code(lzma_stream *strm, lzma_action action) +{ + // Sanity checks + if ((strm->next_in == NULL && strm->avail_in != 0) + || (strm->next_out == NULL && strm->avail_out != 0) + || strm->internal == NULL + || strm->internal->next.code == NULL + || (unsigned int)(action) > LZMA_ACTION_MAX + || !strm->internal->supported_actions[action]) + return LZMA_PROG_ERROR; + + // Check if unsupported members have been set to non-zero or non-NULL, + // which would indicate that some new feature is wanted. + if (strm->reserved_ptr1 != NULL + || strm->reserved_ptr2 != NULL + || strm->reserved_ptr3 != NULL + || strm->reserved_ptr4 != NULL + || strm->reserved_int2 != 0 + || strm->reserved_int3 != 0 + || strm->reserved_int4 != 0 + || strm->reserved_enum1 != LZMA_RESERVED_ENUM + || strm->reserved_enum2 != LZMA_RESERVED_ENUM) + return LZMA_OPTIONS_ERROR; + + switch (strm->internal->sequence) { + case ISEQ_RUN: + switch (action) { + case LZMA_RUN: + break; + + case LZMA_SYNC_FLUSH: + strm->internal->sequence = ISEQ_SYNC_FLUSH; + break; + + case LZMA_FULL_FLUSH: + strm->internal->sequence = ISEQ_FULL_FLUSH; + break; + + case LZMA_FINISH: + strm->internal->sequence = ISEQ_FINISH; + break; + + case LZMA_FULL_BARRIER: + strm->internal->sequence = ISEQ_FULL_BARRIER; + break; + } + + break; + + case ISEQ_SYNC_FLUSH: + // The same action must be used until we return + // LZMA_STREAM_END, and the amount of input must not change. + if (action != LZMA_SYNC_FLUSH + || strm->internal->avail_in != strm->avail_in) + return LZMA_PROG_ERROR; + + break; + + case ISEQ_FULL_FLUSH: + if (action != LZMA_FULL_FLUSH + || strm->internal->avail_in != strm->avail_in) + return LZMA_PROG_ERROR; + + break; + + case ISEQ_FINISH: + if (action != LZMA_FINISH + || strm->internal->avail_in != strm->avail_in) + return LZMA_PROG_ERROR; + + break; + + case ISEQ_FULL_BARRIER: + if (action != LZMA_FULL_BARRIER + || strm->internal->avail_in != strm->avail_in) + return LZMA_PROG_ERROR; + + break; + + case ISEQ_END: + return LZMA_STREAM_END; + + case ISEQ_ERROR: + default: + return LZMA_PROG_ERROR; + } + + size_t in_pos = 0; + size_t out_pos = 0; + lzma_ret ret = strm->internal->next.code( + strm->internal->next.coder, strm->allocator, + strm->next_in, &in_pos, strm->avail_in, + strm->next_out, &out_pos, strm->avail_out, action); + + // Updating next_in and next_out has to be skipped when they are NULL + // to avoid null pointer + 0 (undefined behavior). Do this by checking + // in_pos > 0 and out_pos > 0 because this way NULL + non-zero (a bug) + // will get caught one way or other. + if (in_pos > 0) { + strm->next_in += in_pos; + strm->avail_in -= in_pos; + strm->total_in += in_pos; + } + + if (out_pos > 0) { + strm->next_out += out_pos; + strm->avail_out -= out_pos; + strm->total_out += out_pos; + } + + strm->internal->avail_in = strm->avail_in; + + switch (ret) { + case LZMA_OK: + // Don't return LZMA_BUF_ERROR when it happens the first time. + // This is to avoid returning LZMA_BUF_ERROR when avail_out + // was zero but still there was no more data left to written + // to next_out. + if (out_pos == 0 && in_pos == 0) { + if (strm->internal->allow_buf_error) + ret = LZMA_BUF_ERROR; + else + strm->internal->allow_buf_error = true; + } else { + strm->internal->allow_buf_error = false; + } + break; + + case LZMA_TIMED_OUT: + strm->internal->allow_buf_error = false; + ret = LZMA_OK; + break; + + case LZMA_SEEK_NEEDED: + strm->internal->allow_buf_error = false; + + // If LZMA_FINISH was used, reset it back to the + // LZMA_RUN-based state so that new input can be supplied + // by the application. + if (strm->internal->sequence == ISEQ_FINISH) + strm->internal->sequence = ISEQ_RUN; + + break; + + case LZMA_STREAM_END: + if (strm->internal->sequence == ISEQ_SYNC_FLUSH + || strm->internal->sequence == ISEQ_FULL_FLUSH + || strm->internal->sequence + == ISEQ_FULL_BARRIER) + strm->internal->sequence = ISEQ_RUN; + else + strm->internal->sequence = ISEQ_END; + + FALLTHROUGH; + + case LZMA_NO_CHECK: + case LZMA_UNSUPPORTED_CHECK: + case LZMA_GET_CHECK: + case LZMA_MEMLIMIT_ERROR: + // Something else than LZMA_OK, but not a fatal error, + // that is, coding may be continued (except if ISEQ_END). + strm->internal->allow_buf_error = false; + break; + + default: + // All the other errors are fatal; coding cannot be continued. + assert(ret != LZMA_BUF_ERROR); + strm->internal->sequence = ISEQ_ERROR; + break; + } + + return ret; +} + + +extern LZMA_API(void) +lzma_end(lzma_stream *strm) +{ + if (strm != NULL && strm->internal != NULL) { + lzma_next_end(&strm->internal->next, strm->allocator); + lzma_free(strm->internal, strm->allocator); + strm->internal = NULL; + } + + return; +} + + +#ifdef HAVE_SYMBOL_VERSIONS_LINUX +// This is for compatibility with binaries linked against liblzma that +// has been patched with xz-5.2.2-compat-libs.patch from RHEL/CentOS 7. +LZMA_SYMVER_API("lzma_get_progress@XZ_5.2.2", + void, lzma_get_progress_522)(lzma_stream *strm, + uint64_t *progress_in, uint64_t *progress_out) lzma_nothrow + __attribute__((__alias__("lzma_get_progress_52"))); + +LZMA_SYMVER_API("lzma_get_progress@@XZ_5.2", + void, lzma_get_progress_52)(lzma_stream *strm, + uint64_t *progress_in, uint64_t *progress_out) lzma_nothrow; + +#define lzma_get_progress lzma_get_progress_52 +#endif +extern LZMA_API(void) +lzma_get_progress(lzma_stream *strm, + uint64_t *progress_in, uint64_t *progress_out) +{ + if (strm->internal->next.get_progress != NULL) { + strm->internal->next.get_progress(strm->internal->next.coder, + progress_in, progress_out); + } else { + *progress_in = strm->total_in; + *progress_out = strm->total_out; + } + + return; +} + + +extern LZMA_API(lzma_check) +lzma_get_check(const lzma_stream *strm) +{ + // Return LZMA_CHECK_NONE if we cannot know the check type. + // It's a bug in the application if this happens. + if (strm->internal->next.get_check == NULL) + return LZMA_CHECK_NONE; + + return strm->internal->next.get_check(strm->internal->next.coder); +} + + +extern LZMA_API(uint64_t) +lzma_memusage(const lzma_stream *strm) +{ + uint64_t memusage; + uint64_t old_memlimit; + + if (strm == NULL || strm->internal == NULL + || strm->internal->next.memconfig == NULL + || strm->internal->next.memconfig( + strm->internal->next.coder, + &memusage, &old_memlimit, 0) != LZMA_OK) + return 0; + + return memusage; +} + + +extern LZMA_API(uint64_t) +lzma_memlimit_get(const lzma_stream *strm) +{ + uint64_t old_memlimit; + uint64_t memusage; + + if (strm == NULL || strm->internal == NULL + || strm->internal->next.memconfig == NULL + || strm->internal->next.memconfig( + strm->internal->next.coder, + &memusage, &old_memlimit, 0) != LZMA_OK) + return 0; + + return old_memlimit; +} + + +extern LZMA_API(lzma_ret) +lzma_memlimit_set(lzma_stream *strm, uint64_t new_memlimit) +{ + // Dummy variables to simplify memconfig functions + uint64_t old_memlimit; + uint64_t memusage; + + if (strm == NULL || strm->internal == NULL + || strm->internal->next.memconfig == NULL) + return LZMA_PROG_ERROR; + + // Zero is a special value that cannot be used as an actual limit. + // If 0 was specified, use 1 instead. + if (new_memlimit == 0) + new_memlimit = 1; + + return strm->internal->next.memconfig(strm->internal->next.coder, + &memusage, &old_memlimit, new_memlimit); +} diff --git a/src/liblzma/common/common.h b/src/liblzma/common/common.h new file mode 100644 index 0000000000..20af32f6d6 --- /dev/null +++ b/src/liblzma/common/common.h @@ -0,0 +1,412 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file common.h +/// \brief Definitions common to the whole liblzma library +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_COMMON_H +#define LZMA_COMMON_H + +#include "sysdefs.h" +#include "mythread.h" +#include "tuklib_integer.h" + +// LZMA_API_EXPORT is used to mark the exported API functions. +// It's used to define the LZMA_API macro. +// +// lzma_attr_visibility_hidden is used for marking *declarations* of extern +// variables that are internal to liblzma (-fvisibility=hidden alone is +// enough to hide the *definitions*). Such markings allow slightly more +// efficient code to accesses those variables in ELF shared libraries. +#if defined(_WIN32) || defined(__CYGWIN__) +# ifdef DLL_EXPORT +# define LZMA_API_EXPORT __declspec(dllexport) +# else +# define LZMA_API_EXPORT +# endif +# define lzma_attr_visibility_hidden +// Don't use ifdef or defined() below. +#elif HAVE_VISIBILITY +# define LZMA_API_EXPORT __attribute__((__visibility__("default"))) +# define lzma_attr_visibility_hidden \ + __attribute__((__visibility__("hidden"))) +#else +# define LZMA_API_EXPORT +# define lzma_attr_visibility_hidden +#endif + +#define LZMA_API(type) LZMA_API_EXPORT type LZMA_API_CALL + +#include "lzma.h" + +// This is for detecting modern GCC and Clang attributes +// like __symver__ in GCC >= 10. +#ifdef __has_attribute +# define lzma_has_attribute(attr) __has_attribute(attr) +#else +# define lzma_has_attribute(attr) 0 +#endif + +// The extra symbol versioning in the C files may only be used when +// building a shared library. If HAVE_SYMBOL_VERSIONS_LINUX is defined +// to 2 then symbol versioning is done only if also PIC is defined. +// By default Libtool defines PIC when building a shared library and +// doesn't define it when building a static library but it can be +// overridden with --with-pic and --without-pic. configure let's rely +// on PIC if neither --with-pic or --without-pic was used. +#if defined(HAVE_SYMBOL_VERSIONS_LINUX) \ + && (HAVE_SYMBOL_VERSIONS_LINUX == 2 && !defined(PIC)) +# undef HAVE_SYMBOL_VERSIONS_LINUX +#endif + +#ifdef HAVE_SYMBOL_VERSIONS_LINUX +// To keep link-time optimization (LTO, -flto) working with GCC, +// the __symver__ attribute must be used instead of __asm__(".symver ..."). +// Otherwise the symbol versions may be lost, resulting in broken liblzma +// that has wrong default versions in the exported symbol list! +// The attribute was added in GCC 10; LTO with older GCC is not supported. +// +// To keep -Wmissing-prototypes happy, use LZMA_SYMVER_API only with function +// declarations (including those with __alias__ attribute) and LZMA_API with +// the function definitions. This means a little bit of silly copy-and-paste +// between declarations and definitions though. +// +// As of GCC 12.2, the __symver__ attribute supports only @ and @@ but the +// very convenient @@@ isn't supported (it's supported by GNU assembler +// since 2000). When using @@ instead of @@@, the internal name must not be +// the same as the external name to avoid problems in some situations. This +// is why "#define foo_52 foo" is needed for the default symbol versions. +// +// __has_attribute is supported before GCC 10 and it is supported in Clang 14 +// too (which doesn't support __symver__) so use it to detect if __symver__ +// is available. This should be far more reliable than looking at compiler +// version macros as nowadays especially __GNUC__ is defined by many compilers. +# if lzma_has_attribute(__symver__) +# define LZMA_SYMVER_API(extnamever, type, intname) \ + extern __attribute__((__symver__(extnamever))) \ + LZMA_API(type) intname +# else +# define LZMA_SYMVER_API(extnamever, type, intname) \ + __asm__(".symver " #intname "," extnamever); \ + extern LZMA_API(type) intname +# endif +#endif + +// MSVC has __forceinline which shouldn't be combined with the inline keyword +// (results in a warning). +// +// GCC 3.1 added always_inline attribute so we don't need to check +// for __GNUC__ version. Similarly, all relevant Clang versions +// support it (at least Clang 3.0.0 does already). +// Other compilers might support too which also support __has_attribute +// (Solaris Studio) so do that check too. +#if defined(_MSC_VER) +# define lzma_always_inline __forceinline +#elif defined(__GNUC__) || defined(__clang__) || defined(__INTEL_COMPILER) \ + || lzma_has_attribute(__always_inline__) +# define lzma_always_inline inline __attribute__((__always_inline__)) +#else +# define lzma_always_inline inline +#endif + +// These allow helping the compiler in some often-executed branches, whose +// result is almost always the same. +#ifdef __GNUC__ +# define likely(expr) __builtin_expect(expr, true) +# define unlikely(expr) __builtin_expect(expr, false) +#else +# define likely(expr) (expr) +# define unlikely(expr) (expr) +#endif + + +/// Size of temporary buffers needed in some filters +#define LZMA_BUFFER_SIZE 4096 + + +/// Maximum number of worker threads within one multithreaded component. +/// The limit exists solely to make it simpler to prevent integer overflows +/// when allocating structures etc. This should be big enough for now... +/// the code won't scale anywhere close to this number anyway. +#define LZMA_THREADS_MAX 16384 + + +/// Starting value for memory usage estimates. Instead of calculating size +/// of _every_ structure and taking into account malloc() overhead etc., we +/// add a base size to all memory usage estimates. It's not very accurate +/// but should be easily good enough. +#define LZMA_MEMUSAGE_BASE (UINT64_C(1) << 15) + +/// Start of internal Filter ID space. These IDs must never be used +/// in Streams. +#define LZMA_FILTER_RESERVED_START (LZMA_VLI_C(1) << 62) + + +/// Supported flags that can be passed to lzma_stream_decoder(), +/// lzma_auto_decoder(), or lzma_stream_decoder_mt(). +#define LZMA_SUPPORTED_FLAGS \ + ( LZMA_TELL_NO_CHECK \ + | LZMA_TELL_UNSUPPORTED_CHECK \ + | LZMA_TELL_ANY_CHECK \ + | LZMA_IGNORE_CHECK \ + | LZMA_CONCATENATED \ + | LZMA_FAIL_FAST ) + + +/// Largest valid lzma_action value as unsigned integer. +#define LZMA_ACTION_MAX ((unsigned int)(LZMA_FULL_BARRIER)) + + +/// Special return value (lzma_ret) to indicate that a timeout was reached +/// and lzma_code() must not return LZMA_BUF_ERROR. This is converted to +/// LZMA_OK in lzma_code(). +#define LZMA_TIMED_OUT LZMA_RET_INTERNAL1 + +/// Special return value (lzma_ret) for use in stream_decoder_mt.c to +/// indicate Index was detected instead of a Block Header. +#define LZMA_INDEX_DETECTED LZMA_RET_INTERNAL2 + + +typedef struct lzma_next_coder_s lzma_next_coder; + +typedef struct lzma_filter_info_s lzma_filter_info; + + +/// Type of a function used to initialize a filter encoder or decoder +typedef lzma_ret (*lzma_init_function)( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters); + +/// Type of a function to do some kind of coding work (filters, Stream, +/// Block encoders/decoders etc.). Some special coders use don't use both +/// input and output buffers, but for simplicity they still use this same +/// function prototype. +typedef lzma_ret (*lzma_code_function)( + void *coder, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, + lzma_action action); + +/// Type of a function to free the memory allocated for the coder +typedef void (*lzma_end_function)( + void *coder, const lzma_allocator *allocator); + + +/// Raw coder validates and converts an array of lzma_filter structures to +/// an array of lzma_filter_info structures. This array is used with +/// lzma_next_filter_init to initialize the filter chain. +struct lzma_filter_info_s { + /// Filter ID. This can be used to share the same initiazation + /// function *and* data structures with different Filter IDs + /// (LZMA_FILTER_LZMA1EXT does it), and also by the encoder + /// with lzma_filters_update() if filter chain is updated + /// in the middle of a raw stream or Block (LZMA_SYNC_FLUSH). + lzma_vli id; + + /// Pointer to function used to initialize the filter. + /// This is NULL to indicate end of array. + lzma_init_function init; + + /// Pointer to filter's options structure + void *options; +}; + + +/// Hold data and function pointers of the next filter in the chain. +struct lzma_next_coder_s { + /// Pointer to coder-specific data + void *coder; + + /// Filter ID. This is LZMA_VLI_UNKNOWN when this structure doesn't + /// point to a filter coder. + lzma_vli id; + + /// "Pointer" to init function. This is never called here. + /// We need only to detect if we are initializing a coder + /// that was allocated earlier. See lzma_next_coder_init and + /// lzma_next_strm_init macros in this file. + uintptr_t init; + + /// Pointer to function to do the actual coding + lzma_code_function code; + + /// Pointer to function to free lzma_next_coder.coder. This can + /// be NULL; in that case, lzma_free is called to free + /// lzma_next_coder.coder. + lzma_end_function end; + + /// Pointer to a function to get progress information. If this is NULL, + /// lzma_stream.total_in and .total_out are used instead. + void (*get_progress)(void *coder, + uint64_t *progress_in, uint64_t *progress_out); + + /// Pointer to function to return the type of the integrity check. + /// Most coders won't support this. + lzma_check (*get_check)(const void *coder); + + /// Pointer to function to get and/or change the memory usage limit. + /// If new_memlimit == 0, the limit is not changed. + lzma_ret (*memconfig)(void *coder, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit); + + /// Update the filter-specific options or the whole filter chain + /// in the encoder. + lzma_ret (*update)(void *coder, const lzma_allocator *allocator, + const lzma_filter *filters, + const lzma_filter *reversed_filters); + + /// Set how many bytes of output this coder may produce at maximum. + /// On success LZMA_OK must be returned. + /// If the filter chain as a whole cannot support this feature, + /// this must return LZMA_OPTIONS_ERROR. + /// If no input has been given to the coder and the requested limit + /// is too small, this must return LZMA_BUF_ERROR. If input has been + /// seen, LZMA_OK is allowed too. + lzma_ret (*set_out_limit)(void *coder, uint64_t *uncomp_size, + uint64_t out_limit); +}; + + +/// Macro to initialize lzma_next_coder structure +#define LZMA_NEXT_CODER_INIT \ + (lzma_next_coder){ \ + .coder = NULL, \ + .init = (uintptr_t)(NULL), \ + .id = LZMA_VLI_UNKNOWN, \ + .code = NULL, \ + .end = NULL, \ + .get_progress = NULL, \ + .get_check = NULL, \ + .memconfig = NULL, \ + .update = NULL, \ + .set_out_limit = NULL, \ + } + + +/// Internal data for lzma_strm_init, lzma_code, and lzma_end. A pointer to +/// this is stored in lzma_stream. +struct lzma_internal_s { + /// The actual coder that should do something useful + lzma_next_coder next; + + /// Track the state of the coder. This is used to validate arguments + /// so that the actual coders can rely on e.g. that LZMA_SYNC_FLUSH + /// is used on every call to lzma_code until next.code has returned + /// LZMA_STREAM_END. + enum { + ISEQ_RUN, + ISEQ_SYNC_FLUSH, + ISEQ_FULL_FLUSH, + ISEQ_FINISH, + ISEQ_FULL_BARRIER, + ISEQ_END, + ISEQ_ERROR, + } sequence; + + /// A copy of lzma_stream avail_in. This is used to verify that the + /// amount of input doesn't change once e.g. LZMA_FINISH has been + /// used. + size_t avail_in; + + /// Indicates which lzma_action values are allowed by next.code. + bool supported_actions[LZMA_ACTION_MAX + 1]; + + /// If true, lzma_code will return LZMA_BUF_ERROR if no progress was + /// made (no input consumed and no output produced by next.code). + bool allow_buf_error; +}; + + +/// Allocates memory +lzma_attr_alloc_size(1) +extern void *lzma_alloc(size_t size, const lzma_allocator *allocator); + +/// Allocates memory and zeroes it (like calloc()). This can be faster +/// than lzma_alloc() + memzero() while being backward compatible with +/// custom allocators. +lzma_attr_alloc_size(1) +extern void *lzma_alloc_zero(size_t size, const lzma_allocator *allocator); + +/// Frees memory +extern void lzma_free(void *ptr, const lzma_allocator *allocator); + + +/// Allocates strm->internal if it is NULL, and initializes *strm and +/// strm->internal. This function is only called via lzma_next_strm_init macro. +extern lzma_ret lzma_strm_init(lzma_stream *strm); + +/// Initializes the next filter in the chain, if any. This takes care of +/// freeing the memory of previously initialized filter if it is different +/// than the filter being initialized now. This way the actual filter +/// initialization functions don't need to use lzma_next_coder_init macro. +extern lzma_ret lzma_next_filter_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +/// Update the next filter in the chain, if any. This checks that +/// the application is not trying to change the Filter IDs. +extern lzma_ret lzma_next_filter_update( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *reversed_filters); + +/// Frees the memory allocated for next->coder either using next->end or, +/// if next->end is NULL, using lzma_free. +extern void lzma_next_end(lzma_next_coder *next, + const lzma_allocator *allocator); + + +/// Copy as much data as possible from in[] to out[] and update *in_pos +/// and *out_pos accordingly. Returns the number of bytes copied. +extern size_t lzma_bufcpy(const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size); + + +/// \brief Return if expression doesn't evaluate to LZMA_OK +/// +/// There are several situations where we want to return immediately +/// with the value of expr if it isn't LZMA_OK. This macro shortens +/// the code a little. +#define return_if_error(expr) \ +do { \ + const lzma_ret ret_ = (expr); \ + if (ret_ != LZMA_OK) \ + return ret_; \ +} while (0) + + +/// If next isn't already initialized, free the previous coder. Then mark +/// that next is _possibly_ initialized for the coder using this macro. +/// "Possibly" means that if e.g. allocation of next->coder fails, the +/// structure isn't actually initialized for this coder, but leaving +/// next->init to func is still OK. +#define lzma_next_coder_init(func, next, allocator) \ +do { \ + if ((uintptr_t)(func) != (next)->init) \ + lzma_next_end(next, allocator); \ + (next)->init = (uintptr_t)(func); \ +} while (0) + + +/// Initializes lzma_strm and calls func() to initialize strm->internal->next. +/// (The function being called will use lzma_next_coder_init()). If +/// initialization fails, memory that wasn't freed by func() is freed +/// along strm->internal. +#define lzma_next_strm_init(func, strm, ...) \ +do { \ + return_if_error(lzma_strm_init(strm)); \ + const lzma_ret ret_ = func(&(strm)->internal->next, \ + (strm)->allocator, __VA_ARGS__); \ + if (ret_ != LZMA_OK) { \ + lzma_end(strm); \ + return ret_; \ + } \ +} while (0) + +#endif diff --git a/src/liblzma/common/easy_buffer_encoder.c b/src/liblzma/common/easy_buffer_encoder.c new file mode 100644 index 0000000000..da610cea6b --- /dev/null +++ b/src/liblzma/common/easy_buffer_encoder.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file easy_buffer_encoder.c +/// \brief Easy single-call .xz Stream encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "easy_preset.h" + + +extern LZMA_API(lzma_ret) +lzma_easy_buffer_encode(uint32_t preset, lzma_check check, + const lzma_allocator *allocator, const uint8_t *in, + size_t in_size, uint8_t *out, size_t *out_pos, size_t out_size) +{ + lzma_options_easy opt_easy; + if (lzma_easy_preset(&opt_easy, preset)) + return LZMA_OPTIONS_ERROR; + + return lzma_stream_buffer_encode(opt_easy.filters, check, + allocator, in, in_size, out, out_pos, out_size); +} diff --git a/src/liblzma/common/easy_decoder_memusage.c b/src/liblzma/common/easy_decoder_memusage.c new file mode 100644 index 0000000000..0c76f10033 --- /dev/null +++ b/src/liblzma/common/easy_decoder_memusage.c @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file easy_decoder_memusage.c +/// \brief Decoder memory usage calculation to match easy encoder presets +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "easy_preset.h" + + +extern LZMA_API(uint64_t) +lzma_easy_decoder_memusage(uint32_t preset) +{ + lzma_options_easy opt_easy; + if (lzma_easy_preset(&opt_easy, preset)) + return UINT32_MAX; + + return lzma_raw_decoder_memusage(opt_easy.filters); +} diff --git a/src/liblzma/common/easy_encoder.c b/src/liblzma/common/easy_encoder.c new file mode 100644 index 0000000000..8dfe29610f --- /dev/null +++ b/src/liblzma/common/easy_encoder.c @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file easy_encoder.c +/// \brief Easy .xz Stream encoder initialization +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "easy_preset.h" + + +extern LZMA_API(lzma_ret) +lzma_easy_encoder(lzma_stream *strm, uint32_t preset, lzma_check check) +{ + lzma_options_easy opt_easy; + if (lzma_easy_preset(&opt_easy, preset)) + return LZMA_OPTIONS_ERROR; + + return lzma_stream_encoder(strm, opt_easy.filters, check); +} diff --git a/src/liblzma/common/easy_encoder_memusage.c b/src/liblzma/common/easy_encoder_memusage.c new file mode 100644 index 0000000000..1184ac6654 --- /dev/null +++ b/src/liblzma/common/easy_encoder_memusage.c @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file easy_encoder_memusage.c +/// \brief Easy .xz Stream encoder memory usage calculation +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "easy_preset.h" + + +extern LZMA_API(uint64_t) +lzma_easy_encoder_memusage(uint32_t preset) +{ + lzma_options_easy opt_easy; + if (lzma_easy_preset(&opt_easy, preset)) + return UINT32_MAX; + + return lzma_raw_encoder_memusage(opt_easy.filters); +} diff --git a/src/liblzma/common/easy_preset.c b/src/liblzma/common/easy_preset.c new file mode 100644 index 0000000000..7908a2bb73 --- /dev/null +++ b/src/liblzma/common/easy_preset.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file easy_preset.c +/// \brief Preset handling for easy encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "easy_preset.h" + + +extern bool +lzma_easy_preset(lzma_options_easy *opt_easy, uint32_t preset) +{ + if (lzma_lzma_preset(&opt_easy->opt_lzma, preset)) + return true; + + opt_easy->filters[0].id = LZMA_FILTER_LZMA2; + opt_easy->filters[0].options = &opt_easy->opt_lzma; + opt_easy->filters[1].id = LZMA_VLI_UNKNOWN; + + return false; +} diff --git a/src/liblzma/common/easy_preset.h b/src/liblzma/common/easy_preset.h new file mode 100644 index 0000000000..4ef6d044ad --- /dev/null +++ b/src/liblzma/common/easy_preset.h @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file easy_preset.h +/// \brief Preset handling for easy encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_EASY_PRESET_H +#define LZMA_EASY_PRESET_H + +#include "common.h" + + +typedef struct { + /// We need to keep the filters array available in case + /// LZMA_FULL_FLUSH is used. + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + /// Options for LZMA2 + lzma_options_lzma opt_lzma; + + // Options for more filters can be added later, so this struct + // is not ready to be put into the public API. + +} lzma_options_easy; + + +/// Set *easy to the settings given by the preset. Returns true on error, +/// false on success. +extern bool lzma_easy_preset(lzma_options_easy *easy, uint32_t preset); + +#endif diff --git a/src/liblzma/common/file_info.c b/src/liblzma/common/file_info.c new file mode 100644 index 0000000000..4b2eb5d040 --- /dev/null +++ b/src/liblzma/common/file_info.c @@ -0,0 +1,850 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file file_info.c +/// \brief Decode .xz file information into a lzma_index structure +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "index_decoder.h" + + +typedef struct { + enum { + SEQ_MAGIC_BYTES, + SEQ_PADDING_SEEK, + SEQ_PADDING_DECODE, + SEQ_FOOTER, + SEQ_INDEX_INIT, + SEQ_INDEX_DECODE, + SEQ_HEADER_DECODE, + SEQ_HEADER_COMPARE, + } sequence; + + /// Absolute position of in[*in_pos] in the file. All code that + /// modifies *in_pos also updates this. seek_to_pos() needs this + /// to determine if we need to request the application to seek for + /// us or if we can do the seeking internally by adjusting *in_pos. + uint64_t file_cur_pos; + + /// This refers to absolute positions of interesting parts of the + /// input file. Sometimes it points to the *beginning* of a specific + /// field and sometimes to the *end* of a field. The current target + /// position at each moment is explained in the comments. + uint64_t file_target_pos; + + /// Size of the .xz file (from the application). + uint64_t file_size; + + /// Index decoder + lzma_next_coder index_decoder; + + /// Number of bytes remaining in the Index field that is currently + /// being decoded. + lzma_vli index_remaining; + + /// The Index decoder will store the decoded Index in this pointer. + lzma_index *this_index; + + /// Amount of Stream Padding in the current Stream. + lzma_vli stream_padding; + + /// The final combined index is collected here. + lzma_index *combined_index; + + /// Pointer from the application where to store the index information + /// after successful decoding. + lzma_index **dest_index; + + /// Pointer to lzma_stream.seek_pos to be used when returning + /// LZMA_SEEK_NEEDED. This is set by seek_to_pos() when needed. + uint64_t *external_seek_pos; + + /// Memory usage limit + uint64_t memlimit; + + /// Stream Flags from the very beginning of the file. + lzma_stream_flags first_header_flags; + + /// Stream Flags from Stream Header of the current Stream. + lzma_stream_flags header_flags; + + /// Stream Flags from Stream Footer of the current Stream. + lzma_stream_flags footer_flags; + + size_t temp_pos; + size_t temp_size; + uint8_t temp[8192]; + +} lzma_file_info_coder; + + +/// Copies data from in[*in_pos] into coder->temp until +/// coder->temp_pos == coder->temp_size. This also keeps coder->file_cur_pos +/// in sync with *in_pos. Returns true if more input is needed. +static bool +fill_temp(lzma_file_info_coder *coder, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size) +{ + coder->file_cur_pos += lzma_bufcpy(in, in_pos, in_size, + coder->temp, &coder->temp_pos, coder->temp_size); + return coder->temp_pos < coder->temp_size; +} + + +/// Seeks to the absolute file position specified by target_pos. +/// This tries to do the seeking by only modifying *in_pos, if possible. +/// The main benefit of this is that if one passes the whole file at once +/// to lzma_code(), the decoder will never need to return LZMA_SEEK_NEEDED +/// as all the seeking can be done by adjusting *in_pos in this function. +/// +/// Returns true if an external seek is needed and the caller must return +/// LZMA_SEEK_NEEDED. +static bool +seek_to_pos(lzma_file_info_coder *coder, uint64_t target_pos, + size_t in_start, size_t *in_pos, size_t in_size) +{ + // The input buffer doesn't extend beyond the end of the file. + // This has been checked by file_info_decode() already. + assert(coder->file_size - coder->file_cur_pos >= in_size - *in_pos); + + const uint64_t pos_min = coder->file_cur_pos - (*in_pos - in_start); + const uint64_t pos_max = coder->file_cur_pos + (in_size - *in_pos); + + bool external_seek_needed; + + if (target_pos >= pos_min && target_pos <= pos_max) { + // The requested position is available in the current input + // buffer or right after it. That is, in a corner case we + // end up setting *in_pos == in_size and thus will immediately + // need new input bytes from the application. + *in_pos += (size_t)(target_pos - coder->file_cur_pos); + external_seek_needed = false; + } else { + // Ask the application to seek the input file. + *coder->external_seek_pos = target_pos; + external_seek_needed = true; + + // Mark the whole input buffer as used. This way + // lzma_stream.total_in will have a better estimate + // of the amount of data read. It still won't be perfect + // as the value will depend on the input buffer size that + // the application uses, but it should be good enough for + // those few who want an estimate. + *in_pos = in_size; + } + + // After seeking (internal or external) the current position + // will match the requested target position. + coder->file_cur_pos = target_pos; + + return external_seek_needed; +} + + +/// The caller sets coder->file_target_pos so that it points to the *end* +/// of the desired file position. This function then determines how far +/// backwards from that position we can seek. After seeking fill_temp() +/// can be used to read data into coder->temp. When fill_temp() has finished, +/// coder->temp[coder->temp_size] will match coder->file_target_pos. +/// +/// This also validates that coder->target_file_pos is sane in sense that +/// we aren't trying to seek too far backwards (too close or beyond the +/// beginning of the file). +static lzma_ret +reverse_seek(lzma_file_info_coder *coder, + size_t in_start, size_t *in_pos, size_t in_size) +{ + // Check that there is enough data before the target position + // to contain at least Stream Header and Stream Footer. If there + // isn't, the file cannot be valid. + if (coder->file_target_pos < 2 * LZMA_STREAM_HEADER_SIZE) + return LZMA_DATA_ERROR; + + coder->temp_pos = 0; + + // The Stream Header at the very beginning of the file gets handled + // specially in SEQ_MAGIC_BYTES and thus we will never need to seek + // there. By not seeking to the first LZMA_STREAM_HEADER_SIZE bytes + // we avoid a useless external seek after SEQ_MAGIC_BYTES if the + // application uses an extremely small input buffer and the input + // file is very small. + if (coder->file_target_pos - LZMA_STREAM_HEADER_SIZE + < sizeof(coder->temp)) + coder->temp_size = (size_t)(coder->file_target_pos + - LZMA_STREAM_HEADER_SIZE); + else + coder->temp_size = sizeof(coder->temp); + + // The above if-statements guarantee this. This is important because + // the Stream Header/Footer decoders assume that there's at least + // LZMA_STREAM_HEADER_SIZE bytes in coder->temp. + assert(coder->temp_size >= LZMA_STREAM_HEADER_SIZE); + + if (seek_to_pos(coder, coder->file_target_pos - coder->temp_size, + in_start, in_pos, in_size)) + return LZMA_SEEK_NEEDED; + + return LZMA_OK; +} + + +/// Gets the number of zero-bytes at the end of the buffer. +static size_t +get_padding_size(const uint8_t *buf, size_t buf_size) +{ + size_t padding = 0; + while (buf_size > 0 && buf[--buf_size] == 0x00) + ++padding; + + return padding; +} + + +/// With the Stream Header at the very beginning of the file, LZMA_FORMAT_ERROR +/// is used to tell the application that Magic Bytes didn't match. In other +/// Stream Header/Footer fields (in the middle/end of the file) it could be +/// a bit confusing to return LZMA_FORMAT_ERROR as we already know that there +/// is a valid Stream Header at the beginning of the file. For those cases +/// this function is used to convert LZMA_FORMAT_ERROR to LZMA_DATA_ERROR. +static lzma_ret +hide_format_error(lzma_ret ret) +{ + if (ret == LZMA_FORMAT_ERROR) + ret = LZMA_DATA_ERROR; + + return ret; +} + + +/// Calls the Index decoder and updates coder->index_remaining. +/// This is a separate function because the input can be either directly +/// from the application or from coder->temp. +static lzma_ret +decode_index(lzma_file_info_coder *coder, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, bool update_file_cur_pos) +{ + const size_t in_start = *in_pos; + + const lzma_ret ret = coder->index_decoder.code( + coder->index_decoder.coder, + allocator, in, in_pos, in_size, + NULL, NULL, 0, LZMA_RUN); + + coder->index_remaining -= *in_pos - in_start; + + if (update_file_cur_pos) + coder->file_cur_pos += *in_pos - in_start; + + return ret; +} + + +static lzma_ret +file_info_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, + uint8_t *restrict out lzma_attribute((__unused__)), + size_t *restrict out_pos lzma_attribute((__unused__)), + size_t out_size lzma_attribute((__unused__)), + lzma_action action lzma_attribute((__unused__))) +{ + lzma_file_info_coder *coder = coder_ptr; + const size_t in_start = *in_pos; + + // If the caller provides input past the end of the file, trim + // the extra bytes from the buffer so that we won't read too far. + assert(coder->file_size >= coder->file_cur_pos); + if (coder->file_size - coder->file_cur_pos < in_size - in_start) + in_size = in_start + + (size_t)(coder->file_size - coder->file_cur_pos); + + while (true) + switch (coder->sequence) { + case SEQ_MAGIC_BYTES: + // Decode the Stream Header at the beginning of the file + // first to check if the Magic Bytes match. The flags + // are stored in coder->first_header_flags so that we + // don't need to seek to it again. + // + // Check that the file is big enough to contain at least + // Stream Header. + if (coder->file_size < LZMA_STREAM_HEADER_SIZE) + return LZMA_FORMAT_ERROR; + + // Read the Stream Header field into coder->temp. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // This is the only Stream Header/Footer decoding where we + // want to return LZMA_FORMAT_ERROR if the Magic Bytes don't + // match. Elsewhere it will be converted to LZMA_DATA_ERROR. + return_if_error(lzma_stream_header_decode( + &coder->first_header_flags, coder->temp)); + + // Now that we know that the Magic Bytes match, check the + // file size. It's better to do this here after checking the + // Magic Bytes since this way we can give LZMA_FORMAT_ERROR + // instead of LZMA_DATA_ERROR when the Magic Bytes don't + // match in a file that is too big or isn't a multiple of + // four bytes. + if (coder->file_size > LZMA_VLI_MAX || (coder->file_size & 3)) + return LZMA_DATA_ERROR; + + // Start looking for Stream Padding and Stream Footer + // at the end of the file. + coder->file_target_pos = coder->file_size; + FALLTHROUGH; + + case SEQ_PADDING_SEEK: + coder->sequence = SEQ_PADDING_DECODE; + return_if_error(reverse_seek( + coder, in_start, in_pos, in_size)); + FALLTHROUGH; + + case SEQ_PADDING_DECODE: { + // Copy to coder->temp first. This keeps the code simpler if + // the application only provides input a few bytes at a time. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // Scan the buffer backwards to get the size of the + // Stream Padding field (if any). + const size_t new_padding = get_padding_size( + coder->temp, coder->temp_size); + coder->stream_padding += new_padding; + + // Set the target position to the beginning of Stream Padding + // that has been observed so far. If all Stream Padding has + // been seen, then the target position will be at the end + // of the Stream Footer field. + coder->file_target_pos -= new_padding; + + if (new_padding == coder->temp_size) { + // The whole buffer was padding. Seek backwards in + // the file to get more input. + coder->sequence = SEQ_PADDING_SEEK; + break; + } + + // Size of Stream Padding must be a multiple of 4 bytes. + if (coder->stream_padding & 3) + return LZMA_DATA_ERROR; + + coder->sequence = SEQ_FOOTER; + + // Calculate the amount of non-padding data in coder->temp. + coder->temp_size -= new_padding; + coder->temp_pos = coder->temp_size; + + // We can avoid an external seek if the whole Stream Footer + // is already in coder->temp. In that case SEQ_FOOTER won't + // read more input and will find the Stream Footer from + // coder->temp[coder->temp_size - LZMA_STREAM_HEADER_SIZE]. + // + // Otherwise we will need to seek. The seeking is done so + // that Stream Footer will be at the end of coder->temp. + // This way it's likely that we also get a complete Index + // field into coder->temp without needing a separate seek + // for that (unless the Index field is big). + if (coder->temp_size < LZMA_STREAM_HEADER_SIZE) + return_if_error(reverse_seek( + coder, in_start, in_pos, in_size)); + + FALLTHROUGH; + } + + case SEQ_FOOTER: + // Copy the Stream Footer field into coder->temp. + // If Stream Footer was already available in coder->temp + // in SEQ_PADDING_DECODE, then this does nothing. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // Make coder->file_target_pos and coder->temp_size point + // to the beginning of Stream Footer and thus to the end + // of the Index field. coder->temp_pos will be updated + // a bit later. + coder->file_target_pos -= LZMA_STREAM_HEADER_SIZE; + coder->temp_size -= LZMA_STREAM_HEADER_SIZE; + + // Decode Stream Footer. + return_if_error(hide_format_error(lzma_stream_footer_decode( + &coder->footer_flags, + coder->temp + coder->temp_size))); + + // Check that we won't seek past the beginning of the file. + // + // LZMA_STREAM_HEADER_SIZE is added because there must be + // space for Stream Header too even though we won't seek + // there before decoding the Index field. + // + // There's no risk of integer overflow here because + // Backward Size cannot be greater than 2^34. + if (coder->file_target_pos < coder->footer_flags.backward_size + + LZMA_STREAM_HEADER_SIZE) + return LZMA_DATA_ERROR; + + // Set the target position to the beginning of the Index field. + coder->file_target_pos -= coder->footer_flags.backward_size; + coder->sequence = SEQ_INDEX_INIT; + + // We can avoid an external seek if the whole Index field is + // already available in coder->temp. + if (coder->temp_size >= coder->footer_flags.backward_size) { + // Set coder->temp_pos to point to the beginning + // of the Index. + coder->temp_pos = coder->temp_size + - coder->footer_flags.backward_size; + } else { + // These are set to zero to indicate that there's no + // useful data (Index or anything else) in coder->temp. + coder->temp_pos = 0; + coder->temp_size = 0; + + // Seek to the beginning of the Index field. + if (seek_to_pos(coder, coder->file_target_pos, + in_start, in_pos, in_size)) + return LZMA_SEEK_NEEDED; + } + + FALLTHROUGH; + + case SEQ_INDEX_INIT: { + // Calculate the amount of memory already used by the earlier + // Indexes so that we know how big memory limit to pass to + // the Index decoder. + // + // NOTE: When there are multiple Streams, the separate + // lzma_index structures can use more RAM (as measured by + // lzma_index_memused()) than the final combined lzma_index. + // Thus memlimit may need to be slightly higher than the final + // calculated memory usage will be. This is perhaps a bit + // confusing to the application, but I think it shouldn't + // cause problems in practice. + uint64_t memused = 0; + if (coder->combined_index != NULL) { + memused = lzma_index_memused(coder->combined_index); + assert(memused <= coder->memlimit); + if (memused > coder->memlimit) // Extra sanity check + return LZMA_PROG_ERROR; + } + + // Initialize the Index decoder. + return_if_error(lzma_index_decoder_init( + &coder->index_decoder, allocator, + &coder->this_index, + coder->memlimit - memused)); + + coder->index_remaining = coder->footer_flags.backward_size; + coder->sequence = SEQ_INDEX_DECODE; + FALLTHROUGH; + } + + case SEQ_INDEX_DECODE: { + // Decode (a part of) the Index. If the whole Index is already + // in coder->temp, read it from there. Otherwise read from + // in[*in_pos] onwards. Note that index_decode() updates + // coder->index_remaining and optionally coder->file_cur_pos. + lzma_ret ret; + if (coder->temp_size != 0) { + assert(coder->temp_size - coder->temp_pos + == coder->index_remaining); + ret = decode_index(coder, allocator, coder->temp, + &coder->temp_pos, coder->temp_size, + false); + } else { + // Don't give the decoder more input than the known + // remaining size of the Index field. + size_t in_stop = in_size; + if (in_size - *in_pos > coder->index_remaining) + in_stop = *in_pos + + (size_t)(coder->index_remaining); + + ret = decode_index(coder, allocator, + in, in_pos, in_stop, true); + } + + switch (ret) { + case LZMA_OK: + // If the Index docoder asks for more input when we + // have already given it as much input as Backward Size + // indicated, the file is invalid. + if (coder->index_remaining == 0) + return LZMA_DATA_ERROR; + + // We cannot get here if we were reading Index from + // coder->temp because when reading from coder->temp + // we give the Index decoder exactly + // coder->index_remaining bytes of input. + assert(coder->temp_size == 0); + + return LZMA_OK; + + case LZMA_STREAM_END: + // If the decoding seems to be successful, check also + // that the Index decoder consumed as much input as + // indicated by the Backward Size field. + if (coder->index_remaining != 0) + return LZMA_DATA_ERROR; + + break; + + default: + return ret; + } + + // Calculate how much the Index tells us to seek backwards + // (relative to the beginning of the Index): Total size of + // all Blocks plus the size of the Stream Header field. + // No integer overflow here because lzma_index_total_size() + // cannot return a value greater than LZMA_VLI_MAX. + const uint64_t seek_amount + = lzma_index_total_size(coder->this_index) + + LZMA_STREAM_HEADER_SIZE; + + // Check that Index is sane in sense that seek_amount won't + // make us seek past the beginning of the file when locating + // the Stream Header. + // + // coder->file_target_pos still points to the beginning of + // the Index field. + if (coder->file_target_pos < seek_amount) + return LZMA_DATA_ERROR; + + // Set the target to the beginning of Stream Header. + coder->file_target_pos -= seek_amount; + + if (coder->file_target_pos == 0) { + // We would seek to the beginning of the file, but + // since we already decoded that Stream Header in + // SEQ_MAGIC_BYTES, we can use the cached value from + // coder->first_header_flags to avoid the seek. + coder->header_flags = coder->first_header_flags; + coder->sequence = SEQ_HEADER_COMPARE; + break; + } + + coder->sequence = SEQ_HEADER_DECODE; + + // Make coder->file_target_pos point to the end of + // the Stream Header field. + coder->file_target_pos += LZMA_STREAM_HEADER_SIZE; + + // If coder->temp_size is non-zero, it points to the end + // of the Index field. Then the beginning of the Index + // field is at coder->temp[coder->temp_size + // - coder->footer_flags.backward_size]. + assert(coder->temp_size == 0 || coder->temp_size + >= coder->footer_flags.backward_size); + + // If coder->temp contained the whole Index, see if it has + // enough data to contain also the Stream Header. If so, + // we avoid an external seek. + // + // NOTE: This can happen only with small .xz files and only + // for the non-first Stream as the Stream Flags of the first + // Stream are cached and already handled a few lines above. + // So this isn't as useful as the other seek-avoidance cases. + if (coder->temp_size != 0 && coder->temp_size + - coder->footer_flags.backward_size + >= seek_amount) { + // Make temp_pos and temp_size point to the *end* of + // Stream Header so that SEQ_HEADER_DECODE will find + // the start of Stream Header from coder->temp[ + // coder->temp_size - LZMA_STREAM_HEADER_SIZE]. + coder->temp_pos = coder->temp_size + - coder->footer_flags.backward_size + - seek_amount + + LZMA_STREAM_HEADER_SIZE; + coder->temp_size = coder->temp_pos; + } else { + // Seek so that Stream Header will be at the end of + // coder->temp. With typical multi-Stream files we + // will usually also get the Stream Footer and Index + // of the *previous* Stream in coder->temp and thus + // won't need a separate seek for them. + return_if_error(reverse_seek(coder, + in_start, in_pos, in_size)); + } + + FALLTHROUGH; + } + + case SEQ_HEADER_DECODE: + // Copy the Stream Header field into coder->temp. + // If Stream Header was already available in coder->temp + // in SEQ_INDEX_DECODE, then this does nothing. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // Make all these point to the beginning of Stream Header. + coder->file_target_pos -= LZMA_STREAM_HEADER_SIZE; + coder->temp_size -= LZMA_STREAM_HEADER_SIZE; + coder->temp_pos = coder->temp_size; + + // Decode the Stream Header. + return_if_error(hide_format_error(lzma_stream_header_decode( + &coder->header_flags, + coder->temp + coder->temp_size))); + + coder->sequence = SEQ_HEADER_COMPARE; + FALLTHROUGH; + + case SEQ_HEADER_COMPARE: + // Compare Stream Header against Stream Footer. They must + // match. + return_if_error(lzma_stream_flags_compare( + &coder->header_flags, &coder->footer_flags)); + + // Store the decoded Stream Flags into the Index. Use the + // Footer Flags because it contains Backward Size, although + // it shouldn't matter in practice. + if (lzma_index_stream_flags(coder->this_index, + &coder->footer_flags) != LZMA_OK) + return LZMA_PROG_ERROR; + + // Store also the size of the Stream Padding field. It is + // needed to calculate the offsets of the Streams correctly. + if (lzma_index_stream_padding(coder->this_index, + coder->stream_padding) != LZMA_OK) + return LZMA_PROG_ERROR; + + // Reset it so that it's ready for the next Stream. + coder->stream_padding = 0; + + // Append the earlier decoded Indexes after this_index. + if (coder->combined_index != NULL) + return_if_error(lzma_index_cat(coder->this_index, + coder->combined_index, allocator)); + + coder->combined_index = coder->this_index; + coder->this_index = NULL; + + // If the whole file was decoded, tell the caller that we + // are finished. + if (coder->file_target_pos == 0) { + // The combined index must indicate the same file + // size as was told to us at initialization. + assert(lzma_index_file_size(coder->combined_index) + == coder->file_size); + + // Make the combined index available to + // the application. + *coder->dest_index = coder->combined_index; + coder->combined_index = NULL; + + // Mark the input buffer as used since we may have + // done internal seeking and thus don't know how + // many input bytes were actually used. This way + // lzma_stream.total_in gets a slightly better + // estimate of the amount of input used. + *in_pos = in_size; + return LZMA_STREAM_END; + } + + // We didn't hit the beginning of the file yet, so continue + // reading backwards in the file. If we have unprocessed + // data in coder->temp, use it before requesting more data + // from the application. + // + // coder->file_target_pos, coder->temp_size, and + // coder->temp_pos all point to the beginning of Stream Header + // and thus the end of the previous Stream in the file. + coder->sequence = coder->temp_size > 0 + ? SEQ_PADDING_DECODE : SEQ_PADDING_SEEK; + break; + + default: + assert(0); + return LZMA_PROG_ERROR; + } +} + + +static lzma_ret +file_info_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_file_info_coder *coder = coder_ptr; + + // The memory usage calculation comes from three things: + // + // (1) The Indexes that have already been decoded and processed into + // coder->combined_index. + // + // (2) The latest Index in coder->this_index that has been decoded but + // not yet put into coder->combined_index. + // + // (3) The latest Index that we have started decoding but haven't + // finished and thus isn't available in coder->this_index yet. + // Memory usage and limit information needs to be communicated + // from/to coder->index_decoder. + // + // Care has to be taken to not do both (2) and (3) when calculating + // the memory usage. + uint64_t combined_index_memusage = 0; + uint64_t this_index_memusage = 0; + + // (1) If we have already successfully decoded one or more Indexes, + // get their memory usage. + if (coder->combined_index != NULL) + combined_index_memusage = lzma_index_memused( + coder->combined_index); + + // Choose between (2), (3), or neither. + if (coder->this_index != NULL) { + // (2) The latest Index is available. Use its memory usage. + this_index_memusage = lzma_index_memused(coder->this_index); + + } else if (coder->sequence == SEQ_INDEX_DECODE) { + // (3) The Index decoder is activate and hasn't yet stored + // the new index in coder->this_index. Get the memory usage + // information from the Index decoder. + // + // NOTE: If the Index decoder doesn't yet know how much memory + // it will eventually need, it will return a tiny value here. + uint64_t dummy; + if (coder->index_decoder.memconfig(coder->index_decoder.coder, + &this_index_memusage, &dummy, 0) + != LZMA_OK) { + assert(0); + return LZMA_PROG_ERROR; + } + } + + // Now we know the total memory usage/requirement. If we had neither + // old Indexes nor a new Index, this will be zero which isn't + // acceptable as lzma_memusage() has to return non-zero on success + // and even with an empty .xz file we will end up with a lzma_index + // that takes some memory. + *memusage = combined_index_memusage + this_index_memusage; + if (*memusage == 0) + *memusage = lzma_index_memusage(1, 0); + + *old_memlimit = coder->memlimit; + + // If requested, set a new memory usage limit. + if (new_memlimit != 0) { + if (new_memlimit < *memusage) + return LZMA_MEMLIMIT_ERROR; + + // In the condition (3) we need to tell the Index decoder + // its new memory usage limit. + if (coder->this_index == NULL + && coder->sequence == SEQ_INDEX_DECODE) { + const uint64_t idec_new_memlimit = new_memlimit + - combined_index_memusage; + + assert(this_index_memusage > 0); + assert(idec_new_memlimit > 0); + + uint64_t dummy1; + uint64_t dummy2; + + if (coder->index_decoder.memconfig( + coder->index_decoder.coder, + &dummy1, &dummy2, idec_new_memlimit) + != LZMA_OK) { + assert(0); + return LZMA_PROG_ERROR; + } + } + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +static void +file_info_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_file_info_coder *coder = coder_ptr; + + lzma_next_end(&coder->index_decoder, allocator); + lzma_index_end(coder->this_index, allocator); + lzma_index_end(coder->combined_index, allocator); + + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +lzma_file_info_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, uint64_t *seek_pos, + lzma_index **dest_index, + uint64_t memlimit, uint64_t file_size) +{ + lzma_next_coder_init(&lzma_file_info_decoder_init, next, allocator); + + if (dest_index == NULL) + return LZMA_PROG_ERROR; + + lzma_file_info_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_file_info_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &file_info_decode; + next->end = &file_info_decoder_end; + next->memconfig = &file_info_decoder_memconfig; + + coder->index_decoder = LZMA_NEXT_CODER_INIT; + coder->this_index = NULL; + coder->combined_index = NULL; + } + + coder->sequence = SEQ_MAGIC_BYTES; + coder->file_cur_pos = 0; + coder->file_target_pos = 0; + coder->file_size = file_size; + + lzma_index_end(coder->this_index, allocator); + coder->this_index = NULL; + + lzma_index_end(coder->combined_index, allocator); + coder->combined_index = NULL; + + coder->stream_padding = 0; + + coder->dest_index = dest_index; + coder->external_seek_pos = seek_pos; + + // If memlimit is 0, make it 1 to ensure that lzma_memlimit_get() + // won't return 0 (which would indicate an error). + coder->memlimit = my_max(1, memlimit); + + // Prepare these for reading the first Stream Header into coder->temp. + coder->temp_pos = 0; + coder->temp_size = LZMA_STREAM_HEADER_SIZE; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_file_info_decoder(lzma_stream *strm, lzma_index **dest_index, + uint64_t memlimit, uint64_t file_size) +{ + lzma_next_strm_init(lzma_file_info_decoder_init, strm, &strm->seek_pos, + dest_index, memlimit, file_size); + + // We allow LZMA_FINISH in addition to LZMA_RUN for convenience. + // lzma_code() is able to handle the LZMA_FINISH + LZMA_SEEK_NEEDED + // combination in a sane way. Applications still need to be careful + // if they use LZMA_FINISH so that they remember to reset it back + // to LZMA_RUN after seeking if needed. + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/filter_buffer_decoder.c b/src/liblzma/common/filter_buffer_decoder.c new file mode 100644 index 0000000000..cc0d88cc71 --- /dev/null +++ b/src/liblzma/common/filter_buffer_decoder.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_buffer_decoder.c +/// \brief Single-call raw decoding +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_decoder.h" + + +extern LZMA_API(lzma_ret) +lzma_raw_buffer_decode( + const lzma_filter *filters, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Validate what isn't validated later in filter_common.c. + if (in == NULL || in_pos == NULL || *in_pos > in_size || out == NULL + || out_pos == NULL || *out_pos > out_size) + return LZMA_PROG_ERROR; + + // Initialize the decoder. + lzma_next_coder next = LZMA_NEXT_CODER_INIT; + return_if_error(lzma_raw_decoder_init(&next, allocator, filters)); + + // Store the positions so that we can restore them if something + // goes wrong. + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + // Do the actual decoding and free decoder's memory. + lzma_ret ret = next.code(next.coder, allocator, in, in_pos, in_size, + out, out_pos, out_size, LZMA_FINISH); + + if (ret == LZMA_STREAM_END) { + ret = LZMA_OK; + } else { + if (ret == LZMA_OK) { + // Either the input was truncated or the + // output buffer was too small. + assert(*in_pos == in_size || *out_pos == out_size); + + if (*in_pos != in_size) { + // Since input wasn't consumed completely, + // the output buffer became full and is + // too small. + ret = LZMA_BUF_ERROR; + + } else if (*out_pos != out_size) { + // Since output didn't became full, the input + // has to be truncated. + ret = LZMA_DATA_ERROR; + + } else { + // All the input was consumed and output + // buffer is full. Now we don't immediately + // know the reason for the error. Try + // decoding one more byte. If it succeeds, + // then the output buffer was too small. If + // we cannot get a new output byte, the input + // is truncated. + uint8_t tmp[1]; + size_t tmp_pos = 0; + (void)next.code(next.coder, allocator, + in, in_pos, in_size, + tmp, &tmp_pos, 1, LZMA_FINISH); + + if (tmp_pos == 1) + ret = LZMA_BUF_ERROR; + else + ret = LZMA_DATA_ERROR; + } + } + + // Restore the positions. + *in_pos = in_start; + *out_pos = out_start; + } + + lzma_next_end(&next, allocator); + + return ret; +} diff --git a/src/liblzma/common/filter_buffer_encoder.c b/src/liblzma/common/filter_buffer_encoder.c new file mode 100644 index 0000000000..7fb8922ae9 --- /dev/null +++ b/src/liblzma/common/filter_buffer_encoder.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_buffer_encoder.c +/// \brief Single-call raw encoding +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_encoder.h" + + +extern LZMA_API(lzma_ret) +lzma_raw_buffer_encode( + const lzma_filter *filters, const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Validate what isn't validated later in filter_common.c. + if ((in == NULL && in_size != 0) || out == NULL + || out_pos == NULL || *out_pos > out_size) + return LZMA_PROG_ERROR; + + // Initialize the encoder + lzma_next_coder next = LZMA_NEXT_CODER_INIT; + return_if_error(lzma_raw_encoder_init(&next, allocator, filters)); + + // Store the output position so that we can restore it if + // something goes wrong. + const size_t out_start = *out_pos; + + // Do the actual encoding and free coder's memory. + size_t in_pos = 0; + lzma_ret ret = next.code(next.coder, allocator, in, &in_pos, in_size, + out, out_pos, out_size, LZMA_FINISH); + lzma_next_end(&next, allocator); + + if (ret == LZMA_STREAM_END) { + ret = LZMA_OK; + } else { + if (ret == LZMA_OK) { + // Output buffer was too small. + assert(*out_pos == out_size); + ret = LZMA_BUF_ERROR; + } + + // Restore the output position. + *out_pos = out_start; + } + + return ret; +} diff --git a/src/liblzma/common/filter_common.c b/src/liblzma/common/filter_common.c new file mode 100644 index 0000000000..d15d9cc94f --- /dev/null +++ b/src/liblzma/common/filter_common.c @@ -0,0 +1,393 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_common.c +/// \brief Filter-specific stuff common for both encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_common.h" + + +static const struct { + /// Filter ID + lzma_vli id; + + /// Size of the filter-specific options structure + size_t options_size; + + /// True if it is OK to use this filter as non-last filter in + /// the chain. + bool non_last_ok; + + /// True if it is OK to use this filter as the last filter in + /// the chain. + bool last_ok; + + /// True if the filter may change the size of the data (that is, the + /// amount of encoded output can be different than the amount of + /// uncompressed input). + bool changes_size; + +} features[] = { +#if defined (HAVE_ENCODER_LZMA1) || defined(HAVE_DECODER_LZMA1) + { + .id = LZMA_FILTER_LZMA1, + .options_size = sizeof(lzma_options_lzma), + .non_last_ok = false, + .last_ok = true, + .changes_size = true, + }, + { + .id = LZMA_FILTER_LZMA1EXT, + .options_size = sizeof(lzma_options_lzma), + .non_last_ok = false, + .last_ok = true, + .changes_size = true, + }, +#endif +#if defined(HAVE_ENCODER_LZMA2) || defined(HAVE_DECODER_LZMA2) + { + .id = LZMA_FILTER_LZMA2, + .options_size = sizeof(lzma_options_lzma), + .non_last_ok = false, + .last_ok = true, + .changes_size = true, + }, +#endif +#if defined(HAVE_ENCODER_X86) || defined(HAVE_DECODER_X86) + { + .id = LZMA_FILTER_X86, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_POWERPC) || defined(HAVE_DECODER_POWERPC) + { + .id = LZMA_FILTER_POWERPC, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_IA64) || defined(HAVE_DECODER_IA64) + { + .id = LZMA_FILTER_IA64, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_ARM) || defined(HAVE_DECODER_ARM) + { + .id = LZMA_FILTER_ARM, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_ARMTHUMB) || defined(HAVE_DECODER_ARMTHUMB) + { + .id = LZMA_FILTER_ARMTHUMB, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_ARM64) || defined(HAVE_DECODER_ARM64) + { + .id = LZMA_FILTER_ARM64, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_SPARC) || defined(HAVE_DECODER_SPARC) + { + .id = LZMA_FILTER_SPARC, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_RISCV) || defined(HAVE_DECODER_RISCV) + { + .id = LZMA_FILTER_RISCV, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif +#if defined(HAVE_ENCODER_DELTA) || defined(HAVE_DECODER_DELTA) + { + .id = LZMA_FILTER_DELTA, + .options_size = sizeof(lzma_options_delta), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif + { + .id = LZMA_VLI_UNKNOWN + } +}; + + +extern LZMA_API(lzma_ret) +lzma_filters_copy(const lzma_filter *src, lzma_filter *real_dest, + const lzma_allocator *allocator) +{ + if (src == NULL || real_dest == NULL) + return LZMA_PROG_ERROR; + + // Use a temporary destination so that the real destination + // will never be modified if an error occurs. + lzma_filter dest[LZMA_FILTERS_MAX + 1]; + + lzma_ret ret; + size_t i; + for (i = 0; src[i].id != LZMA_VLI_UNKNOWN; ++i) { + // There must be a maximum of four filters plus + // the array terminator. + if (i == LZMA_FILTERS_MAX) { + ret = LZMA_OPTIONS_ERROR; + goto error; + } + + dest[i].id = src[i].id; + + if (src[i].options == NULL) { + dest[i].options = NULL; + } else { + // See if the filter is supported only when the + // options is not NULL. This might be convenient + // sometimes if the app is actually copying only + // a partial filter chain with a place holder ID. + // + // When options is not NULL, the Filter ID must be + // supported by us, because otherwise we don't know + // how big the options are. + size_t j; + for (j = 0; src[i].id != features[j].id; ++j) { + if (features[j].id == LZMA_VLI_UNKNOWN) { + ret = LZMA_OPTIONS_ERROR; + goto error; + } + } + + // Allocate and copy the options. + dest[i].options = lzma_alloc(features[j].options_size, + allocator); + if (dest[i].options == NULL) { + ret = LZMA_MEM_ERROR; + goto error; + } + + memcpy(dest[i].options, src[i].options, + features[j].options_size); + } + } + + // Terminate the filter array. + assert(i < LZMA_FILTERS_MAX + 1); + dest[i].id = LZMA_VLI_UNKNOWN; + dest[i].options = NULL; + + // Copy it to the caller-supplied array now that we know that + // no errors occurred. + memcpy(real_dest, dest, (i + 1) * sizeof(lzma_filter)); + + return LZMA_OK; + +error: + // Free the options which we have already allocated. + while (i-- > 0) + lzma_free(dest[i].options, allocator); + + return ret; +} + + +extern LZMA_API(void) +lzma_filters_free(lzma_filter *filters, const lzma_allocator *allocator) +{ + if (filters == NULL) + return; + + for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + if (i == LZMA_FILTERS_MAX) { + // The API says that LZMA_FILTERS_MAX + 1 is the + // maximum allowed size including the terminating + // element. Thus, we should never get here but in + // case there is a bug and we do anyway, don't go + // past the (probable) end of the array. + assert(0); + break; + } + + lzma_free(filters[i].options, allocator); + filters[i].options = NULL; + filters[i].id = LZMA_VLI_UNKNOWN; + } + + return; +} + + +extern lzma_ret +lzma_validate_chain(const lzma_filter *filters, size_t *count) +{ + // There must be at least one filter. + if (filters == NULL || filters[0].id == LZMA_VLI_UNKNOWN) + return LZMA_PROG_ERROR; + + // Number of non-last filters that may change the size of the data + // significantly (that is, more than 1-2 % or so). + size_t changes_size_count = 0; + + // True if it is OK to add a new filter after the current filter. + bool non_last_ok = true; + + // True if the last filter in the given chain is actually usable as + // the last filter. Only filters that support embedding End of Payload + // Marker can be used as the last filter in the chain. + bool last_ok = false; + + size_t i = 0; + do { + size_t j; + for (j = 0; filters[i].id != features[j].id; ++j) + if (features[j].id == LZMA_VLI_UNKNOWN) + return LZMA_OPTIONS_ERROR; + + // If the previous filter in the chain cannot be a non-last + // filter, the chain is invalid. + if (!non_last_ok) + return LZMA_OPTIONS_ERROR; + + non_last_ok = features[j].non_last_ok; + last_ok = features[j].last_ok; + changes_size_count += features[j].changes_size; + + } while (filters[++i].id != LZMA_VLI_UNKNOWN); + + // There must be 1-4 filters. The last filter must be usable as + // the last filter in the chain. A maximum of three filters are + // allowed to change the size of the data. + if (i > LZMA_FILTERS_MAX || !last_ok || changes_size_count > 3) + return LZMA_OPTIONS_ERROR; + + *count = i; + return LZMA_OK; +} + + +extern lzma_ret +lzma_raw_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *options, + lzma_filter_find coder_find, bool is_encoder) +{ + // Do some basic validation and get the number of filters. + size_t count; + return_if_error(lzma_validate_chain(options, &count)); + + // Set the filter functions and copy the options pointer. + lzma_filter_info filters[LZMA_FILTERS_MAX + 1]; + if (is_encoder) { + for (size_t i = 0; i < count; ++i) { + // The order of the filters is reversed in the + // encoder. It allows more efficient handling + // of the uncompressed data. + const size_t j = count - i - 1; + + const lzma_filter_coder *const fc + = coder_find(options[i].id); + if (fc == NULL || fc->init == NULL) + return LZMA_OPTIONS_ERROR; + + filters[j].id = options[i].id; + filters[j].init = fc->init; + filters[j].options = options[i].options; + } + } else { + for (size_t i = 0; i < count; ++i) { + const lzma_filter_coder *const fc + = coder_find(options[i].id); + if (fc == NULL || fc->init == NULL) + return LZMA_OPTIONS_ERROR; + + filters[i].id = options[i].id; + filters[i].init = fc->init; + filters[i].options = options[i].options; + } + } + + // Terminate the array. + filters[count].id = LZMA_VLI_UNKNOWN; + filters[count].init = NULL; + + // Initialize the filters. + const lzma_ret ret = lzma_next_filter_init(next, allocator, filters); + if (ret != LZMA_OK) + lzma_next_end(next, allocator); + + return ret; +} + + +extern uint64_t +lzma_raw_coder_memusage(lzma_filter_find coder_find, + const lzma_filter *filters) +{ + // The chain has to have at least one filter. + { + size_t tmp; + if (lzma_validate_chain(filters, &tmp) != LZMA_OK) + return UINT64_MAX; + } + + uint64_t total = 0; + size_t i = 0; + + do { + const lzma_filter_coder *const fc + = coder_find(filters[i].id); + if (fc == NULL) + return UINT64_MAX; // Unsupported Filter ID + + if (fc->memusage == NULL) { + // This filter doesn't have a function to calculate + // the memory usage and validate the options. Such + // filters need only little memory, so we use 1 KiB + // as a good estimate. They also accept all possible + // options, so there's no need to worry about lack + // of validation. + total += 1024; + } else { + // Call the filter-specific memory usage calculation + // function. + const uint64_t usage + = fc->memusage(filters[i].options); + if (usage == UINT64_MAX) + return UINT64_MAX; // Invalid options + + total += usage; + } + } while (filters[++i].id != LZMA_VLI_UNKNOWN); + + // Add some fixed amount of extra. It's to compensate memory usage + // of Stream, Block etc. coders, malloc() overhead, stack etc. + return total + LZMA_MEMUSAGE_BASE; +} diff --git a/src/liblzma/common/filter_common.h b/src/liblzma/common/filter_common.h new file mode 100644 index 0000000000..95f9fe2701 --- /dev/null +++ b/src/liblzma/common/filter_common.h @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_common.h +/// \brief Filter-specific stuff common for both encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_FILTER_COMMON_H +#define LZMA_FILTER_COMMON_H + +#include "common.h" + + +/// Both lzma_filter_encoder and lzma_filter_decoder begin with these members. +typedef struct { + /// Filter ID + lzma_vli id; + + /// Initializes the filter encoder and calls lzma_next_filter_init() + /// for filters + 1. + lzma_init_function init; + + /// Calculates memory usage of the encoder. If the options are + /// invalid, UINT64_MAX is returned. + uint64_t (*memusage)(const void *options); + +} lzma_filter_coder; + + +typedef const lzma_filter_coder *(*lzma_filter_find)(lzma_vli id); + + +extern lzma_ret lzma_validate_chain(const lzma_filter *filters, size_t *count); + + +extern lzma_ret lzma_raw_coder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *filters, + lzma_filter_find coder_find, bool is_encoder); + + +extern uint64_t lzma_raw_coder_memusage(lzma_filter_find coder_find, + const lzma_filter *filters); + + +#endif diff --git a/src/liblzma/common/filter_decoder.c b/src/liblzma/common/filter_decoder.c new file mode 100644 index 0000000000..cbdeb5858f --- /dev/null +++ b/src/liblzma/common/filter_decoder.c @@ -0,0 +1,214 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_decoder.c +/// \brief Filter ID mapping to filter-specific functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_decoder.h" +#include "filter_common.h" +#include "lzma_decoder.h" +#include "lzma2_decoder.h" +#include "simple_decoder.h" +#include "delta_decoder.h" + + +typedef struct { + /// Filter ID + lzma_vli id; + + /// Initializes the filter encoder and calls lzma_next_filter_init() + /// for filters + 1. + lzma_init_function init; + + /// Calculates memory usage of the encoder. If the options are + /// invalid, UINT64_MAX is returned. + uint64_t (*memusage)(const void *options); + + /// Decodes Filter Properties. + /// + /// \return - LZMA_OK: Properties decoded successfully. + /// - LZMA_OPTIONS_ERROR: Unsupported properties + /// - LZMA_MEM_ERROR: Memory allocation failed. + lzma_ret (*props_decode)( + void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size); + +} lzma_filter_decoder; + + +static const lzma_filter_decoder decoders[] = { +#ifdef HAVE_DECODER_LZMA1 + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_decoder_init, + .memusage = &lzma_lzma_decoder_memusage, + .props_decode = &lzma_lzma_props_decode, + }, + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_decoder_init, + .memusage = &lzma_lzma_decoder_memusage, + .props_decode = &lzma_lzma_props_decode, + }, +#endif +#ifdef HAVE_DECODER_LZMA2 + { + .id = LZMA_FILTER_LZMA2, + .init = &lzma_lzma2_decoder_init, + .memusage = &lzma_lzma2_decoder_memusage, + .props_decode = &lzma_lzma2_props_decode, + }, +#endif +#ifdef HAVE_DECODER_X86 + { + .id = LZMA_FILTER_X86, + .init = &lzma_simple_x86_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_POWERPC + { + .id = LZMA_FILTER_POWERPC, + .init = &lzma_simple_powerpc_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_IA64 + { + .id = LZMA_FILTER_IA64, + .init = &lzma_simple_ia64_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_ARM + { + .id = LZMA_FILTER_ARM, + .init = &lzma_simple_arm_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_ARMTHUMB + { + .id = LZMA_FILTER_ARMTHUMB, + .init = &lzma_simple_armthumb_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_ARM64 + { + .id = LZMA_FILTER_ARM64, + .init = &lzma_simple_arm64_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_SPARC + { + .id = LZMA_FILTER_SPARC, + .init = &lzma_simple_sparc_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_RISCV + { + .id = LZMA_FILTER_RISCV, + .init = &lzma_simple_riscv_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif +#ifdef HAVE_DECODER_DELTA + { + .id = LZMA_FILTER_DELTA, + .init = &lzma_delta_decoder_init, + .memusage = &lzma_delta_coder_memusage, + .props_decode = &lzma_delta_props_decode, + }, +#endif +}; + + +static const lzma_filter_decoder * +decoder_find(lzma_vli id) +{ + for (size_t i = 0; i < ARRAY_SIZE(decoders); ++i) + if (decoders[i].id == id) + return decoders + i; + + return NULL; +} + + +// lzma_filter_coder begins with the same members as lzma_filter_decoder. +// This function is a wrapper with a type that is compatible with the +// typedef of lzma_filter_find in filter_common.h. +static const lzma_filter_coder * +coder_find(lzma_vli id) +{ + return (const lzma_filter_coder *)decoder_find(id); +} + + +extern LZMA_API(lzma_bool) +lzma_filter_decoder_is_supported(lzma_vli id) +{ + return decoder_find(id) != NULL; +} + + +extern lzma_ret +lzma_raw_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *options) +{ + return lzma_raw_coder_init(next, allocator, + options, &coder_find, false); +} + + +extern LZMA_API(lzma_ret) +lzma_raw_decoder(lzma_stream *strm, const lzma_filter *options) +{ + lzma_next_strm_init(lzma_raw_decoder_init, strm, options); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} + + +extern LZMA_API(uint64_t) +lzma_raw_decoder_memusage(const lzma_filter *filters) +{ + return lzma_raw_coder_memusage(&coder_find, filters); +} + + +extern LZMA_API(lzma_ret) +lzma_properties_decode(lzma_filter *filter, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size) +{ + // Make it always NULL so that the caller can always safely free() it. + filter->options = NULL; + + const lzma_filter_decoder *const fd = decoder_find(filter->id); + if (fd == NULL) + return LZMA_OPTIONS_ERROR; + + if (fd->props_decode == NULL) + return props_size == 0 ? LZMA_OK : LZMA_OPTIONS_ERROR; + + return fd->props_decode( + &filter->options, allocator, props, props_size); +} diff --git a/src/liblzma/common/filter_decoder.h b/src/liblzma/common/filter_decoder.h new file mode 100644 index 0000000000..e610bc1f44 --- /dev/null +++ b/src/liblzma/common/filter_decoder.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_decoder.h +/// \brief Filter ID mapping to filter-specific functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_FILTER_DECODER_H +#define LZMA_FILTER_DECODER_H + +#include "common.h" + + +extern lzma_ret lzma_raw_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *options); + +#endif diff --git a/src/liblzma/common/filter_encoder.c b/src/liblzma/common/filter_encoder.c new file mode 100644 index 0000000000..bc39444898 --- /dev/null +++ b/src/liblzma/common/filter_encoder.c @@ -0,0 +1,330 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_decoder.c +/// \brief Filter ID mapping to filter-specific functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_encoder.h" +#include "filter_common.h" +#include "lzma_encoder.h" +#include "lzma2_encoder.h" +#include "simple_encoder.h" +#include "delta_encoder.h" + + +typedef struct { + /// Filter ID + lzma_vli id; + + /// Initializes the filter encoder and calls lzma_next_filter_init() + /// for filters + 1. + lzma_init_function init; + + /// Calculates memory usage of the encoder. If the options are + /// invalid, UINT64_MAX is returned. + uint64_t (*memusage)(const void *options); + + /// Calculates the recommended Uncompressed Size for .xz Blocks to + /// which the input data can be split to make multithreaded + /// encoding possible. If this is NULL, it is assumed that + /// the encoder is fast enough with single thread. If the options + /// are invalid, UINT64_MAX is returned. + uint64_t (*block_size)(const void *options); + + /// Tells the size of the Filter Properties field. If options are + /// invalid, LZMA_OPTIONS_ERROR is returned and size is set to + /// UINT32_MAX. + lzma_ret (*props_size_get)(uint32_t *size, const void *options); + + /// Some filters will always have the same size Filter Properties + /// field. If props_size_get is NULL, this value is used. + uint32_t props_size_fixed; + + /// Encodes Filter Properties. + /// + /// \return - LZMA_OK: Properties encoded successfully. + /// - LZMA_OPTIONS_ERROR: Unsupported options + /// - LZMA_PROG_ERROR: Invalid options or not enough + /// output space + lzma_ret (*props_encode)(const void *options, uint8_t *out); + +} lzma_filter_encoder; + + +static const lzma_filter_encoder encoders[] = { +#ifdef HAVE_ENCODER_LZMA1 + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_encoder_init, + .memusage = &lzma_lzma_encoder_memusage, + .block_size = NULL, // Not needed for LZMA1 + .props_size_get = NULL, + .props_size_fixed = 5, + .props_encode = &lzma_lzma_props_encode, + }, + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_encoder_init, + .memusage = &lzma_lzma_encoder_memusage, + .block_size = NULL, // Not needed for LZMA1 + .props_size_get = NULL, + .props_size_fixed = 5, + .props_encode = &lzma_lzma_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_LZMA2 + { + .id = LZMA_FILTER_LZMA2, + .init = &lzma_lzma2_encoder_init, + .memusage = &lzma_lzma2_encoder_memusage, + .block_size = &lzma_lzma2_block_size, + .props_size_get = NULL, + .props_size_fixed = 1, + .props_encode = &lzma_lzma2_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_X86 + { + .id = LZMA_FILTER_X86, + .init = &lzma_simple_x86_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_POWERPC + { + .id = LZMA_FILTER_POWERPC, + .init = &lzma_simple_powerpc_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_IA64 + { + .id = LZMA_FILTER_IA64, + .init = &lzma_simple_ia64_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_ARM + { + .id = LZMA_FILTER_ARM, + .init = &lzma_simple_arm_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_ARMTHUMB + { + .id = LZMA_FILTER_ARMTHUMB, + .init = &lzma_simple_armthumb_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_ARM64 + { + .id = LZMA_FILTER_ARM64, + .init = &lzma_simple_arm64_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_SPARC + { + .id = LZMA_FILTER_SPARC, + .init = &lzma_simple_sparc_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_RISCV + { + .id = LZMA_FILTER_RISCV, + .init = &lzma_simple_riscv_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif +#ifdef HAVE_ENCODER_DELTA + { + .id = LZMA_FILTER_DELTA, + .init = &lzma_delta_encoder_init, + .memusage = &lzma_delta_coder_memusage, + .block_size = NULL, + .props_size_get = NULL, + .props_size_fixed = 1, + .props_encode = &lzma_delta_props_encode, + }, +#endif +}; + + +static const lzma_filter_encoder * +encoder_find(lzma_vli id) +{ + for (size_t i = 0; i < ARRAY_SIZE(encoders); ++i) + if (encoders[i].id == id) + return encoders + i; + + return NULL; +} + + +// lzma_filter_coder begins with the same members as lzma_filter_encoder. +// This function is a wrapper with a type that is compatible with the +// typedef of lzma_filter_find in filter_common.h. +static const lzma_filter_coder * +coder_find(lzma_vli id) +{ + return (const lzma_filter_coder *)encoder_find(id); +} + + +extern LZMA_API(lzma_bool) +lzma_filter_encoder_is_supported(lzma_vli id) +{ + return encoder_find(id) != NULL; +} + + +extern LZMA_API(lzma_ret) +lzma_filters_update(lzma_stream *strm, const lzma_filter *filters) +{ + if (strm->internal->next.update == NULL) + return LZMA_PROG_ERROR; + + // Validate the filter chain. + if (lzma_raw_encoder_memusage(filters) == UINT64_MAX) + return LZMA_OPTIONS_ERROR; + + // The actual filter chain in the encoder is reversed. Some things + // still want the normal order chain, so we provide both. + size_t count = 1; + while (filters[count].id != LZMA_VLI_UNKNOWN) + ++count; + + lzma_filter reversed_filters[LZMA_FILTERS_MAX + 1]; + for (size_t i = 0; i < count; ++i) + reversed_filters[count - i - 1] = filters[i]; + + reversed_filters[count].id = LZMA_VLI_UNKNOWN; + + return strm->internal->next.update(strm->internal->next.coder, + strm->allocator, filters, reversed_filters); +} + + +extern lzma_ret +lzma_raw_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *filters) +{ + return lzma_raw_coder_init(next, allocator, + filters, &coder_find, true); +} + + +extern LZMA_API(lzma_ret) +lzma_raw_encoder(lzma_stream *strm, const lzma_filter *filters) +{ + lzma_next_strm_init(lzma_raw_coder_init, strm, filters, + &coder_find, true); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_SYNC_FLUSH] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} + + +extern LZMA_API(uint64_t) +lzma_raw_encoder_memusage(const lzma_filter *filters) +{ + return lzma_raw_coder_memusage(&coder_find, filters); +} + + +extern LZMA_API(uint64_t) +lzma_mt_block_size(const lzma_filter *filters) +{ + if (filters == NULL) + return UINT64_MAX; + + uint64_t max = 0; + + for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + const lzma_filter_encoder *const fe + = encoder_find(filters[i].id); + if (fe == NULL) + return UINT64_MAX; + + if (fe->block_size != NULL) { + const uint64_t size + = fe->block_size(filters[i].options); + if (size > max) + max = size; + } + } + + return max == 0 ? UINT64_MAX : max; +} + + +extern LZMA_API(lzma_ret) +lzma_properties_size(uint32_t *size, const lzma_filter *filter) +{ + const lzma_filter_encoder *const fe = encoder_find(filter->id); + if (fe == NULL) { + // Unknown filter - if the Filter ID is a proper VLI, + // return LZMA_OPTIONS_ERROR instead of LZMA_PROG_ERROR, + // because it's possible that we just don't have support + // compiled in for the requested filter. + return filter->id <= LZMA_VLI_MAX + ? LZMA_OPTIONS_ERROR : LZMA_PROG_ERROR; + } + + if (fe->props_size_get == NULL) { + // No props_size_get() function, use props_size_fixed. + *size = fe->props_size_fixed; + return LZMA_OK; + } + + return fe->props_size_get(size, filter->options); +} + + +extern LZMA_API(lzma_ret) +lzma_properties_encode(const lzma_filter *filter, uint8_t *props) +{ + const lzma_filter_encoder *const fe = encoder_find(filter->id); + if (fe == NULL) + return LZMA_PROG_ERROR; + + if (fe->props_encode == NULL) + return LZMA_OK; + + return fe->props_encode(filter->options, props); +} diff --git a/src/liblzma/common/filter_encoder.h b/src/liblzma/common/filter_encoder.h new file mode 100644 index 0000000000..88f2dafa43 --- /dev/null +++ b/src/liblzma/common/filter_encoder.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_encoder.h +/// \brief Filter ID mapping to filter-specific functions +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_FILTER_ENCODER_H +#define LZMA_FILTER_ENCODER_H + +#include "common.h" + + +extern lzma_ret lzma_raw_encoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *filters); + +#endif diff --git a/src/liblzma/common/filter_flags_decoder.c b/src/liblzma/common/filter_flags_decoder.c new file mode 100644 index 0000000000..0f5d204d47 --- /dev/null +++ b/src/liblzma/common/filter_flags_decoder.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_flags_decoder.c +/// \brief Decodes a Filter Flags field +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_decoder.h" + + +extern LZMA_API(lzma_ret) +lzma_filter_flags_decode( + lzma_filter *filter, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size) +{ + // Set the pointer to NULL so the caller can always safely free it. + filter->options = NULL; + + // Filter ID + return_if_error(lzma_vli_decode(&filter->id, NULL, + in, in_pos, in_size)); + + if (filter->id >= LZMA_FILTER_RESERVED_START) + return LZMA_DATA_ERROR; + + // Size of Properties + lzma_vli props_size; + return_if_error(lzma_vli_decode(&props_size, NULL, + in, in_pos, in_size)); + + // Filter Properties + if (in_size - *in_pos < props_size) + return LZMA_DATA_ERROR; + + const lzma_ret ret = lzma_properties_decode( + filter, allocator, in + *in_pos, props_size); + + *in_pos += props_size; + + return ret; +} diff --git a/src/liblzma/common/filter_flags_encoder.c b/src/liblzma/common/filter_flags_encoder.c new file mode 100644 index 0000000000..e1d65884fb --- /dev/null +++ b/src/liblzma/common/filter_flags_encoder.c @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file filter_flags_encoder.c +/// \brief Encodes a Filter Flags field +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_encoder.h" + + +extern LZMA_API(lzma_ret) +lzma_filter_flags_size(uint32_t *size, const lzma_filter *filter) +{ + if (filter->id >= LZMA_FILTER_RESERVED_START) + return LZMA_PROG_ERROR; + + return_if_error(lzma_properties_size(size, filter)); + + *size += lzma_vli_size(filter->id) + lzma_vli_size(*size); + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_filter_flags_encode(const lzma_filter *filter, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Filter ID + if (filter->id >= LZMA_FILTER_RESERVED_START) + return LZMA_PROG_ERROR; + + return_if_error(lzma_vli_encode(filter->id, NULL, + out, out_pos, out_size)); + + // Size of Properties + uint32_t props_size; + return_if_error(lzma_properties_size(&props_size, filter)); + return_if_error(lzma_vli_encode(props_size, NULL, + out, out_pos, out_size)); + + // Filter Properties + if (out_size - *out_pos < props_size) + return LZMA_PROG_ERROR; + + return_if_error(lzma_properties_encode(filter, out + *out_pos)); + + *out_pos += props_size; + + return LZMA_OK; +} diff --git a/src/liblzma/common/hardware_cputhreads.c b/src/liblzma/common/hardware_cputhreads.c new file mode 100644 index 0000000000..4ce852b42c --- /dev/null +++ b/src/liblzma/common/hardware_cputhreads.c @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file hardware_cputhreads.c +/// \brief Get the number of CPU threads or cores +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + +#include "tuklib_cpucores.h" + + +#ifdef HAVE_SYMBOL_VERSIONS_LINUX +// This is for compatibility with binaries linked against liblzma that +// has been patched with xz-5.2.2-compat-libs.patch from RHEL/CentOS 7. +LZMA_SYMVER_API("lzma_cputhreads@XZ_5.2.2", + uint32_t, lzma_cputhreads_522)(void) lzma_nothrow + __attribute__((__alias__("lzma_cputhreads_52"))); + +LZMA_SYMVER_API("lzma_cputhreads@@XZ_5.2", + uint32_t, lzma_cputhreads_52)(void) lzma_nothrow; + +#define lzma_cputhreads lzma_cputhreads_52 +#endif +extern LZMA_API(uint32_t) +lzma_cputhreads(void) +{ + return tuklib_cpucores(); +} diff --git a/src/liblzma/common/hardware_physmem.c b/src/liblzma/common/hardware_physmem.c new file mode 100644 index 0000000000..1bc34864e8 --- /dev/null +++ b/src/liblzma/common/hardware_physmem.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file hardware_physmem.c +/// \brief Get the total amount of physical memory (RAM) +// +// Author: Jonathan Nieder +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + +#include "tuklib_physmem.h" + + +extern LZMA_API(uint64_t) +lzma_physmem(void) +{ + // It is simpler to make lzma_physmem() a wrapper for + // tuklib_physmem() than to hack appropriate symbol visibility + // support for the tuklib modules. + return tuklib_physmem(); +} diff --git a/src/liblzma/common/index.c b/src/liblzma/common/index.c new file mode 100644 index 0000000000..6add6a6835 --- /dev/null +++ b/src/liblzma/common/index.c @@ -0,0 +1,1268 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index.c +/// \brief Handling of .xz Indexes and some other Stream information +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "index.h" +#include "stream_flags_common.h" + + +/// \brief How many Records to allocate at once +/// +/// This should be big enough to avoid making lots of tiny allocations +/// but small enough to avoid too much unused memory at once. +#define INDEX_GROUP_SIZE 512 + + +/// \brief How many Records can be allocated at once at maximum +#define PREALLOC_MAX ((SIZE_MAX - sizeof(index_group)) / sizeof(index_record)) + + +/// \brief Base structure for index_stream and index_group structures +typedef struct index_tree_node_s index_tree_node; +struct index_tree_node_s { + /// Uncompressed start offset of this Stream (relative to the + /// beginning of the file) or Block (relative to the beginning + /// of the Stream) + lzma_vli uncompressed_base; + + /// Compressed start offset of this Stream or Block + lzma_vli compressed_base; + + index_tree_node *parent; + index_tree_node *left; + index_tree_node *right; +}; + + +/// \brief AVL tree to hold index_stream or index_group structures +typedef struct { + /// Root node + index_tree_node *root; + + /// Leftmost node. Since the tree will be filled sequentially, + /// this won't change after the first node has been added to + /// the tree. + index_tree_node *leftmost; + + /// The rightmost node in the tree. Since the tree is filled + /// sequentially, this is always the node where to add the new data. + index_tree_node *rightmost; + + /// Number of nodes in the tree + uint32_t count; + +} index_tree; + + +typedef struct { + lzma_vli uncompressed_sum; + lzma_vli unpadded_sum; +} index_record; + + +typedef struct { + /// Every Record group is part of index_stream.groups tree. + index_tree_node node; + + /// Number of Blocks in this Stream before this group. + lzma_vli number_base; + + /// Number of Records that can be put in records[]. + size_t allocated; + + /// Index of the last Record in use. + size_t last; + + /// The sizes in this array are stored as cumulative sums relative + /// to the beginning of the Stream. This makes it possible to + /// use binary search in lzma_index_locate(). + /// + /// Note that the cumulative summing is done specially for + /// unpadded_sum: The previous value is rounded up to the next + /// multiple of four before adding the Unpadded Size of the new + /// Block. The total encoded size of the Blocks in the Stream + /// is records[last].unpadded_sum in the last Record group of + /// the Stream. + /// + /// For example, if the Unpadded Sizes are 39, 57, and 81, the + /// stored values are 39, 97 (40 + 57), and 181 (100 + 181). + /// The total encoded size of these Blocks is 184. + /// + /// This is a flexible array, because it makes easy to optimize + /// memory usage in case someone concatenates many Streams that + /// have only one or few Blocks. + index_record records[]; + +} index_group; + + +typedef struct { + /// Every index_stream is a node in the tree of Streams. + index_tree_node node; + + /// Number of this Stream (first one is 1) + uint32_t number; + + /// Total number of Blocks before this Stream + lzma_vli block_number_base; + + /// Record groups of this Stream are stored in a tree. + /// It's a T-tree with AVL-tree balancing. There are + /// INDEX_GROUP_SIZE Records per node by default. + /// This keeps the number of memory allocations reasonable + /// and finding a Record is fast. + index_tree groups; + + /// Number of Records in this Stream + lzma_vli record_count; + + /// Size of the List of Records field in this Stream. This is used + /// together with record_count to calculate the size of the Index + /// field and thus the total size of the Stream. + lzma_vli index_list_size; + + /// Stream Flags of this Stream. This is meaningful only if + /// the Stream Flags have been told us with lzma_index_stream_flags(). + /// Initially stream_flags.version is set to UINT32_MAX to indicate + /// that the Stream Flags are unknown. + lzma_stream_flags stream_flags; + + /// Amount of Stream Padding after this Stream. This defaults to + /// zero and can be set with lzma_index_stream_padding(). + lzma_vli stream_padding; + +} index_stream; + + +struct lzma_index_s { + /// AVL-tree containing the Stream(s). Often there is just one + /// Stream, but using a tree keeps lookups fast even when there + /// are many concatenated Streams. + index_tree streams; + + /// Uncompressed size of all the Blocks in the Stream(s) + lzma_vli uncompressed_size; + + /// Total size of all the Blocks in the Stream(s) + lzma_vli total_size; + + /// Total number of Records in all Streams in this lzma_index + lzma_vli record_count; + + /// Size of the List of Records field if all the Streams in this + /// lzma_index were packed into a single Stream (makes it simpler to + /// take many .xz files and combine them into a single Stream). + /// + /// This value together with record_count is needed to calculate + /// Backward Size that is stored into Stream Footer. + lzma_vli index_list_size; + + /// How many Records to allocate at once in lzma_index_append(). + /// This defaults to INDEX_GROUP_SIZE but can be overridden with + /// lzma_index_prealloc(). + size_t prealloc; + + /// Bitmask indicating what integrity check types have been used + /// as set by lzma_index_stream_flags(). The bit of the last Stream + /// is not included here, since it is possible to change it by + /// calling lzma_index_stream_flags() again. + uint32_t checks; +}; + + +static void +index_tree_init(index_tree *tree) +{ + tree->root = NULL; + tree->leftmost = NULL; + tree->rightmost = NULL; + tree->count = 0; + return; +} + + +/// Helper for index_tree_end() +static void +index_tree_node_end(index_tree_node *node, const lzma_allocator *allocator, + void (*free_func)(void *node, const lzma_allocator *allocator)) +{ + // The tree won't ever be very huge, so recursion should be fine. + // 20 levels in the tree is likely quite a lot already in practice. + if (node->left != NULL) + index_tree_node_end(node->left, allocator, free_func); + + if (node->right != NULL) + index_tree_node_end(node->right, allocator, free_func); + + free_func(node, allocator); + return; +} + + +/// Free the memory allocated for a tree. Each node is freed using the +/// given free_func which is either &lzma_free or &index_stream_end. +/// The latter is used to free the Record groups from each index_stream +/// before freeing the index_stream itself. +static void +index_tree_end(index_tree *tree, const lzma_allocator *allocator, + void (*free_func)(void *node, const lzma_allocator *allocator)) +{ + assert(free_func != NULL); + + if (tree->root != NULL) + index_tree_node_end(tree->root, allocator, free_func); + + return; +} + + +/// Add a new node to the tree. node->uncompressed_base and +/// node->compressed_base must have been set by the caller already. +static void +index_tree_append(index_tree *tree, index_tree_node *node) +{ + node->parent = tree->rightmost; + node->left = NULL; + node->right = NULL; + + ++tree->count; + + // Handle the special case of adding the first node. + if (tree->root == NULL) { + tree->root = node; + tree->leftmost = node; + tree->rightmost = node; + return; + } + + // The tree is always filled sequentially. + assert(tree->rightmost->uncompressed_base <= node->uncompressed_base); + assert(tree->rightmost->compressed_base < node->compressed_base); + + // Add the new node after the rightmost node. It's the correct + // place due to the reason above. + tree->rightmost->right = node; + tree->rightmost = node; + + // Balance the AVL-tree if needed. We don't need to keep the balance + // factors in nodes, because we always fill the tree sequentially, + // and thus know the state of the tree just by looking at the node + // count. From the node count we can calculate how many steps to go + // up in the tree to find the rotation root. + uint32_t up = tree->count ^ (UINT32_C(1) << bsr32(tree->count)); + if (up != 0) { + // Locate the root node for the rotation. + up = ctz32(tree->count) + 2; + do { + node = node->parent; + } while (--up > 0); + + // Rotate left using node as the rotation root. + index_tree_node *pivot = node->right; + + if (node->parent == NULL) { + tree->root = pivot; + } else { + assert(node->parent->right == node); + node->parent->right = pivot; + } + + pivot->parent = node->parent; + + node->right = pivot->left; + if (node->right != NULL) + node->right->parent = node; + + pivot->left = node; + node->parent = pivot; + } + + return; +} + + +/// Get the next node in the tree. Return NULL if there are no more nodes. +static void * +index_tree_next(const index_tree_node *node) +{ + if (node->right != NULL) { + node = node->right; + while (node->left != NULL) + node = node->left; + + return (void *)(node); + } + + while (node->parent != NULL && node->parent->right == node) + node = node->parent; + + return (void *)(node->parent); +} + + +/// Locate a node that contains the given uncompressed offset. It is +/// caller's job to check that target is not bigger than the uncompressed +/// size of the tree (the last node would be returned in that case still). +static void * +index_tree_locate(const index_tree *tree, lzma_vli target) +{ + const index_tree_node *result = NULL; + const index_tree_node *node = tree->root; + + assert(tree->leftmost == NULL + || tree->leftmost->uncompressed_base == 0); + + // Consecutive nodes may have the same uncompressed_base. + // We must pick the rightmost one. + while (node != NULL) { + if (node->uncompressed_base > target) { + node = node->left; + } else { + result = node; + node = node->right; + } + } + + return (void *)(result); +} + + +/// Allocate and initialize a new Stream using the given base offsets. +static index_stream * +index_stream_init(lzma_vli compressed_base, lzma_vli uncompressed_base, + uint32_t stream_number, lzma_vli block_number_base, + const lzma_allocator *allocator) +{ + index_stream *s = lzma_alloc(sizeof(index_stream), allocator); + if (s == NULL) + return NULL; + + s->node.uncompressed_base = uncompressed_base; + s->node.compressed_base = compressed_base; + s->node.parent = NULL; + s->node.left = NULL; + s->node.right = NULL; + + s->number = stream_number; + s->block_number_base = block_number_base; + + index_tree_init(&s->groups); + + s->record_count = 0; + s->index_list_size = 0; + s->stream_flags.version = UINT32_MAX; + s->stream_padding = 0; + + return s; +} + + +/// Free the memory allocated for a Stream and its Record groups. +static void +index_stream_end(void *node, const lzma_allocator *allocator) +{ + index_stream *s = node; + index_tree_end(&s->groups, allocator, &lzma_free); + lzma_free(s, allocator); + return; +} + + +static lzma_index * +index_init_plain(const lzma_allocator *allocator) +{ + lzma_index *i = lzma_alloc(sizeof(lzma_index), allocator); + if (i != NULL) { + index_tree_init(&i->streams); + i->uncompressed_size = 0; + i->total_size = 0; + i->record_count = 0; + i->index_list_size = 0; + i->prealloc = INDEX_GROUP_SIZE; + i->checks = 0; + } + + return i; +} + + +extern LZMA_API(lzma_index *) +lzma_index_init(const lzma_allocator *allocator) +{ + lzma_index *i = index_init_plain(allocator); + if (i == NULL) + return NULL; + + index_stream *s = index_stream_init(0, 0, 1, 0, allocator); + if (s == NULL) { + lzma_free(i, allocator); + return NULL; + } + + index_tree_append(&i->streams, &s->node); + + return i; +} + + +extern LZMA_API(void) +lzma_index_end(lzma_index *i, const lzma_allocator *allocator) +{ + // NOTE: If you modify this function, check also the bottom + // of lzma_index_cat(). + if (i != NULL) { + index_tree_end(&i->streams, allocator, &index_stream_end); + lzma_free(i, allocator); + } + + return; +} + + +extern void +lzma_index_prealloc(lzma_index *i, lzma_vli records) +{ + if (records > PREALLOC_MAX) + records = PREALLOC_MAX; + + i->prealloc = (size_t)(records); + return; +} + + +extern LZMA_API(uint64_t) +lzma_index_memusage(lzma_vli streams, lzma_vli blocks) +{ + // This calculates an upper bound that is only a little bit + // bigger than the exact maximum memory usage with the given + // parameters. + + // Typical malloc() overhead is 2 * sizeof(void *) but we take + // a little bit extra just in case. Using LZMA_MEMUSAGE_BASE + // instead would give too inaccurate estimate. + const size_t alloc_overhead = 4 * sizeof(void *); + + // Amount of memory needed for each Stream base structures. + // We assume that every Stream has at least one Block and + // thus at least one group. + const size_t stream_base = sizeof(index_stream) + + sizeof(index_group) + 2 * alloc_overhead; + + // Amount of memory needed per group. + const size_t group_base = sizeof(index_group) + + INDEX_GROUP_SIZE * sizeof(index_record) + + alloc_overhead; + + // Number of groups. There may actually be more, but that overhead + // has been taken into account in stream_base already. + const lzma_vli groups + = (blocks + INDEX_GROUP_SIZE - 1) / INDEX_GROUP_SIZE; + + // Memory used by index_stream and index_group structures. + const uint64_t streams_mem = streams * stream_base; + const uint64_t groups_mem = groups * group_base; + + // Memory used by the base structure. + const uint64_t index_base = sizeof(lzma_index) + alloc_overhead; + + // Validate the arguments and catch integer overflows. + // Maximum number of Streams is "only" UINT32_MAX, because + // that limit is used by the tree containing the Streams. + const uint64_t limit = UINT64_MAX - index_base; + if (streams == 0 || streams > UINT32_MAX || blocks > LZMA_VLI_MAX + || streams > limit / stream_base + || groups > limit / group_base + || limit - streams_mem < groups_mem) + return UINT64_MAX; + + return index_base + streams_mem + groups_mem; +} + + +extern LZMA_API(uint64_t) +lzma_index_memused(const lzma_index *i) +{ + return lzma_index_memusage(i->streams.count, i->record_count); +} + + +extern LZMA_API(lzma_vli) +lzma_index_block_count(const lzma_index *i) +{ + return i->record_count; +} + + +extern LZMA_API(lzma_vli) +lzma_index_stream_count(const lzma_index *i) +{ + return i->streams.count; +} + + +extern LZMA_API(lzma_vli) +lzma_index_size(const lzma_index *i) +{ + return index_size(i->record_count, i->index_list_size); +} + + +extern LZMA_API(lzma_vli) +lzma_index_total_size(const lzma_index *i) +{ + return i->total_size; +} + + +extern LZMA_API(lzma_vli) +lzma_index_stream_size(const lzma_index *i) +{ + // Stream Header + Blocks + Index + Stream Footer + return LZMA_STREAM_HEADER_SIZE + i->total_size + + index_size(i->record_count, i->index_list_size) + + LZMA_STREAM_HEADER_SIZE; +} + + +static lzma_vli +index_file_size(lzma_vli compressed_base, lzma_vli unpadded_sum, + lzma_vli record_count, lzma_vli index_list_size, + lzma_vli stream_padding) +{ + // Earlier Streams and Stream Paddings + Stream Header + // + Blocks + Index + Stream Footer + Stream Padding + // + // This might go over LZMA_VLI_MAX due to too big unpadded_sum + // when this function is used in lzma_index_append(). + lzma_vli file_size = compressed_base + 2 * LZMA_STREAM_HEADER_SIZE + + stream_padding + vli_ceil4(unpadded_sum); + if (file_size > LZMA_VLI_MAX) + return LZMA_VLI_UNKNOWN; + + // The same applies here. + file_size += index_size(record_count, index_list_size); + if (file_size > LZMA_VLI_MAX) + return LZMA_VLI_UNKNOWN; + + return file_size; +} + + +extern LZMA_API(lzma_vli) +lzma_index_file_size(const lzma_index *i) +{ + const index_stream *s = (const index_stream *)(i->streams.rightmost); + const index_group *g = (const index_group *)(s->groups.rightmost); + return index_file_size(s->node.compressed_base, + g == NULL ? 0 : g->records[g->last].unpadded_sum, + s->record_count, s->index_list_size, + s->stream_padding); +} + + +extern LZMA_API(lzma_vli) +lzma_index_uncompressed_size(const lzma_index *i) +{ + return i->uncompressed_size; +} + + +extern LZMA_API(uint32_t) +lzma_index_checks(const lzma_index *i) +{ + uint32_t checks = i->checks; + + // Get the type of the Check of the last Stream too. + const index_stream *s = (const index_stream *)(i->streams.rightmost); + if (s->stream_flags.version != UINT32_MAX) + checks |= UINT32_C(1) << s->stream_flags.check; + + return checks; +} + + +extern uint32_t +lzma_index_padding_size(const lzma_index *i) +{ + return (LZMA_VLI_C(4) - index_size_unpadded( + i->record_count, i->index_list_size)) & 3; +} + + +extern LZMA_API(lzma_ret) +lzma_index_stream_flags(lzma_index *i, const lzma_stream_flags *stream_flags) +{ + if (i == NULL || stream_flags == NULL) + return LZMA_PROG_ERROR; + + // Validate the Stream Flags. + return_if_error(lzma_stream_flags_compare( + stream_flags, stream_flags)); + + index_stream *s = (index_stream *)(i->streams.rightmost); + s->stream_flags = *stream_flags; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_index_stream_padding(lzma_index *i, lzma_vli stream_padding) +{ + if (i == NULL || stream_padding > LZMA_VLI_MAX + || (stream_padding & 3) != 0) + return LZMA_PROG_ERROR; + + index_stream *s = (index_stream *)(i->streams.rightmost); + + // Check that the new value won't make the file grow too big. + const lzma_vli old_stream_padding = s->stream_padding; + s->stream_padding = 0; + if (lzma_index_file_size(i) + stream_padding > LZMA_VLI_MAX) { + s->stream_padding = old_stream_padding; + return LZMA_DATA_ERROR; + } + + s->stream_padding = stream_padding; + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_index_append(lzma_index *i, const lzma_allocator *allocator, + lzma_vli unpadded_size, lzma_vli uncompressed_size) +{ + // Validate. + if (i == NULL || unpadded_size < UNPADDED_SIZE_MIN + || unpadded_size > UNPADDED_SIZE_MAX + || uncompressed_size > LZMA_VLI_MAX) + return LZMA_PROG_ERROR; + + index_stream *s = (index_stream *)(i->streams.rightmost); + index_group *g = (index_group *)(s->groups.rightmost); + + const lzma_vli compressed_base = g == NULL ? 0 + : vli_ceil4(g->records[g->last].unpadded_sum); + const lzma_vli uncompressed_base = g == NULL ? 0 + : g->records[g->last].uncompressed_sum; + const uint32_t index_list_size_add = lzma_vli_size(unpadded_size) + + lzma_vli_size(uncompressed_size); + + // Check that uncompressed size will not overflow. + if (uncompressed_base + uncompressed_size > LZMA_VLI_MAX) + return LZMA_DATA_ERROR; + + // Check that the new unpadded sum will not overflow. This is + // checked again in index_file_size(), but the unpadded sum is + // passed to vli_ceil4() which expects a valid lzma_vli value. + if (compressed_base + unpadded_size > UNPADDED_SIZE_MAX) + return LZMA_DATA_ERROR; + + // Check that the file size will stay within limits. + if (index_file_size(s->node.compressed_base, + compressed_base + unpadded_size, s->record_count + 1, + s->index_list_size + index_list_size_add, + s->stream_padding) == LZMA_VLI_UNKNOWN) + return LZMA_DATA_ERROR; + + // The size of the Index field must not exceed the maximum value + // that can be stored in the Backward Size field. + if (index_size(i->record_count + 1, + i->index_list_size + index_list_size_add) + > LZMA_BACKWARD_SIZE_MAX) + return LZMA_DATA_ERROR; + + if (g != NULL && g->last + 1 < g->allocated) { + // There is space in the last group at least for one Record. + ++g->last; + } else { + // We need to allocate a new group. + g = lzma_alloc(sizeof(index_group) + + i->prealloc * sizeof(index_record), + allocator); + if (g == NULL) + return LZMA_MEM_ERROR; + + g->last = 0; + g->allocated = i->prealloc; + + // Reset prealloc so that if the application happens to + // add new Records, the allocation size will be sane. + i->prealloc = INDEX_GROUP_SIZE; + + // Set the start offsets of this group. + g->node.uncompressed_base = uncompressed_base; + g->node.compressed_base = compressed_base; + g->number_base = s->record_count + 1; + + // Add the new group to the Stream. + index_tree_append(&s->groups, &g->node); + } + + // Add the new Record to the group. + g->records[g->last].uncompressed_sum + = uncompressed_base + uncompressed_size; + g->records[g->last].unpadded_sum + = compressed_base + unpadded_size; + + // Update the totals. + ++s->record_count; + s->index_list_size += index_list_size_add; + + i->total_size += vli_ceil4(unpadded_size); + i->uncompressed_size += uncompressed_size; + ++i->record_count; + i->index_list_size += index_list_size_add; + + return LZMA_OK; +} + + +/// Structure to pass info to index_cat_helper() +typedef struct { + /// Uncompressed size of the destination + lzma_vli uncompressed_size; + + /// Compressed file size of the destination + lzma_vli file_size; + + /// Same as above but for Block numbers + lzma_vli block_number_add; + + /// Number of Streams that were in the destination index before we + /// started appending new Streams from the source index. This is + /// used to fix the Stream numbering. + uint32_t stream_number_add; + + /// Destination index' Stream tree + index_tree *streams; + +} index_cat_info; + + +/// Add the Stream nodes from the source index to dest using recursion. +/// Simplest iterative traversal of the source tree wouldn't work, because +/// we update the pointers in nodes when moving them to the destination tree. +static void +index_cat_helper(const index_cat_info *info, index_stream *this) +{ + index_stream *left = (index_stream *)(this->node.left); + index_stream *right = (index_stream *)(this->node.right); + + if (left != NULL) + index_cat_helper(info, left); + + this->node.uncompressed_base += info->uncompressed_size; + this->node.compressed_base += info->file_size; + this->number += info->stream_number_add; + this->block_number_base += info->block_number_add; + index_tree_append(info->streams, &this->node); + + if (right != NULL) + index_cat_helper(info, right); + + return; +} + + +extern LZMA_API(lzma_ret) +lzma_index_cat(lzma_index *restrict dest, lzma_index *restrict src, + const lzma_allocator *allocator) +{ + if (dest == NULL || src == NULL) + return LZMA_PROG_ERROR; + + const lzma_vli dest_file_size = lzma_index_file_size(dest); + + // Check that we don't exceed the file size limits. + if (dest_file_size + lzma_index_file_size(src) > LZMA_VLI_MAX + || dest->uncompressed_size + src->uncompressed_size + > LZMA_VLI_MAX) + return LZMA_DATA_ERROR; + + // Check that the encoded size of the combined lzma_indexes stays + // within limits. In theory, this should be done only if we know + // that the user plans to actually combine the Streams and thus + // construct a single Index (probably rare). However, exceeding + // this limit is quite theoretical, so we do this check always + // to simplify things elsewhere. + { + const lzma_vli dest_size = index_size_unpadded( + dest->record_count, dest->index_list_size); + const lzma_vli src_size = index_size_unpadded( + src->record_count, src->index_list_size); + if (vli_ceil4(dest_size + src_size) > LZMA_BACKWARD_SIZE_MAX) + return LZMA_DATA_ERROR; + } + + // Optimize the last group to minimize memory usage. Allocation has + // to be done before modifying dest or src. + { + index_stream *s = (index_stream *)(dest->streams.rightmost); + index_group *g = (index_group *)(s->groups.rightmost); + if (g != NULL && g->last + 1 < g->allocated) { + assert(g->node.left == NULL); + assert(g->node.right == NULL); + + index_group *newg = lzma_alloc(sizeof(index_group) + + (g->last + 1) + * sizeof(index_record), + allocator); + if (newg == NULL) + return LZMA_MEM_ERROR; + + newg->node = g->node; + newg->allocated = g->last + 1; + newg->last = g->last; + newg->number_base = g->number_base; + + memcpy(newg->records, g->records, newg->allocated + * sizeof(index_record)); + + if (g->node.parent != NULL) { + assert(g->node.parent->right == &g->node); + g->node.parent->right = &newg->node; + } + + if (s->groups.leftmost == &g->node) { + assert(s->groups.root == &g->node); + s->groups.leftmost = &newg->node; + s->groups.root = &newg->node; + } + + assert(s->groups.rightmost == &g->node); + s->groups.rightmost = &newg->node; + + lzma_free(g, allocator); + + // NOTE: newg isn't leaked here because + // newg == (void *)&newg->node. + } + } + + // dest->checks includes the check types of all except the last Stream + // in dest. Set the bit for the check type of the last Stream now so + // that it won't get lost when Stream(s) from src are appended to dest. + dest->checks = lzma_index_checks(dest); + + // Add all the Streams from src to dest. Update the base offsets + // of each Stream from src. + const index_cat_info info = { + .uncompressed_size = dest->uncompressed_size, + .file_size = dest_file_size, + .stream_number_add = dest->streams.count, + .block_number_add = dest->record_count, + .streams = &dest->streams, + }; + index_cat_helper(&info, (index_stream *)(src->streams.root)); + + // Update info about all the combined Streams. + dest->uncompressed_size += src->uncompressed_size; + dest->total_size += src->total_size; + dest->record_count += src->record_count; + dest->index_list_size += src->index_list_size; + dest->checks |= src->checks; + + // There's nothing else left in src than the base structure. + lzma_free(src, allocator); + + return LZMA_OK; +} + + +/// Duplicate an index_stream. +static index_stream * +index_dup_stream(const index_stream *src, const lzma_allocator *allocator) +{ + // Catch a somewhat theoretical integer overflow. + if (src->record_count > PREALLOC_MAX) + return NULL; + + // Allocate and initialize a new Stream. + index_stream *dest = index_stream_init(src->node.compressed_base, + src->node.uncompressed_base, src->number, + src->block_number_base, allocator); + if (dest == NULL) + return NULL; + + // Copy the overall information. + dest->record_count = src->record_count; + dest->index_list_size = src->index_list_size; + dest->stream_flags = src->stream_flags; + dest->stream_padding = src->stream_padding; + + // Return if there are no groups to duplicate. + if (src->groups.leftmost == NULL) + return dest; + + // Allocate memory for the Records. We put all the Records into + // a single group. It's simplest and also tends to make + // lzma_index_locate() a little bit faster with very big Indexes. + index_group *destg = lzma_alloc(sizeof(index_group) + + src->record_count * sizeof(index_record), + allocator); + if (destg == NULL) { + index_stream_end(dest, allocator); + return NULL; + } + + // Initialize destg. + destg->node.uncompressed_base = 0; + destg->node.compressed_base = 0; + destg->number_base = 1; + destg->allocated = src->record_count; + destg->last = src->record_count - 1; + + // Go through all the groups in src and copy the Records into destg. + const index_group *srcg = (const index_group *)(src->groups.leftmost); + size_t i = 0; + do { + memcpy(destg->records + i, srcg->records, + (srcg->last + 1) * sizeof(index_record)); + i += srcg->last + 1; + srcg = index_tree_next(&srcg->node); + } while (srcg != NULL); + + assert(i == destg->allocated); + + // Add the group to the new Stream. + index_tree_append(&dest->groups, &destg->node); + + return dest; +} + + +extern LZMA_API(lzma_index *) +lzma_index_dup(const lzma_index *src, const lzma_allocator *allocator) +{ + // Allocate the base structure (no initial Stream). + lzma_index *dest = index_init_plain(allocator); + if (dest == NULL) + return NULL; + + // Copy the totals. + dest->uncompressed_size = src->uncompressed_size; + dest->total_size = src->total_size; + dest->record_count = src->record_count; + dest->index_list_size = src->index_list_size; + + // Copy the Streams and the groups in them. + const index_stream *srcstream + = (const index_stream *)(src->streams.leftmost); + do { + index_stream *deststream = index_dup_stream( + srcstream, allocator); + if (deststream == NULL) { + lzma_index_end(dest, allocator); + return NULL; + } + + index_tree_append(&dest->streams, &deststream->node); + + srcstream = index_tree_next(&srcstream->node); + } while (srcstream != NULL); + + return dest; +} + + +/// Indexing for lzma_index_iter.internal[] +enum { + ITER_INDEX, + ITER_STREAM, + ITER_GROUP, + ITER_RECORD, + ITER_METHOD, +}; + + +/// Values for lzma_index_iter.internal[ITER_METHOD].s +enum { + ITER_METHOD_NORMAL, + ITER_METHOD_NEXT, + ITER_METHOD_LEFTMOST, +}; + + +static void +iter_set_info(lzma_index_iter *iter) +{ + const lzma_index *i = iter->internal[ITER_INDEX].p; + const index_stream *stream = iter->internal[ITER_STREAM].p; + const index_group *group = iter->internal[ITER_GROUP].p; + const size_t record = iter->internal[ITER_RECORD].s; + + // lzma_index_iter.internal must not contain a pointer to the last + // group in the index, because that may be reallocated by + // lzma_index_cat(). + if (group == NULL) { + // There are no groups. + assert(stream->groups.root == NULL); + iter->internal[ITER_METHOD].s = ITER_METHOD_LEFTMOST; + + } else if (i->streams.rightmost != &stream->node + || stream->groups.rightmost != &group->node) { + // The group is not not the last group in the index. + iter->internal[ITER_METHOD].s = ITER_METHOD_NORMAL; + + } else if (stream->groups.leftmost != &group->node) { + // The group isn't the only group in the Stream, thus we + // know that it must have a parent group i.e. it's not + // the root node. + assert(stream->groups.root != &group->node); + assert(group->node.parent->right == &group->node); + iter->internal[ITER_METHOD].s = ITER_METHOD_NEXT; + iter->internal[ITER_GROUP].p = group->node.parent; + + } else { + // The Stream has only one group. + assert(stream->groups.root == &group->node); + assert(group->node.parent == NULL); + iter->internal[ITER_METHOD].s = ITER_METHOD_LEFTMOST; + iter->internal[ITER_GROUP].p = NULL; + } + + // NOTE: lzma_index_iter.stream.number is lzma_vli but we use uint32_t + // internally. + iter->stream.number = stream->number; + iter->stream.block_count = stream->record_count; + iter->stream.compressed_offset = stream->node.compressed_base; + iter->stream.uncompressed_offset = stream->node.uncompressed_base; + + // iter->stream.flags will be NULL if the Stream Flags haven't been + // set with lzma_index_stream_flags(). + iter->stream.flags = stream->stream_flags.version == UINT32_MAX + ? NULL : &stream->stream_flags; + iter->stream.padding = stream->stream_padding; + + if (stream->groups.rightmost == NULL) { + // Stream has no Blocks. + iter->stream.compressed_size = index_size(0, 0) + + 2 * LZMA_STREAM_HEADER_SIZE; + iter->stream.uncompressed_size = 0; + } else { + const index_group *g = (const index_group *)( + stream->groups.rightmost); + + // Stream Header + Stream Footer + Index + Blocks + iter->stream.compressed_size = 2 * LZMA_STREAM_HEADER_SIZE + + index_size(stream->record_count, + stream->index_list_size) + + vli_ceil4(g->records[g->last].unpadded_sum); + iter->stream.uncompressed_size + = g->records[g->last].uncompressed_sum; + } + + if (group != NULL) { + iter->block.number_in_stream = group->number_base + record; + iter->block.number_in_file = iter->block.number_in_stream + + stream->block_number_base; + + iter->block.compressed_stream_offset + = record == 0 ? group->node.compressed_base + : vli_ceil4(group->records[ + record - 1].unpadded_sum); + iter->block.uncompressed_stream_offset + = record == 0 ? group->node.uncompressed_base + : group->records[record - 1].uncompressed_sum; + + iter->block.uncompressed_size + = group->records[record].uncompressed_sum + - iter->block.uncompressed_stream_offset; + iter->block.unpadded_size + = group->records[record].unpadded_sum + - iter->block.compressed_stream_offset; + iter->block.total_size = vli_ceil4(iter->block.unpadded_size); + + iter->block.compressed_stream_offset + += LZMA_STREAM_HEADER_SIZE; + + iter->block.compressed_file_offset + = iter->block.compressed_stream_offset + + iter->stream.compressed_offset; + iter->block.uncompressed_file_offset + = iter->block.uncompressed_stream_offset + + iter->stream.uncompressed_offset; + } + + return; +} + + +extern LZMA_API(void) +lzma_index_iter_init(lzma_index_iter *iter, const lzma_index *i) +{ + iter->internal[ITER_INDEX].p = i; + lzma_index_iter_rewind(iter); + return; +} + + +extern LZMA_API(void) +lzma_index_iter_rewind(lzma_index_iter *iter) +{ + iter->internal[ITER_STREAM].p = NULL; + iter->internal[ITER_GROUP].p = NULL; + iter->internal[ITER_RECORD].s = 0; + iter->internal[ITER_METHOD].s = ITER_METHOD_NORMAL; + return; +} + + +extern LZMA_API(lzma_bool) +lzma_index_iter_next(lzma_index_iter *iter, lzma_index_iter_mode mode) +{ + // Catch unsupported mode values. + if ((unsigned int)(mode) > LZMA_INDEX_ITER_NONEMPTY_BLOCK) + return true; + + const lzma_index *i = iter->internal[ITER_INDEX].p; + const index_stream *stream = iter->internal[ITER_STREAM].p; + const index_group *group = NULL; + size_t record = iter->internal[ITER_RECORD].s; + + // If we are being asked for the next Stream, leave group to NULL + // so that the rest of the this function thinks that this Stream + // has no groups and will thus go to the next Stream. + if (mode != LZMA_INDEX_ITER_STREAM) { + // Get the pointer to the current group. See iter_set_inf() + // for explanation. + switch (iter->internal[ITER_METHOD].s) { + case ITER_METHOD_NORMAL: + group = iter->internal[ITER_GROUP].p; + break; + + case ITER_METHOD_NEXT: + group = index_tree_next(iter->internal[ITER_GROUP].p); + break; + + case ITER_METHOD_LEFTMOST: + group = (const index_group *)( + stream->groups.leftmost); + break; + } + } + +again: + if (stream == NULL) { + // We at the beginning of the lzma_index. + // Locate the first Stream. + stream = (const index_stream *)(i->streams.leftmost); + if (mode >= LZMA_INDEX_ITER_BLOCK) { + // Since we are being asked to return information + // about the first a Block, skip Streams that have + // no Blocks. + while (stream->groups.leftmost == NULL) { + stream = index_tree_next(&stream->node); + if (stream == NULL) + return true; + } + } + + // Start from the first Record in the Stream. + group = (const index_group *)(stream->groups.leftmost); + record = 0; + + } else if (group != NULL && record < group->last) { + // The next Record is in the same group. + ++record; + + } else { + // This group has no more Records or this Stream has + // no Blocks at all. + record = 0; + + // If group is not NULL, this Stream has at least one Block + // and thus at least one group. Find the next group. + if (group != NULL) + group = index_tree_next(&group->node); + + if (group == NULL) { + // This Stream has no more Records. Find the next + // Stream. If we are being asked to return information + // about a Block, we skip empty Streams. + do { + stream = index_tree_next(&stream->node); + if (stream == NULL) + return true; + } while (mode >= LZMA_INDEX_ITER_BLOCK + && stream->groups.leftmost == NULL); + + group = (const index_group *)( + stream->groups.leftmost); + } + } + + if (mode == LZMA_INDEX_ITER_NONEMPTY_BLOCK) { + // We need to look for the next Block again if this Block + // is empty. + if (record == 0) { + if (group->node.uncompressed_base + == group->records[0].uncompressed_sum) + goto again; + } else if (group->records[record - 1].uncompressed_sum + == group->records[record].uncompressed_sum) { + goto again; + } + } + + iter->internal[ITER_STREAM].p = stream; + iter->internal[ITER_GROUP].p = group; + iter->internal[ITER_RECORD].s = record; + + iter_set_info(iter); + + return false; +} + + +extern LZMA_API(lzma_bool) +lzma_index_iter_locate(lzma_index_iter *iter, lzma_vli target) +{ + const lzma_index *i = iter->internal[ITER_INDEX].p; + + // If the target is past the end of the file, return immediately. + if (i->uncompressed_size <= target) + return true; + + // Locate the Stream containing the target offset. + const index_stream *stream = index_tree_locate(&i->streams, target); + assert(stream != NULL); + target -= stream->node.uncompressed_base; + + // Locate the group containing the target offset. + const index_group *group = index_tree_locate(&stream->groups, target); + assert(group != NULL); + + // Use binary search to locate the exact Record. It is the first + // Record whose uncompressed_sum is greater than target. + // This is because we want the rightmost Record that fulfills the + // search criterion. It is possible that there are empty Blocks; + // we don't want to return them. + size_t left = 0; + size_t right = group->last; + + while (left < right) { + const size_t pos = left + (right - left) / 2; + if (group->records[pos].uncompressed_sum <= target) + left = pos + 1; + else + right = pos; + } + + iter->internal[ITER_STREAM].p = stream; + iter->internal[ITER_GROUP].p = group; + iter->internal[ITER_RECORD].s = left; + + iter_set_info(iter); + + return false; +} diff --git a/src/liblzma/common/index.h b/src/liblzma/common/index.h new file mode 100644 index 0000000000..007e1188f2 --- /dev/null +++ b/src/liblzma/common/index.h @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index.h +/// \brief Handling of Index +/// \note This header file does not include common.h or lzma.h because +/// this file is needed by both liblzma internally and by the +/// tests. Including common.h will include and define many things +/// the tests do not need and prevents issues with header file +/// include order. This way, if lzma.h or common.h are not +/// included before this file it will break on every OS instead +/// of causing more subtle errors. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_INDEX_H +#define LZMA_INDEX_H + + +/// Minimum Unpadded Size +#define UNPADDED_SIZE_MIN LZMA_VLI_C(5) + +/// Maximum Unpadded Size +#define UNPADDED_SIZE_MAX (LZMA_VLI_MAX & ~LZMA_VLI_C(3)) + +/// Index Indicator based on xz specification +#define INDEX_INDICATOR 0 + + +/// Get the size of the Index Padding field. This is needed by Index encoder +/// and decoder, but applications should have no use for this. +extern uint32_t lzma_index_padding_size(const lzma_index *i); + + +/// Set for how many Records to allocate memory the next time +/// lzma_index_append() needs to allocate space for a new Record. +/// This is used only by the Index decoder. +extern void lzma_index_prealloc(lzma_index *i, lzma_vli records); + + +/// Round the variable-length integer to the next multiple of four. +static inline lzma_vli +vli_ceil4(lzma_vli vli) +{ + assert(vli <= UNPADDED_SIZE_MAX); + return (vli + 3) & ~LZMA_VLI_C(3); +} + + +/// Calculate the size of the Index field excluding Index Padding +static inline lzma_vli +index_size_unpadded(lzma_vli count, lzma_vli index_list_size) +{ + // Index Indicator + Number of Records + List of Records + CRC32 + return 1 + lzma_vli_size(count) + index_list_size + 4; +} + + +/// Calculate the size of the Index field including Index Padding +static inline lzma_vli +index_size(lzma_vli count, lzma_vli index_list_size) +{ + return vli_ceil4(index_size_unpadded(count, index_list_size)); +} + + +/// Calculate the total size of the Stream +static inline lzma_vli +index_stream_size(lzma_vli blocks_size, + lzma_vli count, lzma_vli index_list_size) +{ + return LZMA_STREAM_HEADER_SIZE + blocks_size + + index_size(count, index_list_size) + + LZMA_STREAM_HEADER_SIZE; +} + +#endif diff --git a/src/liblzma/common/index_decoder.c b/src/liblzma/common/index_decoder.c new file mode 100644 index 0000000000..4eab56d942 --- /dev/null +++ b/src/liblzma/common/index_decoder.c @@ -0,0 +1,369 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index_decoder.c +/// \brief Decodes the Index field +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "index_decoder.h" +#include "check.h" + + +typedef struct { + enum { + SEQ_INDICATOR, + SEQ_COUNT, + SEQ_MEMUSAGE, + SEQ_UNPADDED, + SEQ_UNCOMPRESSED, + SEQ_PADDING_INIT, + SEQ_PADDING, + SEQ_CRC32, + } sequence; + + /// Memory usage limit + uint64_t memlimit; + + /// Target Index + lzma_index *index; + + /// Pointer give by the application, which is set after + /// successful decoding. + lzma_index **index_ptr; + + /// Number of Records left to decode. + lzma_vli count; + + /// The most recent Unpadded Size field + lzma_vli unpadded_size; + + /// The most recent Uncompressed Size field + lzma_vli uncompressed_size; + + /// Position in integers + size_t pos; + + /// CRC32 of the List of Records field + uint32_t crc32; +} lzma_index_coder; + + +static lzma_ret +index_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, + uint8_t *restrict out lzma_attribute((__unused__)), + size_t *restrict out_pos lzma_attribute((__unused__)), + size_t out_size lzma_attribute((__unused__)), + lzma_action action lzma_attribute((__unused__))) +{ + lzma_index_coder *coder = coder_ptr; + + // Similar optimization as in index_encoder.c + const size_t in_start = *in_pos; + lzma_ret ret = LZMA_OK; + + while (*in_pos < in_size) + switch (coder->sequence) { + case SEQ_INDICATOR: + // Return LZMA_DATA_ERROR instead of e.g. LZMA_PROG_ERROR or + // LZMA_FORMAT_ERROR, because a typical usage case for Index + // decoder is when parsing the Stream backwards. If seeking + // backward from the Stream Footer gives us something that + // doesn't begin with Index Indicator, the file is considered + // corrupt, not "programming error" or "unrecognized file + // format". One could argue that the application should + // verify the Index Indicator before trying to decode the + // Index, but well, I suppose it is simpler this way. + if (in[(*in_pos)++] != INDEX_INDICATOR) + return LZMA_DATA_ERROR; + + coder->sequence = SEQ_COUNT; + break; + + case SEQ_COUNT: + ret = lzma_vli_decode(&coder->count, &coder->pos, + in, in_pos, in_size); + if (ret != LZMA_STREAM_END) + goto out; + + coder->pos = 0; + coder->sequence = SEQ_MEMUSAGE; + FALLTHROUGH; + + case SEQ_MEMUSAGE: + if (lzma_index_memusage(1, coder->count) > coder->memlimit) { + ret = LZMA_MEMLIMIT_ERROR; + goto out; + } + + // Tell the Index handling code how many Records this + // Index has to allow it to allocate memory more efficiently. + lzma_index_prealloc(coder->index, coder->count); + + ret = LZMA_OK; + coder->sequence = coder->count == 0 + ? SEQ_PADDING_INIT : SEQ_UNPADDED; + break; + + case SEQ_UNPADDED: + case SEQ_UNCOMPRESSED: { + lzma_vli *size = coder->sequence == SEQ_UNPADDED + ? &coder->unpadded_size + : &coder->uncompressed_size; + + ret = lzma_vli_decode(size, &coder->pos, + in, in_pos, in_size); + if (ret != LZMA_STREAM_END) + goto out; + + ret = LZMA_OK; + coder->pos = 0; + + if (coder->sequence == SEQ_UNPADDED) { + // Validate that encoded Unpadded Size isn't too small + // or too big. + if (coder->unpadded_size < UNPADDED_SIZE_MIN + || coder->unpadded_size + > UNPADDED_SIZE_MAX) + return LZMA_DATA_ERROR; + + coder->sequence = SEQ_UNCOMPRESSED; + } else { + // Add the decoded Record to the Index. + return_if_error(lzma_index_append( + coder->index, allocator, + coder->unpadded_size, + coder->uncompressed_size)); + + // Check if this was the last Record. + coder->sequence = --coder->count == 0 + ? SEQ_PADDING_INIT + : SEQ_UNPADDED; + } + + break; + } + + case SEQ_PADDING_INIT: + coder->pos = lzma_index_padding_size(coder->index); + coder->sequence = SEQ_PADDING; + FALLTHROUGH; + + case SEQ_PADDING: + if (coder->pos > 0) { + --coder->pos; + if (in[(*in_pos)++] != 0x00) + return LZMA_DATA_ERROR; + + break; + } + + // Finish the CRC32 calculation. + coder->crc32 = lzma_crc32(in + in_start, + *in_pos - in_start, coder->crc32); + + coder->sequence = SEQ_CRC32; + FALLTHROUGH; + + case SEQ_CRC32: + do { + if (*in_pos == in_size) + return LZMA_OK; + + if (((coder->crc32 >> (coder->pos * 8)) & 0xFF) + != in[(*in_pos)++]) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + return LZMA_DATA_ERROR; +#endif + } + + } while (++coder->pos < 4); + + // Decoding was successful, now we can let the application + // see the decoded Index. + *coder->index_ptr = coder->index; + + // Make index NULL so we don't free it unintentionally. + coder->index = NULL; + + return LZMA_STREAM_END; + + default: + assert(0); + return LZMA_PROG_ERROR; + } + +out: + // Update the CRC32. + // + // Avoid null pointer + 0 (undefined behavior) in "in + in_start". + // In such a case we had no input and thus in_used == 0. + { + const size_t in_used = *in_pos - in_start; + if (in_used > 0) + coder->crc32 = lzma_crc32(in + in_start, + in_used, coder->crc32); + } + + return ret; +} + + +static void +index_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_index_coder *coder = coder_ptr; + lzma_index_end(coder->index, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +index_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_index_coder *coder = coder_ptr; + + *memusage = lzma_index_memusage(1, coder->count); + *old_memlimit = coder->memlimit; + + if (new_memlimit != 0) { + if (new_memlimit < *memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +static lzma_ret +index_decoder_reset(lzma_index_coder *coder, const lzma_allocator *allocator, + lzma_index **i, uint64_t memlimit) +{ + // Remember the pointer given by the application. We will set it + // to point to the decoded Index only if decoding is successful. + // Before that, keep it NULL so that applications can always safely + // pass it to lzma_index_end() no matter did decoding succeed or not. + coder->index_ptr = i; + *i = NULL; + + // We always allocate a new lzma_index. + coder->index = lzma_index_init(allocator); + if (coder->index == NULL) + return LZMA_MEM_ERROR; + + // Initialize the rest. + coder->sequence = SEQ_INDICATOR; + coder->memlimit = my_max(1, memlimit); + coder->count = 0; // Needs to be initialized due to _memconfig(). + coder->pos = 0; + coder->crc32 = 0; + + return LZMA_OK; +} + + +extern lzma_ret +lzma_index_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + lzma_index **i, uint64_t memlimit) +{ + lzma_next_coder_init(&lzma_index_decoder_init, next, allocator); + + if (i == NULL) + return LZMA_PROG_ERROR; + + lzma_index_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_index_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &index_decode; + next->end = &index_decoder_end; + next->memconfig = &index_decoder_memconfig; + coder->index = NULL; + } else { + lzma_index_end(coder->index, allocator); + } + + return index_decoder_reset(coder, allocator, i, memlimit); +} + + +extern LZMA_API(lzma_ret) +lzma_index_decoder(lzma_stream *strm, lzma_index **i, uint64_t memlimit) +{ + // If i isn't NULL, *i must always be initialized due to + // the wording in the API docs. This way it is initialized + // if we return LZMA_PROG_ERROR due to strm == NULL. + if (i != NULL) + *i = NULL; + + lzma_next_strm_init(lzma_index_decoder_init, strm, i, memlimit); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_index_buffer_decode(lzma_index **i, uint64_t *memlimit, + const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size) +{ + // If i isn't NULL, *i must always be initialized due to + // the wording in the API docs. + if (i != NULL) + *i = NULL; + + // Sanity checks + if (i == NULL || memlimit == NULL + || in == NULL || in_pos == NULL || *in_pos > in_size) + return LZMA_PROG_ERROR; + + // Initialize the decoder. + lzma_index_coder coder; + return_if_error(index_decoder_reset(&coder, allocator, i, *memlimit)); + + // Store the input start position so that we can restore it in case + // of an error. + const size_t in_start = *in_pos; + + // Do the actual decoding. + lzma_ret ret = index_decode(&coder, allocator, in, in_pos, in_size, + NULL, NULL, 0, LZMA_RUN); + + if (ret == LZMA_STREAM_END) { + ret = LZMA_OK; + } else { + // Something went wrong, free the Index structure and restore + // the input position. + lzma_index_end(coder.index, allocator); + *in_pos = in_start; + + if (ret == LZMA_OK) { + // The input is truncated or otherwise corrupt. + // Use LZMA_DATA_ERROR instead of LZMA_BUF_ERROR + // like lzma_vli_decode() does in single-call mode. + ret = LZMA_DATA_ERROR; + + } else if (ret == LZMA_MEMLIMIT_ERROR) { + // Tell the caller how much memory would have + // been needed. + *memlimit = lzma_index_memusage(1, coder.count); + } + } + + return ret; +} diff --git a/src/liblzma/common/index_decoder.h b/src/liblzma/common/index_decoder.h new file mode 100644 index 0000000000..5351d2f0df --- /dev/null +++ b/src/liblzma/common/index_decoder.h @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index_decoder.h +/// \brief Decodes the Index field +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_INDEX_DECODER_H +#define LZMA_INDEX_DECODER_H + +#include "common.h" +#include "index.h" + + +extern lzma_ret lzma_index_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + lzma_index **i, uint64_t memlimit); + + +#endif diff --git a/src/liblzma/common/index_encoder.c b/src/liblzma/common/index_encoder.c new file mode 100644 index 0000000000..80f1be1e3a --- /dev/null +++ b/src/liblzma/common/index_encoder.c @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index_encoder.c +/// \brief Encodes the Index field +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "index_encoder.h" +#include "index.h" +#include "check.h" + + +typedef struct { + enum { + SEQ_INDICATOR, + SEQ_COUNT, + SEQ_UNPADDED, + SEQ_UNCOMPRESSED, + SEQ_NEXT, + SEQ_PADDING, + SEQ_CRC32, + } sequence; + + /// Index being encoded + const lzma_index *index; + + /// Iterator for the Index being encoded + lzma_index_iter iter; + + /// Position in integers + size_t pos; + + /// CRC32 of the List of Records field + uint32_t crc32; +} lzma_index_coder; + + +static lzma_ret +index_encode(void *coder_ptr, + const lzma_allocator *allocator lzma_attribute((__unused__)), + const uint8_t *restrict in lzma_attribute((__unused__)), + size_t *restrict in_pos lzma_attribute((__unused__)), + size_t in_size lzma_attribute((__unused__)), + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, + lzma_action action lzma_attribute((__unused__))) +{ + lzma_index_coder *coder = coder_ptr; + + // Position where to start calculating CRC32. The idea is that we + // need to call lzma_crc32() only once per call to index_encode(). + const size_t out_start = *out_pos; + + // Return value to use if we return at the end of this function. + // We use "goto out" to jump out of the while-switch construct + // instead of returning directly, because that way we don't need + // to copypaste the lzma_crc32() call to many places. + lzma_ret ret = LZMA_OK; + + while (*out_pos < out_size) + switch (coder->sequence) { + case SEQ_INDICATOR: + out[*out_pos] = INDEX_INDICATOR; + ++*out_pos; + coder->sequence = SEQ_COUNT; + break; + + case SEQ_COUNT: { + const lzma_vli count = lzma_index_block_count(coder->index); + ret = lzma_vli_encode(count, &coder->pos, + out, out_pos, out_size); + if (ret != LZMA_STREAM_END) + goto out; + + ret = LZMA_OK; + coder->pos = 0; + coder->sequence = SEQ_NEXT; + break; + } + + case SEQ_NEXT: + if (lzma_index_iter_next( + &coder->iter, LZMA_INDEX_ITER_BLOCK)) { + // Get the size of the Index Padding field. + coder->pos = lzma_index_padding_size(coder->index); + assert(coder->pos <= 3); + coder->sequence = SEQ_PADDING; + break; + } + + coder->sequence = SEQ_UNPADDED; + FALLTHROUGH; + + case SEQ_UNPADDED: + case SEQ_UNCOMPRESSED: { + const lzma_vli size = coder->sequence == SEQ_UNPADDED + ? coder->iter.block.unpadded_size + : coder->iter.block.uncompressed_size; + + ret = lzma_vli_encode(size, &coder->pos, + out, out_pos, out_size); + if (ret != LZMA_STREAM_END) + goto out; + + ret = LZMA_OK; + coder->pos = 0; + + // Advance to SEQ_UNCOMPRESSED or SEQ_NEXT. + ++coder->sequence; + break; + } + + case SEQ_PADDING: + if (coder->pos > 0) { + --coder->pos; + out[(*out_pos)++] = 0x00; + break; + } + + // Finish the CRC32 calculation. + coder->crc32 = lzma_crc32(out + out_start, + *out_pos - out_start, coder->crc32); + + coder->sequence = SEQ_CRC32; + FALLTHROUGH; + + case SEQ_CRC32: + // We don't use the main loop, because we don't want + // coder->crc32 to be touched anymore. + do { + if (*out_pos == out_size) + return LZMA_OK; + + out[*out_pos] = (coder->crc32 >> (coder->pos * 8)) + & 0xFF; + ++*out_pos; + + } while (++coder->pos < 4); + + return LZMA_STREAM_END; + + default: + assert(0); + return LZMA_PROG_ERROR; + } + +out: + // Update the CRC32. + // + // Avoid null pointer + 0 (undefined behavior) in "out + out_start". + // In such a case we had no input and thus out_used == 0. + { + const size_t out_used = *out_pos - out_start; + if (out_used > 0) + coder->crc32 = lzma_crc32(out + out_start, + out_used, coder->crc32); + } + + return ret; +} + + +static void +index_encoder_end(void *coder, const lzma_allocator *allocator) +{ + lzma_free(coder, allocator); + return; +} + + +static void +index_encoder_reset(lzma_index_coder *coder, const lzma_index *i) +{ + lzma_index_iter_init(&coder->iter, i); + + coder->sequence = SEQ_INDICATOR; + coder->index = i; + coder->pos = 0; + coder->crc32 = 0; + + return; +} + + +extern lzma_ret +lzma_index_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_index *i) +{ + lzma_next_coder_init(&lzma_index_encoder_init, next, allocator); + + if (i == NULL) + return LZMA_PROG_ERROR; + + if (next->coder == NULL) { + next->coder = lzma_alloc(sizeof(lzma_index_coder), allocator); + if (next->coder == NULL) + return LZMA_MEM_ERROR; + + next->code = &index_encode; + next->end = &index_encoder_end; + } + + index_encoder_reset(next->coder, i); + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_index_encoder(lzma_stream *strm, const lzma_index *i) +{ + lzma_next_strm_init(lzma_index_encoder_init, strm, i); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_index_buffer_encode(const lzma_index *i, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Validate the arguments. + if (i == NULL || out == NULL || out_pos == NULL || *out_pos > out_size) + return LZMA_PROG_ERROR; + + // Don't try to encode if there's not enough output space. + if (out_size - *out_pos < lzma_index_size(i)) + return LZMA_BUF_ERROR; + + // The Index encoder needs just one small data structure so we can + // allocate it on stack. + lzma_index_coder coder; + index_encoder_reset(&coder, i); + + // Do the actual encoding. This should never fail, but store + // the original *out_pos just in case. + const size_t out_start = *out_pos; + lzma_ret ret = index_encode(&coder, NULL, NULL, NULL, 0, + out, out_pos, out_size, LZMA_RUN); + + if (ret == LZMA_STREAM_END) { + ret = LZMA_OK; + } else { + // We should never get here, but just in case, restore the + // output position and set the error accordingly if something + // goes wrong and debugging isn't enabled. + assert(0); + *out_pos = out_start; + ret = LZMA_PROG_ERROR; + } + + return ret; +} diff --git a/src/liblzma/common/index_encoder.h b/src/liblzma/common/index_encoder.h new file mode 100644 index 0000000000..29ba110669 --- /dev/null +++ b/src/liblzma/common/index_encoder.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index_encoder.h +/// \brief Encodes the Index field +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_INDEX_ENCODER_H +#define LZMA_INDEX_ENCODER_H + +#include "common.h" + + +extern lzma_ret lzma_index_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, const lzma_index *i); + + +#endif diff --git a/src/liblzma/common/index_hash.c b/src/liblzma/common/index_hash.c new file mode 100644 index 0000000000..b7f1b6b58d --- /dev/null +++ b/src/liblzma/common/index_hash.c @@ -0,0 +1,341 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index_hash.c +/// \brief Validates Index by using a hash function +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "index.h" +#include "check.h" + + +typedef struct { + /// Sum of the Block sizes (including Block Padding) + lzma_vli blocks_size; + + /// Sum of the Uncompressed Size fields + lzma_vli uncompressed_size; + + /// Number of Records + lzma_vli count; + + /// Size of the List of Index Records as bytes + lzma_vli index_list_size; + + /// Check calculated from Unpadded Sizes and Uncompressed Sizes. + lzma_check_state check; + +} lzma_index_hash_info; + + +struct lzma_index_hash_s { + enum { + SEQ_BLOCK, + SEQ_COUNT, + SEQ_UNPADDED, + SEQ_UNCOMPRESSED, + SEQ_PADDING_INIT, + SEQ_PADDING, + SEQ_CRC32, + } sequence; + + /// Information collected while decoding the actual Blocks. + lzma_index_hash_info blocks; + + /// Information collected from the Index field. + lzma_index_hash_info records; + + /// Number of Records not fully decoded + lzma_vli remaining; + + /// Unpadded Size currently being read from an Index Record. + lzma_vli unpadded_size; + + /// Uncompressed Size currently being read from an Index Record. + lzma_vli uncompressed_size; + + /// Position in variable-length integers when decoding them from + /// the List of Records. + size_t pos; + + /// CRC32 of the Index + uint32_t crc32; +}; + + +extern LZMA_API(lzma_index_hash *) +lzma_index_hash_init(lzma_index_hash *index_hash, + const lzma_allocator *allocator) +{ + if (index_hash == NULL) { + index_hash = lzma_alloc(sizeof(lzma_index_hash), allocator); + if (index_hash == NULL) + return NULL; + } + + index_hash->sequence = SEQ_BLOCK; + index_hash->blocks.blocks_size = 0; + index_hash->blocks.uncompressed_size = 0; + index_hash->blocks.count = 0; + index_hash->blocks.index_list_size = 0; + index_hash->records.blocks_size = 0; + index_hash->records.uncompressed_size = 0; + index_hash->records.count = 0; + index_hash->records.index_list_size = 0; + index_hash->unpadded_size = 0; + index_hash->uncompressed_size = 0; + index_hash->pos = 0; + index_hash->crc32 = 0; + + // These cannot fail because LZMA_CHECK_BEST is known to be supported. + (void)lzma_check_init(&index_hash->blocks.check, LZMA_CHECK_BEST); + (void)lzma_check_init(&index_hash->records.check, LZMA_CHECK_BEST); + + return index_hash; +} + + +extern LZMA_API(void) +lzma_index_hash_end(lzma_index_hash *index_hash, + const lzma_allocator *allocator) +{ + lzma_free(index_hash, allocator); + return; +} + + +extern LZMA_API(lzma_vli) +lzma_index_hash_size(const lzma_index_hash *index_hash) +{ + // Get the size of the Index from ->blocks instead of ->records for + // cases where application wants to know the Index Size before + // decoding the Index. + return index_size(index_hash->blocks.count, + index_hash->blocks.index_list_size); +} + + +/// Updates the sizes and the hash without any validation. +static void +hash_append(lzma_index_hash_info *info, lzma_vli unpadded_size, + lzma_vli uncompressed_size) +{ + info->blocks_size += vli_ceil4(unpadded_size); + info->uncompressed_size += uncompressed_size; + info->index_list_size += lzma_vli_size(unpadded_size) + + lzma_vli_size(uncompressed_size); + ++info->count; + + const lzma_vli sizes[2] = { unpadded_size, uncompressed_size }; + lzma_check_update(&info->check, LZMA_CHECK_BEST, + (const uint8_t *)(sizes), sizeof(sizes)); + + return; +} + + +extern LZMA_API(lzma_ret) +lzma_index_hash_append(lzma_index_hash *index_hash, lzma_vli unpadded_size, + lzma_vli uncompressed_size) +{ + // Validate the arguments. + if (index_hash == NULL || index_hash->sequence != SEQ_BLOCK + || unpadded_size < UNPADDED_SIZE_MIN + || unpadded_size > UNPADDED_SIZE_MAX + || uncompressed_size > LZMA_VLI_MAX) + return LZMA_PROG_ERROR; + + // Update the hash. + hash_append(&index_hash->blocks, unpadded_size, uncompressed_size); + + // Validate the properties of *info are still in allowed limits. + if (index_hash->blocks.blocks_size > LZMA_VLI_MAX + || index_hash->blocks.uncompressed_size > LZMA_VLI_MAX + || index_size(index_hash->blocks.count, + index_hash->blocks.index_list_size) + > LZMA_BACKWARD_SIZE_MAX + || index_stream_size(index_hash->blocks.blocks_size, + index_hash->blocks.count, + index_hash->blocks.index_list_size) + > LZMA_VLI_MAX) + return LZMA_DATA_ERROR; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_index_hash_decode(lzma_index_hash *index_hash, const uint8_t *in, + size_t *in_pos, size_t in_size) +{ + // Catch zero input buffer here, because in contrast to Index encoder + // and decoder functions, applications call this function directly + // instead of via lzma_code(), which does the buffer checking. + if (*in_pos >= in_size) + return LZMA_BUF_ERROR; + + // NOTE: This function has many similarities to index_encode() and + // index_decode() functions found from index_encoder.c and + // index_decoder.c. See the comments especially in index_encoder.c. + const size_t in_start = *in_pos; + lzma_ret ret = LZMA_OK; + + while (*in_pos < in_size) + switch (index_hash->sequence) { + case SEQ_BLOCK: + // Check the Index Indicator is present. + if (in[(*in_pos)++] != INDEX_INDICATOR) + return LZMA_DATA_ERROR; + + index_hash->sequence = SEQ_COUNT; + break; + + case SEQ_COUNT: { + ret = lzma_vli_decode(&index_hash->remaining, + &index_hash->pos, in, in_pos, in_size); + if (ret != LZMA_STREAM_END) + goto out; + + // The count must match the count of the Blocks decoded. + if (index_hash->remaining != index_hash->blocks.count) + return LZMA_DATA_ERROR; + + ret = LZMA_OK; + index_hash->pos = 0; + + // Handle the special case when there are no Blocks. + index_hash->sequence = index_hash->remaining == 0 + ? SEQ_PADDING_INIT : SEQ_UNPADDED; + break; + } + + case SEQ_UNPADDED: + case SEQ_UNCOMPRESSED: { + lzma_vli *size = index_hash->sequence == SEQ_UNPADDED + ? &index_hash->unpadded_size + : &index_hash->uncompressed_size; + + ret = lzma_vli_decode(size, &index_hash->pos, + in, in_pos, in_size); + if (ret != LZMA_STREAM_END) + goto out; + + ret = LZMA_OK; + index_hash->pos = 0; + + if (index_hash->sequence == SEQ_UNPADDED) { + if (index_hash->unpadded_size < UNPADDED_SIZE_MIN + || index_hash->unpadded_size + > UNPADDED_SIZE_MAX) + return LZMA_DATA_ERROR; + + index_hash->sequence = SEQ_UNCOMPRESSED; + } else { + // Update the hash. + hash_append(&index_hash->records, + index_hash->unpadded_size, + index_hash->uncompressed_size); + + // Verify that we don't go over the known sizes. Note + // that this validation is simpler than the one used + // in lzma_index_hash_append(), because here we know + // that values in index_hash->blocks are already + // validated and we are fine as long as we don't + // exceed them in index_hash->records. + if (index_hash->blocks.blocks_size + < index_hash->records.blocks_size + || index_hash->blocks.uncompressed_size + < index_hash->records.uncompressed_size + || index_hash->blocks.index_list_size + < index_hash->records.index_list_size) + return LZMA_DATA_ERROR; + + // Check if this was the last Record. + index_hash->sequence = --index_hash->remaining == 0 + ? SEQ_PADDING_INIT : SEQ_UNPADDED; + } + + break; + } + + case SEQ_PADDING_INIT: + index_hash->pos = (LZMA_VLI_C(4) - index_size_unpadded( + index_hash->records.count, + index_hash->records.index_list_size)) & 3; + + index_hash->sequence = SEQ_PADDING; + FALLTHROUGH; + + case SEQ_PADDING: + if (index_hash->pos > 0) { + --index_hash->pos; + if (in[(*in_pos)++] != 0x00) + return LZMA_DATA_ERROR; + + break; + } + + // Compare the sizes. + if (index_hash->blocks.blocks_size + != index_hash->records.blocks_size + || index_hash->blocks.uncompressed_size + != index_hash->records.uncompressed_size + || index_hash->blocks.index_list_size + != index_hash->records.index_list_size) + return LZMA_DATA_ERROR; + + // Finish the hashes and compare them. + lzma_check_finish(&index_hash->blocks.check, LZMA_CHECK_BEST); + lzma_check_finish(&index_hash->records.check, LZMA_CHECK_BEST); + if (memcmp(index_hash->blocks.check.buffer.u8, + index_hash->records.check.buffer.u8, + lzma_check_size(LZMA_CHECK_BEST)) != 0) + return LZMA_DATA_ERROR; + + // Finish the CRC32 calculation. + index_hash->crc32 = lzma_crc32(in + in_start, + *in_pos - in_start, index_hash->crc32); + + index_hash->sequence = SEQ_CRC32; + FALLTHROUGH; + + case SEQ_CRC32: + do { + if (*in_pos == in_size) + return LZMA_OK; + + if (((index_hash->crc32 >> (index_hash->pos * 8)) + & 0xFF) != in[(*in_pos)++]) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + return LZMA_DATA_ERROR; +#endif + } + + } while (++index_hash->pos < 4); + + return LZMA_STREAM_END; + + default: + assert(0); + return LZMA_PROG_ERROR; + } + +out: + // Update the CRC32. + // + // Avoid null pointer + 0 (undefined behavior) in "in + in_start". + // In such a case we had no input and thus in_used == 0. + { + const size_t in_used = *in_pos - in_start; + if (in_used > 0) + index_hash->crc32 = lzma_crc32(in + in_start, + in_used, index_hash->crc32); + } + + return ret; +} diff --git a/src/liblzma/common/lzip_decoder.c b/src/liblzma/common/lzip_decoder.c new file mode 100644 index 0000000000..4dff2d5889 --- /dev/null +++ b/src/liblzma/common/lzip_decoder.c @@ -0,0 +1,413 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzip_decoder.c +/// \brief Decodes .lz (lzip) files +// +// Author: Michał Górny +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzip_decoder.h" +#include "lzma_decoder.h" +#include "check.h" + + +// .lz format version 0 lacks the 64-bit Member size field in the footer. +#define LZIP_V0_FOOTER_SIZE 12 +#define LZIP_V1_FOOTER_SIZE 20 +#define LZIP_FOOTER_SIZE_MAX LZIP_V1_FOOTER_SIZE + +// lc/lp/pb are hardcoded in the .lz format. +#define LZIP_LC 3 +#define LZIP_LP 0 +#define LZIP_PB 2 + + +typedef struct { + enum { + SEQ_ID_STRING, + SEQ_VERSION, + SEQ_DICT_SIZE, + SEQ_CODER_INIT, + SEQ_LZMA_STREAM, + SEQ_MEMBER_FOOTER, + } sequence; + + /// .lz member format version + uint32_t version; + + /// CRC32 of the uncompressed data in the .lz member + uint32_t crc32; + + /// Uncompressed size of the .lz member + uint64_t uncompressed_size; + + /// Compressed size of the .lz member + uint64_t member_size; + + /// Memory usage limit + uint64_t memlimit; + + /// Amount of memory actually needed + uint64_t memusage; + + /// If true, LZMA_GET_CHECK is returned after decoding the header + /// fields. As all files use CRC32 this is redundant but it's + /// implemented anyway since the initialization functions supports + /// all other flags in addition to LZMA_TELL_ANY_CHECK. + bool tell_any_check; + + /// If true, we won't calculate or verify the CRC32 of + /// the uncompressed data. + bool ignore_check; + + /// If true, we will decode concatenated .lz members and stop if + /// non-.lz data is seen after at least one member has been + /// successfully decoded. + bool concatenated; + + /// When decoding concatenated .lz members, this is true as long as + /// we are decoding the first .lz member. This is needed to avoid + /// incorrect LZMA_FORMAT_ERROR in case there is non-.lz data at + /// the end of the file. + bool first_member; + + /// Reading position in the header and footer fields + size_t pos; + + /// Buffer to hold the .lz footer fields + uint8_t buffer[LZIP_FOOTER_SIZE_MAX]; + + /// Options decoded from the .lz header that needed to initialize + /// the LZMA1 decoder. + lzma_options_lzma options; + + /// LZMA1 decoder + lzma_next_coder lzma_decoder; + +} lzma_lzip_coder; + + +static lzma_ret +lzip_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_lzip_coder *coder = coder_ptr; + + while (true) + switch (coder->sequence) { + case SEQ_ID_STRING: { + // The "ID string" or magic bytes are "LZIP" in US-ASCII. + const uint8_t lzip_id_string[4] = { 0x4C, 0x5A, 0x49, 0x50 }; + + while (coder->pos < sizeof(lzip_id_string)) { + if (*in_pos >= in_size) { + // If we are on the 2nd+ concatenated member + // and the input ends before we can read + // the magic bytes, we discard the bytes that + // were already read (up to 3) and finish. + // See the reasoning below. + return !coder->first_member + && action == LZMA_FINISH + ? LZMA_STREAM_END : LZMA_OK; + } + + if (in[*in_pos] != lzip_id_string[coder->pos]) { + // The .lz format allows putting non-.lz data + // at the end of the file. If we have seen + // at least one valid .lz member already, + // then we won't consume the byte at *in_pos + // and will return LZMA_STREAM_END. This way + // apps can easily locate and read the non-.lz + // data after the .lz member(s). + // + // NOTE: If the first 1-3 bytes of the non-.lz + // data match the .lz ID string then the first + // 1-3 bytes of the junk will get ignored by + // us. If apps want to properly locate the + // trailing data they must ensure that the + // first byte of their custom data isn't the + // same as the first byte of .lz ID string. + // With the liblzma API we cannot rewind the + // input position across calls to lzma_code(). + return !coder->first_member + ? LZMA_STREAM_END : LZMA_FORMAT_ERROR; + } + + ++*in_pos; + ++coder->pos; + } + + coder->pos = 0; + + coder->crc32 = 0; + coder->uncompressed_size = 0; + coder->member_size = sizeof(lzip_id_string); + + coder->sequence = SEQ_VERSION; + FALLTHROUGH; + } + + case SEQ_VERSION: + if (*in_pos >= in_size) + return LZMA_OK; + + coder->version = in[(*in_pos)++]; + + // We support version 0 and unextended version 1. + if (coder->version > 1) + return LZMA_OPTIONS_ERROR; + + ++coder->member_size; + coder->sequence = SEQ_DICT_SIZE; + + // .lz versions 0 and 1 use CRC32 as the integrity check + // so if the application wanted to know that + // (LZMA_TELL_ANY_CHECK) we can tell it now. + if (coder->tell_any_check) + return LZMA_GET_CHECK; + + FALLTHROUGH; + + case SEQ_DICT_SIZE: { + if (*in_pos >= in_size) + return LZMA_OK; + + const uint32_t ds = in[(*in_pos)++]; + ++coder->member_size; + + // The five lowest bits are for the base-2 logarithm of + // the dictionary size and the highest three bits are + // the fractional part (0/16 to 7/16) that will be + // subtracted to get the final value. + // + // For example, with 0xB5: + // b2log = 21 + // fracnum = 5 + // dict_size = 2^21 - 2^21 * 5 / 16 = 1408 KiB + const uint32_t b2log = ds & 0x1F; + const uint32_t fracnum = ds >> 5; + + // The format versions 0 and 1 allow dictionary size in the + // range [4 KiB, 512 MiB]. + if (b2log < 12 || b2log > 29 || (b2log == 12 && fracnum > 0)) + return LZMA_DATA_ERROR; + + // 2^[b2log] - 2^[b2log] * [fracnum] / 16 + // = 2^[b2log] - [fracnum] * 2^([b2log] - 4) + coder->options.dict_size = (UINT32_C(1) << b2log) + - (fracnum << (b2log - 4)); + + assert(coder->options.dict_size >= 4096); + assert(coder->options.dict_size <= (UINT32_C(512) << 20)); + + coder->options.preset_dict = NULL; + coder->options.lc = LZIP_LC; + coder->options.lp = LZIP_LP; + coder->options.pb = LZIP_PB; + + // Calculate the memory usage. + coder->memusage = lzma_lzma_decoder_memusage(&coder->options) + + LZMA_MEMUSAGE_BASE; + + // Initialization is a separate step because if we return + // LZMA_MEMLIMIT_ERROR we need to be able to restart after + // the memlimit has been increased. + coder->sequence = SEQ_CODER_INIT; + FALLTHROUGH; + } + + case SEQ_CODER_INIT: { + if (coder->memusage > coder->memlimit) + return LZMA_MEMLIMIT_ERROR; + + const lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_decoder_init, + .options = &coder->options, + }, { + .init = NULL, + } + }; + + return_if_error(lzma_next_filter_init(&coder->lzma_decoder, + allocator, filters)); + + coder->crc32 = 0; + coder->sequence = SEQ_LZMA_STREAM; + FALLTHROUGH; + } + + case SEQ_LZMA_STREAM: { + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + const lzma_ret ret = coder->lzma_decoder.code( + coder->lzma_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + + const size_t out_used = *out_pos - out_start; + + coder->member_size += *in_pos - in_start; + coder->uncompressed_size += out_used; + + // Don't update the CRC32 if the integrity check will be + // ignored or if there was no new output. The latter is + // important in case out == NULL to avoid null pointer + 0 + // which is undefined behavior. + if (!coder->ignore_check && out_used > 0) + coder->crc32 = lzma_crc32(out + out_start, out_used, + coder->crc32); + + if (ret != LZMA_STREAM_END) + return ret; + + coder->sequence = SEQ_MEMBER_FOOTER; + FALLTHROUGH; + } + + case SEQ_MEMBER_FOOTER: { + // The footer of .lz version 0 lacks the Member size field. + // This is the only difference between version 0 and + // unextended version 1 formats. + const size_t footer_size = coder->version == 0 + ? LZIP_V0_FOOTER_SIZE + : LZIP_V1_FOOTER_SIZE; + + // Copy the CRC32, Data size, and Member size fields to + // the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + footer_size); + + // Return if we didn't get the whole footer yet. + if (coder->pos < footer_size) + return LZMA_OK; + + coder->pos = 0; + coder->member_size += footer_size; + + // Check that the footer fields match the observed data. + if (!coder->ignore_check + && coder->crc32 != read32le(&coder->buffer[0])) + return LZMA_DATA_ERROR; + + if (coder->uncompressed_size != read64le(&coder->buffer[4])) + return LZMA_DATA_ERROR; + + if (coder->version > 0) { + // .lz version 0 has no Member size field. + if (coder->member_size != read64le(&coder->buffer[12])) + return LZMA_DATA_ERROR; + } + + // Decoding is finished if we weren't requested to decode + // more than one .lz member. + if (!coder->concatenated) + return LZMA_STREAM_END; + + coder->first_member = false; + coder->sequence = SEQ_ID_STRING; + break; + } + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + // Never reached +} + + +static void +lzip_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_lzip_coder *coder = coder_ptr; + lzma_next_end(&coder->lzma_decoder, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_check +lzip_decoder_get_check(const void *coder_ptr lzma_attribute((__unused__))) +{ + return LZMA_CHECK_CRC32; +} + + +static lzma_ret +lzip_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_lzip_coder *coder = coder_ptr; + + *memusage = coder->memusage; + *old_memlimit = coder->memlimit; + + if (new_memlimit != 0) { + if (new_memlimit < coder->memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +extern lzma_ret +lzma_lzip_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags) +{ + lzma_next_coder_init(&lzma_lzip_decoder_init, next, allocator); + + if (flags & ~LZMA_SUPPORTED_FLAGS) + return LZMA_OPTIONS_ERROR; + + lzma_lzip_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_lzip_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &lzip_decode; + next->end = &lzip_decoder_end; + next->get_check = &lzip_decoder_get_check; + next->memconfig = &lzip_decoder_memconfig; + + coder->lzma_decoder = LZMA_NEXT_CODER_INIT; + } + + coder->sequence = SEQ_ID_STRING; + coder->memlimit = my_max(1, memlimit); + coder->memusage = LZMA_MEMUSAGE_BASE; + coder->tell_any_check = (flags & LZMA_TELL_ANY_CHECK) != 0; + coder->ignore_check = (flags & LZMA_IGNORE_CHECK) != 0; + coder->concatenated = (flags & LZMA_CONCATENATED) != 0; + coder->first_member = true; + coder->pos = 0; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_lzip_decoder(lzma_stream *strm, uint64_t memlimit, uint32_t flags) +{ + lzma_next_strm_init(lzma_lzip_decoder_init, strm, memlimit, flags); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/lzip_decoder.h b/src/liblzma/common/lzip_decoder.h new file mode 100644 index 0000000000..0e1f7bebd4 --- /dev/null +++ b/src/liblzma/common/lzip_decoder.h @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzip_decoder.h +/// \brief Decodes .lz (lzip) files +// +// Author: Michał Górny +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZIP_DECODER_H +#define LZMA_LZIP_DECODER_H + +#include "common.h" + +extern lzma_ret lzma_lzip_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags); + +#endif diff --git a/src/liblzma/common/memcmplen.h b/src/liblzma/common/memcmplen.h new file mode 100644 index 0000000000..82e9085422 --- /dev/null +++ b/src/liblzma/common/memcmplen.h @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file memcmplen.h +/// \brief Optimized comparison of two buffers +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_MEMCMPLEN_H +#define LZMA_MEMCMPLEN_H + +#include "common.h" + +#ifdef HAVE_IMMINTRIN_H +# include +#endif + +// Only include if it is needed. The header is only needed +// on Windows when using an MSVC compatible compiler. The Intel compiler +// can use the intrinsics without the header file. +#if defined(TUKLIB_FAST_UNALIGNED_ACCESS) \ + && defined(_MSC_VER) \ + && (defined(_M_X64) \ + || defined(_M_ARM64) || defined(_M_ARM64EC)) \ + && !defined(__INTEL_COMPILER) +# include +#endif + + +/// Find out how many equal bytes the two buffers have. +/// +/// \param buf1 First buffer +/// \param buf2 Second buffer +/// \param len How many bytes have already been compared and will +/// be assumed to match +/// \param limit How many bytes to compare at most, including the +/// already-compared bytes. This must be significantly +/// smaller than UINT32_MAX to avoid integer overflows. +/// Up to LZMA_MEMCMPLEN_EXTRA bytes may be read past +/// the specified limit from both buf1 and buf2. +/// +/// \return Number of equal bytes in the buffers is returned. +/// This is always at least len and at most limit. +/// +/// \note LZMA_MEMCMPLEN_EXTRA defines how many extra bytes may be read. +/// It's rounded up to 2^n. This extra amount needs to be +/// allocated in the buffers being used. It needs to be +/// initialized too to keep Valgrind quiet. +static lzma_always_inline uint32_t +lzma_memcmplen(const uint8_t *buf1, const uint8_t *buf2, + uint32_t len, uint32_t limit) +{ + assert(len <= limit); + assert(limit <= UINT32_MAX / 2); + +#if defined(TUKLIB_FAST_UNALIGNED_ACCESS) \ + && (((TUKLIB_GNUC_REQ(3, 4) || defined(__clang__)) \ + && SIZE_MAX == UINT64_MAX) \ + || (defined(__INTEL_COMPILER) && defined(__x86_64__)) \ + || (defined(__INTEL_COMPILER) && defined(_M_X64)) \ + || (defined(_MSC_VER) && (defined(_M_X64) \ + || defined(_M_ARM64) || defined(_M_ARM64EC)))) + // This is only for x86-64 and ARM64 for now. This might be fine on + // other 64-bit processors too. + // + // Reasons to use subtraction instead of xor: + // + // - On some x86-64 processors (Intel Sandy Bridge to Tiger Lake), + // sub+jz and sub+jnz can be fused but xor+jz or xor+jnz cannot. + // Thus using subtraction has potential to be a tiny amount faster + // since the code checks if the quotient is non-zero. + // + // - Some processors (Intel Pentium 4) used to have more ALU + // resources for add/sub instructions than and/or/xor. + // + // The processor info is based on Agner Fog's microarchitecture.pdf + // version 2023-05-26. https://www.agner.org/optimize/ +#define LZMA_MEMCMPLEN_EXTRA 8 + while (len < limit) { +# ifdef WORDS_BIGENDIAN + const uint64_t x = read64ne(buf1 + len) ^ read64ne(buf2 + len); +# else + const uint64_t x = read64ne(buf1 + len) - read64ne(buf2 + len); +# endif + if (x != 0) { + // MSVC or Intel C compiler on Windows +# if defined(_MSC_VER) || defined(__INTEL_COMPILER) + unsigned long tmp; + _BitScanForward64(&tmp, x); + len += (uint32_t)tmp >> 3; + // GCC, Clang, or Intel C compiler +# elif defined(WORDS_BIGENDIAN) + len += (uint32_t)__builtin_clzll(x) >> 3; +# else + len += (uint32_t)__builtin_ctzll(x) >> 3; +# endif + return my_min(len, limit); + } + + len += 8; + } + + return limit; + +#elif defined(TUKLIB_FAST_UNALIGNED_ACCESS) \ + && defined(HAVE__MM_MOVEMASK_EPI8) \ + && (defined(__SSE2__) \ + || (defined(_MSC_VER) && defined(_M_IX86_FP) \ + && _M_IX86_FP >= 2)) + // NOTE: This will use 128-bit unaligned access which + // TUKLIB_FAST_UNALIGNED_ACCESS wasn't meant to permit, + // but it's convenient here since this is x86-only. + // + // SSE2 version for 32-bit and 64-bit x86. On x86-64 the above + // version is sometimes significantly faster and sometimes + // slightly slower than this SSE2 version, so this SSE2 + // version isn't used on x86-64. +# define LZMA_MEMCMPLEN_EXTRA 16 + while (len < limit) { + const uint32_t x = 0xFFFF ^ (uint32_t)_mm_movemask_epi8( + _mm_cmpeq_epi8( + _mm_loadu_si128((const __m128i *)(buf1 + len)), + _mm_loadu_si128((const __m128i *)(buf2 + len)))); + + if (x != 0) { + len += ctz32(x); + return my_min(len, limit); + } + + len += 16; + } + + return limit; + +#elif defined(TUKLIB_FAST_UNALIGNED_ACCESS) && !defined(WORDS_BIGENDIAN) + // Generic 32-bit little endian method +# define LZMA_MEMCMPLEN_EXTRA 4 + while (len < limit) { + uint32_t x = read32ne(buf1 + len) - read32ne(buf2 + len); + if (x != 0) { + if ((x & 0xFFFF) == 0) { + len += 2; + x >>= 16; + } + + if ((x & 0xFF) == 0) + ++len; + + return my_min(len, limit); + } + + len += 4; + } + + return limit; + +#elif defined(TUKLIB_FAST_UNALIGNED_ACCESS) && defined(WORDS_BIGENDIAN) + // Generic 32-bit big endian method +# define LZMA_MEMCMPLEN_EXTRA 4 + while (len < limit) { + uint32_t x = read32ne(buf1 + len) ^ read32ne(buf2 + len); + if (x != 0) { + if ((x & 0xFFFF0000) == 0) { + len += 2; + x <<= 16; + } + + if ((x & 0xFF000000) == 0) + ++len; + + return my_min(len, limit); + } + + len += 4; + } + + return limit; + +#else + // Simple portable version that doesn't use unaligned access. +# define LZMA_MEMCMPLEN_EXTRA 0 + while (len < limit && buf1[len] == buf2[len]) + ++len; + + return len; +#endif +} + +#endif diff --git a/src/liblzma/common/microlzma_decoder.c b/src/liblzma/common/microlzma_decoder.c new file mode 100644 index 0000000000..882cb2c808 --- /dev/null +++ b/src/liblzma/common/microlzma_decoder.c @@ -0,0 +1,220 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file microlzma_decoder.c +/// \brief Decode MicroLZMA format +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma_decoder.h" +#include "lz_decoder.h" + + +typedef struct { + /// LZMA1 decoder + lzma_next_coder lzma; + + /// Compressed size of the stream as given by the application. + /// This must be exactly correct. + /// + /// This will be decremented when input is read. + uint64_t comp_size; + + /// Uncompressed size of the stream as given by the application. + /// This may be less than the actual uncompressed size if + /// uncomp_size_is_exact is false. + /// + /// This will be decremented when output is produced. + lzma_vli uncomp_size; + + /// LZMA dictionary size as given by the application + uint32_t dict_size; + + /// If true, the exact uncompressed size is known. If false, + /// uncomp_size may be smaller than the real uncompressed size; + /// uncomp_size may never be bigger than the real uncompressed size. + bool uncomp_size_is_exact; + + /// True once the first byte of the MicroLZMA stream + /// has been processed. + bool props_decoded; +} lzma_microlzma_coder; + + +static lzma_ret +microlzma_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_microlzma_coder *coder = coder_ptr; + + // Remember the in start position so that we can update comp_size. + const size_t in_start = *in_pos; + + // Remember the out start position so that we can update uncomp_size. + const size_t out_start = *out_pos; + + // Limit the amount of input so that the decoder won't read more than + // comp_size. This is required when uncomp_size isn't exact because + // in that case the LZMA decoder will try to decode more input even + // when it has no output space (it can be looking for EOPM). + if (in_size - *in_pos > coder->comp_size) + in_size = *in_pos + (size_t)(coder->comp_size); + + // When the exact uncompressed size isn't known, we must limit + // the available output space to prevent the LZMA decoder from + // trying to decode too much. + if (!coder->uncomp_size_is_exact + && out_size - *out_pos > coder->uncomp_size) + out_size = *out_pos + (size_t)(coder->uncomp_size); + + if (!coder->props_decoded) { + // There must be at least one byte of input to decode + // the properties byte. + if (*in_pos >= in_size) + return LZMA_OK; + + lzma_options_lzma options = { + .dict_size = coder->dict_size, + .preset_dict = NULL, + .preset_dict_size = 0, + .ext_flags = 0, // EOPM not allowed when size is known + .ext_size_low = UINT32_MAX, // Unknown size by default + .ext_size_high = UINT32_MAX, + }; + + if (coder->uncomp_size_is_exact) + lzma_set_ext_size(options, coder->uncomp_size); + + // The properties are stored as bitwise-negation + // of the typical encoding. + if (lzma_lzma_lclppb_decode(&options, ~in[*in_pos])) + return LZMA_OPTIONS_ERROR; + + ++*in_pos; + + // Initialize the decoder. + lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_decoder_init, + .options = &options, + }, { + .init = NULL, + } + }; + + return_if_error(lzma_next_filter_init(&coder->lzma, + allocator, filters)); + + // Pass one dummy 0x00 byte to the LZMA decoder since that + // is what it expects the first byte to be. + const uint8_t dummy_in = 0; + size_t dummy_in_pos = 0; + if (coder->lzma.code(coder->lzma.coder, allocator, + &dummy_in, &dummy_in_pos, 1, + out, out_pos, out_size, LZMA_RUN) != LZMA_OK) + return LZMA_PROG_ERROR; + + assert(dummy_in_pos == 1); + coder->props_decoded = true; + } + + // The rest is normal LZMA decoding. + lzma_ret ret = coder->lzma.code(coder->lzma.coder, allocator, + in, in_pos, in_size, + out, out_pos, out_size, action); + + // Update the remaining compressed size. + assert(coder->comp_size >= *in_pos - in_start); + coder->comp_size -= *in_pos - in_start; + + if (coder->uncomp_size_is_exact) { + // After successful decompression of the complete stream + // the compressed size must match. + if (ret == LZMA_STREAM_END && coder->comp_size != 0) + ret = LZMA_DATA_ERROR; + } else { + // Update the amount of output remaining. + assert(coder->uncomp_size >= *out_pos - out_start); + coder->uncomp_size -= *out_pos - out_start; + + // - We must not get LZMA_STREAM_END because the stream + // shouldn't have EOPM. + // - We must use uncomp_size to determine when to + // return LZMA_STREAM_END. + if (ret == LZMA_STREAM_END) + ret = LZMA_DATA_ERROR; + else if (coder->uncomp_size == 0) + ret = LZMA_STREAM_END; + } + + return ret; +} + + +static void +microlzma_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_microlzma_coder *coder = coder_ptr; + lzma_next_end(&coder->lzma, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +microlzma_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t comp_size, + uint64_t uncomp_size, bool uncomp_size_is_exact, + uint32_t dict_size) +{ + lzma_next_coder_init(µlzma_decoder_init, next, allocator); + + lzma_microlzma_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_microlzma_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = µlzma_decode; + next->end = µlzma_decoder_end; + + coder->lzma = LZMA_NEXT_CODER_INIT; + } + + // The public API is uint64_t but the internal LZ decoder API uses + // lzma_vli. + if (uncomp_size > LZMA_VLI_MAX) + return LZMA_OPTIONS_ERROR; + + coder->comp_size = comp_size; + coder->uncomp_size = uncomp_size; + coder->uncomp_size_is_exact = uncomp_size_is_exact; + coder->dict_size = dict_size; + + coder->props_decoded = false; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_microlzma_decoder(lzma_stream *strm, uint64_t comp_size, + uint64_t uncomp_size, lzma_bool uncomp_size_is_exact, + uint32_t dict_size) +{ + lzma_next_strm_init(microlzma_decoder_init, strm, comp_size, + uncomp_size, uncomp_size_is_exact, dict_size); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/microlzma_encoder.c b/src/liblzma/common/microlzma_encoder.c new file mode 100644 index 0000000000..45ec0b12f4 --- /dev/null +++ b/src/liblzma/common/microlzma_encoder.c @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file microlzma_encoder.c +/// \brief Encode into MicroLZMA format +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma_encoder.h" + + +typedef struct { + /// LZMA1 encoder + lzma_next_coder lzma; + + /// LZMA properties byte (lc/lp/pb) + uint8_t props; +} lzma_microlzma_coder; + + +static lzma_ret +microlzma_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_microlzma_coder *coder = coder_ptr; + + // Remember *out_pos so that we can overwrite the first byte with + // the LZMA properties byte. + const size_t out_start = *out_pos; + + // Remember *in_pos so that we can set it based on how many + // uncompressed bytes were actually encoded. + const size_t in_start = *in_pos; + + // Set the output size limit based on the available output space. + // We know that the encoder supports set_out_limit() so + // LZMA_OPTIONS_ERROR isn't possible. LZMA_BUF_ERROR is possible + // but lzma_code() has an assertion to not allow it to be returned + // from here and I don't want to change that for now, so + // LZMA_BUF_ERROR becomes LZMA_PROG_ERROR. + uint64_t uncomp_size; + if (coder->lzma.set_out_limit(coder->lzma.coder, + &uncomp_size, out_size - *out_pos) != LZMA_OK) + return LZMA_PROG_ERROR; + + // set_out_limit fails if this isn't true. + assert(out_size - *out_pos >= 6); + + // Encode as much as possible. + const lzma_ret ret = coder->lzma.code(coder->lzma.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, action); + + if (ret != LZMA_STREAM_END) { + if (ret == LZMA_OK) { + assert(0); + return LZMA_PROG_ERROR; + } + + return ret; + } + + // The first output byte is bitwise-negation of the properties byte. + // We know that there is space for this byte because set_out_limit + // and the actual encoding succeeded. + out[out_start] = (uint8_t)(~coder->props); + + // The LZMA encoder likely read more input than it was able to encode. + // Set *in_pos based on uncomp_size. + assert(uncomp_size <= in_size - in_start); + *in_pos = in_start + (size_t)(uncomp_size); + + return ret; +} + + +static void +microlzma_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_microlzma_coder *coder = coder_ptr; + lzma_next_end(&coder->lzma, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +microlzma_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_options_lzma *options) +{ + lzma_next_coder_init(µlzma_encoder_init, next, allocator); + + lzma_microlzma_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_microlzma_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = µlzma_encode; + next->end = µlzma_encoder_end; + + coder->lzma = LZMA_NEXT_CODER_INIT; + } + + // Encode the properties byte. Bitwise-negation of it will be the + // first output byte. + if (lzma_lzma_lclppb_encode(options, &coder->props)) + return LZMA_OPTIONS_ERROR; + + // Initialize the LZMA encoder. + const lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_encoder_init, + .options = (void *)(options), + }, { + .init = NULL, + } + }; + + return lzma_next_filter_init(&coder->lzma, allocator, filters); +} + + +extern LZMA_API(lzma_ret) +lzma_microlzma_encoder(lzma_stream *strm, const lzma_options_lzma *options) +{ + lzma_next_strm_init(microlzma_encoder_init, strm, options); + + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; + +} diff --git a/src/liblzma/common/outqueue.c b/src/liblzma/common/outqueue.c new file mode 100644 index 0000000000..eb018eb42b --- /dev/null +++ b/src/liblzma/common/outqueue.c @@ -0,0 +1,286 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file outqueue.c +/// \brief Output queue handling in multithreaded coding +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "outqueue.h" + + +/// Get the maximum number of buffers that may be allocated based +/// on the number of threads. For now this is twice the number of threads. +/// It's a compromise between RAM usage and keeping the worker threads busy +/// when buffers finish out of order. +#define GET_BUFS_LIMIT(threads) (2 * (threads)) + + +extern uint64_t +lzma_outq_memusage(uint64_t buf_size_max, uint32_t threads) +{ + // This is to ease integer overflow checking: We may allocate up to + // GET_BUFS_LIMIT(LZMA_THREADS_MAX) buffers and we need some extra + // memory for other data structures too (that's the /2). + // + // lzma_outq_prealloc_buf() will still accept bigger buffers than this. + const uint64_t limit + = UINT64_MAX / GET_BUFS_LIMIT(LZMA_THREADS_MAX) / 2; + + if (threads > LZMA_THREADS_MAX || buf_size_max > limit) + return UINT64_MAX; + + return GET_BUFS_LIMIT(threads) + * lzma_outq_outbuf_memusage(buf_size_max); +} + + +static void +move_head_to_cache(lzma_outq *outq, const lzma_allocator *allocator) +{ + assert(outq->head != NULL); + assert(outq->tail != NULL); + assert(outq->bufs_in_use > 0); + + lzma_outbuf *buf = outq->head; + outq->head = buf->next; + if (outq->head == NULL) + outq->tail = NULL; + + if (outq->cache != NULL && outq->cache->allocated != buf->allocated) + lzma_outq_clear_cache(outq, allocator); + + buf->next = outq->cache; + outq->cache = buf; + + --outq->bufs_in_use; + outq->mem_in_use -= lzma_outq_outbuf_memusage(buf->allocated); + + return; +} + + +static void +free_one_cached_buffer(lzma_outq *outq, const lzma_allocator *allocator) +{ + assert(outq->cache != NULL); + + lzma_outbuf *buf = outq->cache; + outq->cache = buf->next; + + --outq->bufs_allocated; + outq->mem_allocated -= lzma_outq_outbuf_memusage(buf->allocated); + + lzma_free(buf, allocator); + return; +} + + +extern void +lzma_outq_clear_cache(lzma_outq *outq, const lzma_allocator *allocator) +{ + while (outq->cache != NULL) + free_one_cached_buffer(outq, allocator); + + return; +} + + +extern void +lzma_outq_clear_cache2(lzma_outq *outq, const lzma_allocator *allocator, + size_t keep_size) +{ + if (outq->cache == NULL) + return; + + // Free all but one. + while (outq->cache->next != NULL) + free_one_cached_buffer(outq, allocator); + + // Free the last one only if its size doesn't equal to keep_size. + if (outq->cache->allocated != keep_size) + free_one_cached_buffer(outq, allocator); + + return; +} + + +extern lzma_ret +lzma_outq_init(lzma_outq *outq, const lzma_allocator *allocator, + uint32_t threads) +{ + if (threads > LZMA_THREADS_MAX) + return LZMA_OPTIONS_ERROR; + + const uint32_t bufs_limit = GET_BUFS_LIMIT(threads); + + // Clear head/tail. + while (outq->head != NULL) + move_head_to_cache(outq, allocator); + + // If new buf_limit is lower than the old one, we may need to free + // a few cached buffers. + while (bufs_limit < outq->bufs_allocated) + free_one_cached_buffer(outq, allocator); + + outq->bufs_limit = bufs_limit; + outq->read_pos = 0; + + return LZMA_OK; +} + + +extern void +lzma_outq_end(lzma_outq *outq, const lzma_allocator *allocator) +{ + while (outq->head != NULL) + move_head_to_cache(outq, allocator); + + lzma_outq_clear_cache(outq, allocator); + return; +} + + +extern lzma_ret +lzma_outq_prealloc_buf(lzma_outq *outq, const lzma_allocator *allocator, + size_t size) +{ + // Caller must have checked it with lzma_outq_has_buf(). + assert(outq->bufs_in_use < outq->bufs_limit); + + // If there already is appropriately-sized buffer in the cache, + // we need to do nothing. + if (outq->cache != NULL && outq->cache->allocated == size) + return LZMA_OK; + + if (size > SIZE_MAX - sizeof(lzma_outbuf)) + return LZMA_MEM_ERROR; + + const size_t alloc_size = lzma_outq_outbuf_memusage(size); + + // The cache may have buffers but their size is wrong. + lzma_outq_clear_cache(outq, allocator); + + outq->cache = lzma_alloc(alloc_size, allocator); + if (outq->cache == NULL) + return LZMA_MEM_ERROR; + + outq->cache->next = NULL; + outq->cache->allocated = size; + + ++outq->bufs_allocated; + outq->mem_allocated += alloc_size; + + return LZMA_OK; +} + + +extern lzma_outbuf * +lzma_outq_get_buf(lzma_outq *outq, void *worker) +{ + // Caller must have used lzma_outq_prealloc_buf() to ensure these. + assert(outq->bufs_in_use < outq->bufs_limit); + assert(outq->bufs_in_use < outq->bufs_allocated); + assert(outq->cache != NULL); + + lzma_outbuf *buf = outq->cache; + outq->cache = buf->next; + buf->next = NULL; + + if (outq->tail != NULL) { + assert(outq->head != NULL); + outq->tail->next = buf; + } else { + assert(outq->head == NULL); + outq->head = buf; + } + + outq->tail = buf; + + buf->worker = worker; + buf->finished = false; + buf->finish_ret = LZMA_STREAM_END; + buf->pos = 0; + buf->decoder_in_pos = 0; + + buf->unpadded_size = 0; + buf->uncompressed_size = 0; + + ++outq->bufs_in_use; + outq->mem_in_use += lzma_outq_outbuf_memusage(buf->allocated); + + return buf; +} + + +extern bool +lzma_outq_is_readable(const lzma_outq *outq) +{ + if (outq->head == NULL) + return false; + + return outq->read_pos < outq->head->pos || outq->head->finished; +} + + +extern lzma_ret +lzma_outq_read(lzma_outq *restrict outq, + const lzma_allocator *restrict allocator, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, + lzma_vli *restrict unpadded_size, + lzma_vli *restrict uncompressed_size) +{ + // There must be at least one buffer from which to read. + if (outq->bufs_in_use == 0) + return LZMA_OK; + + // Get the buffer. + lzma_outbuf *buf = outq->head; + + // Copy from the buffer to output. + // + // FIXME? In threaded decoder it may be bad to do this copy while + // the mutex is being held. + lzma_bufcpy(buf->buf, &outq->read_pos, buf->pos, + out, out_pos, out_size); + + // Return if we didn't get all the data from the buffer. + if (!buf->finished || outq->read_pos < buf->pos) + return LZMA_OK; + + // The buffer was finished. Tell the caller its size information. + if (unpadded_size != NULL) + *unpadded_size = buf->unpadded_size; + + if (uncompressed_size != NULL) + *uncompressed_size = buf->uncompressed_size; + + // Remember the return value. + const lzma_ret finish_ret = buf->finish_ret; + + // Free this buffer for further use. + move_head_to_cache(outq, allocator); + outq->read_pos = 0; + + return finish_ret; +} + + +extern void +lzma_outq_enable_partial_output(lzma_outq *outq, + void (*enable_partial_output)(void *worker)) +{ + if (outq->head != NULL && !outq->head->finished + && outq->head->worker != NULL) { + enable_partial_output(outq->head->worker); + + // Set it to NULL since calling it twice is pointless. + outq->head->worker = NULL; + } + + return; +} diff --git a/src/liblzma/common/outqueue.h b/src/liblzma/common/outqueue.h new file mode 100644 index 0000000000..25f071977a --- /dev/null +++ b/src/liblzma/common/outqueue.h @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file outqueue.h +/// \brief Output queue handling in multithreaded coding +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_OUTQUEUE_H +#define LZMA_OUTQUEUE_H + +#include "common.h" + + +/// Output buffer for a single thread +typedef struct lzma_outbuf_s lzma_outbuf; +struct lzma_outbuf_s { + /// Pointer to the next buffer. This is used for the cached buffers. + /// The worker thread must not modify this. + lzma_outbuf *next; + + /// This initialized by lzma_outq_get_buf() and + /// is used by lzma_outq_enable_partial_output(). + /// The worker thread must not modify this. + void *worker; + + /// Amount of memory allocated for buf[]. + /// The worker thread must not modify this. + size_t allocated; + + /// Writing position in the worker thread or, in other words, the + /// amount of finished data written to buf[] which can be copied out + /// + /// \note This is read by another thread and thus access + /// to this variable needs a mutex. + size_t pos; + + /// Decompression: Position in the input buffer in the worker thread + /// that matches the output "pos" above. This is used to detect if + /// more output might be possible from the worker thread: if it has + /// consumed all its input, then more output isn't possible. + /// + /// \note This is read by another thread and thus access + /// to this variable needs a mutex. + size_t decoder_in_pos; + + /// True when no more data will be written into this buffer. + /// + /// \note This is read by another thread and thus access + /// to this variable needs a mutex. + bool finished; + + /// Return value for lzma_outq_read() when the last byte from + /// a finished buffer has been read. Defaults to LZMA_STREAM_END. + /// This must *not* be LZMA_OK. The idea is to allow a decoder to + /// pass an error code to the main thread, setting the code here + /// together with finished = true. + lzma_ret finish_ret; + + /// Additional size information. lzma_outq_read() may read these + /// when "finished" is true. + lzma_vli unpadded_size; + lzma_vli uncompressed_size; + + /// Buffer of "allocated" bytes + uint8_t buf[]; +}; + + +typedef struct { + /// Linked list of buffers in use. The next output byte will be + /// read from the head and buffers for the next thread will be + /// appended to the tail. tail->next is always NULL. + lzma_outbuf *head; + lzma_outbuf *tail; + + /// Number of bytes read from head->buf[] in lzma_outq_read() + size_t read_pos; + + /// Linked list of allocated buffers that aren't currently used. + /// This way buffers of similar size can be reused and don't + /// need to be reallocated every time. For simplicity, all + /// cached buffers in the list have the same allocated size. + lzma_outbuf *cache; + + /// Total amount of memory allocated for buffers + uint64_t mem_allocated; + + /// Amount of memory used by the buffers that are in use in + /// the head...tail linked list. + uint64_t mem_in_use; + + /// Number of buffers in use in the head...tail list. If and only if + /// this is zero, the pointers head and tail above are NULL. + uint32_t bufs_in_use; + + /// Number of buffers allocated (in use + cached) + uint32_t bufs_allocated; + + /// Maximum allowed number of allocated buffers + uint32_t bufs_limit; +} lzma_outq; + + +/** + * \brief Calculate the memory usage of an output queue + * + * \return Approximate memory usage in bytes or UINT64_MAX on error. + */ +extern uint64_t lzma_outq_memusage(uint64_t buf_size_max, uint32_t threads); + + +/// \brief Initialize an output queue +/// +/// \param outq Pointer to an output queue. Before calling +/// this function the first time, *outq should +/// have been zeroed with memzero() so that this +/// function knows that there are no previous +/// allocations to free. +/// \param allocator Pointer to allocator or NULL +/// \param threads Number of buffers that may be in use +/// concurrently. Note that more than this number +/// of buffers may actually get allocated to +/// improve performance when buffers finish +/// out of order. The actual maximum number of +/// allocated buffers is derived from the number +/// of threads. +/// +/// \return - LZMA_OK +/// - LZMA_MEM_ERROR +/// +extern lzma_ret lzma_outq_init(lzma_outq *outq, + const lzma_allocator *allocator, uint32_t threads); + + +/// \brief Free the memory associated with the output queue +extern void lzma_outq_end(lzma_outq *outq, const lzma_allocator *allocator); + + +/// \brief Free all cached buffers that consume memory but aren't in use +extern void lzma_outq_clear_cache( + lzma_outq *outq, const lzma_allocator *allocator); + + +/// \brief Like lzma_outq_clear_cache() but might keep one buffer +/// +/// One buffer is not freed if its size is equal to keep_size. +/// This is useful if the caller knows that it will soon need a buffer of +/// keep_size bytes. This way it won't be freed and immediately reallocated. +extern void lzma_outq_clear_cache2( + lzma_outq *outq, const lzma_allocator *allocator, + size_t keep_size); + + +/// \brief Preallocate a new buffer into cache +/// +/// Splitting the buffer allocation into a separate function makes it +/// possible to ensure that way lzma_outq_get_buf() cannot fail. +/// If the preallocated buffer isn't actually used (for example, some +/// other error occurs), the caller has to do nothing as the buffer will +/// be used later or cleared from the cache when not needed. +/// +/// \return LZMA_OK on success, LZMA_MEM_ERROR if allocation fails +/// +extern lzma_ret lzma_outq_prealloc_buf( + lzma_outq *outq, const lzma_allocator *allocator, size_t size); + + +/// \brief Get a new buffer +/// +/// lzma_outq_prealloc_buf() must be used to ensure that there is a buffer +/// available before calling lzma_outq_get_buf(). +/// +extern lzma_outbuf *lzma_outq_get_buf(lzma_outq *outq, void *worker); + + +/// \brief Test if there is data ready to be read +/// +/// Call to this function must be protected with the same mutex that +/// is used to protect lzma_outbuf.finished. +/// +extern bool lzma_outq_is_readable(const lzma_outq *outq); + + +/// \brief Read finished data +/// +/// \param outq Pointer to an output queue +/// \param out Beginning of the output buffer +/// \param out_pos The next byte will be written to +/// out[*out_pos]. +/// \param out_size Size of the out buffer; the first byte into +/// which no data is written to is out[out_size]. +/// \param unpadded_size Unpadded Size from the Block encoder +/// \param uncompressed_size Uncompressed Size from the Block encoder +/// +/// \return - LZMA: All OK. Either no data was available or the buffer +/// being read didn't become empty yet. +/// - LZMA_STREAM_END: The buffer being read was finished. +/// *unpadded_size and *uncompressed_size were set if they +/// were not NULL. +/// +/// \note This reads lzma_outbuf.finished and .pos variables and thus +/// calls to this function need to be protected with a mutex. +/// +extern lzma_ret lzma_outq_read(lzma_outq *restrict outq, + const lzma_allocator *restrict allocator, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, lzma_vli *restrict unpadded_size, + lzma_vli *restrict uncompressed_size); + + +/// \brief Enable partial output from a worker thread +/// +/// If the buffer at the head of the output queue isn't finished, +/// this will call enable_partial_output on the worker associated with +/// that output buffer. +/// +/// \note This reads a lzma_outbuf.finished variable and thus +/// calls to this function need to be protected with a mutex. +/// +extern void lzma_outq_enable_partial_output(lzma_outq *outq, + void (*enable_partial_output)(void *worker)); + + +/// \brief Test if there is at least one buffer free +/// +/// This must be used before getting a new buffer with lzma_outq_get_buf(). +/// +static inline bool +lzma_outq_has_buf(const lzma_outq *outq) +{ + return outq->bufs_in_use < outq->bufs_limit; +} + + +/// \brief Test if the queue is completely empty +static inline bool +lzma_outq_is_empty(const lzma_outq *outq) +{ + return outq->bufs_in_use == 0; +} + + +/// \brief Get the amount of memory needed for a single lzma_outbuf +/// +/// \note Caller must check that the argument is significantly less +/// than SIZE_MAX to avoid an integer overflow! +static inline uint64_t +lzma_outq_outbuf_memusage(size_t buf_size) +{ + assert(buf_size <= SIZE_MAX - sizeof(lzma_outbuf)); + return sizeof(lzma_outbuf) + buf_size; +} + +#endif diff --git a/src/liblzma/common/stream_buffer_decoder.c b/src/liblzma/common/stream_buffer_decoder.c new file mode 100644 index 0000000000..c4f91fb498 --- /dev/null +++ b/src/liblzma/common/stream_buffer_decoder.c @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_buffer_decoder.c +/// \brief Single-call .xz Stream decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "stream_decoder.h" + + +extern LZMA_API(lzma_ret) +lzma_stream_buffer_decode(uint64_t *memlimit, uint32_t flags, + const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + // Sanity checks + if (in_pos == NULL || (in == NULL && *in_pos != in_size) + || *in_pos > in_size || out_pos == NULL + || (out == NULL && *out_pos != out_size) + || *out_pos > out_size) + return LZMA_PROG_ERROR; + + // Catch flags that are not allowed in buffer-to-buffer decoding. + if (flags & LZMA_TELL_ANY_CHECK) + return LZMA_PROG_ERROR; + + // Initialize the Stream decoder. + // TODO: We need something to tell the decoder that it can use the + // output buffer as workspace, and thus save significant amount of RAM. + lzma_next_coder stream_decoder = LZMA_NEXT_CODER_INIT; + lzma_ret ret = lzma_stream_decoder_init( + &stream_decoder, allocator, *memlimit, flags); + + if (ret == LZMA_OK) { + // Save the positions so that we can restore them in case + // an error occurs. + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + // Do the actual decoding. + ret = stream_decoder.code(stream_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + LZMA_FINISH); + + if (ret == LZMA_STREAM_END) { + ret = LZMA_OK; + } else { + // Something went wrong, restore the positions. + *in_pos = in_start; + *out_pos = out_start; + + if (ret == LZMA_OK) { + // Either the input was truncated or the + // output buffer was too small. + assert(*in_pos == in_size + || *out_pos == out_size); + + // If all the input was consumed, then the + // input is truncated, even if the output + // buffer is also full. This is because + // processing the last byte of the Stream + // never produces output. + if (*in_pos == in_size) + ret = LZMA_DATA_ERROR; + else + ret = LZMA_BUF_ERROR; + + } else if (ret == LZMA_MEMLIMIT_ERROR) { + // Let the caller know how much memory would + // have been needed. + uint64_t memusage; + (void)stream_decoder.memconfig( + stream_decoder.coder, + memlimit, &memusage, 0); + } + } + } + + // Free the decoder memory. This needs to be done even if + // initialization fails, because the internal API doesn't + // require the initialization function to free its memory on error. + lzma_next_end(&stream_decoder, allocator); + + return ret; +} diff --git a/src/liblzma/common/stream_buffer_encoder.c b/src/liblzma/common/stream_buffer_encoder.c new file mode 100644 index 0000000000..04d5869594 --- /dev/null +++ b/src/liblzma/common/stream_buffer_encoder.c @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_buffer_encoder.c +/// \brief Single-call .xz Stream encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "index.h" + + +/// Maximum size of Index that has exactly one Record. +/// Index Indicator + Number of Records + Record + CRC32 rounded up to +/// the next multiple of four. +#define INDEX_BOUND ((1 + 1 + 2 * LZMA_VLI_BYTES_MAX + 4 + 3) & ~3) + +/// Stream Header, Stream Footer, and Index +#define HEADERS_BOUND (2 * LZMA_STREAM_HEADER_SIZE + INDEX_BOUND) + + +extern LZMA_API(size_t) +lzma_stream_buffer_bound(size_t uncompressed_size) +{ + // Get the maximum possible size of a Block. + const size_t block_bound = lzma_block_buffer_bound(uncompressed_size); + if (block_bound == 0) + return 0; + + // Catch the possible integer overflow and also prevent the size of + // the Stream exceeding LZMA_VLI_MAX (theoretically possible on + // 64-bit systems). + if (my_min(SIZE_MAX, LZMA_VLI_MAX) - block_bound < HEADERS_BOUND) + return 0; + + return block_bound + HEADERS_BOUND; +} + + +extern LZMA_API(lzma_ret) +lzma_stream_buffer_encode(lzma_filter *filters, lzma_check check, + const lzma_allocator *allocator, + const uint8_t *in, size_t in_size, + uint8_t *out, size_t *out_pos_ptr, size_t out_size) +{ + // Sanity checks + if (filters == NULL || (unsigned int)(check) > LZMA_CHECK_ID_MAX + || (in == NULL && in_size != 0) || out == NULL + || out_pos_ptr == NULL || *out_pos_ptr > out_size) + return LZMA_PROG_ERROR; + + if (!lzma_check_is_supported(check)) + return LZMA_UNSUPPORTED_CHECK; + + // Note for the paranoids: Index encoder prevents the Stream from + // getting too big and still being accepted with LZMA_OK, and Block + // encoder catches if the input is too big. So we don't need to + // separately check if the buffers are too big. + + // Use a local copy. We update *out_pos_ptr only if everything + // succeeds. + size_t out_pos = *out_pos_ptr; + + // Check that there's enough space for both Stream Header and + // Stream Footer. + if (out_size - out_pos <= 2 * LZMA_STREAM_HEADER_SIZE) + return LZMA_BUF_ERROR; + + // Reserve space for Stream Footer so we don't need to check for + // available space again before encoding Stream Footer. + out_size -= LZMA_STREAM_HEADER_SIZE; + + // Encode the Stream Header. + lzma_stream_flags stream_flags = { + .version = 0, + .check = check, + }; + + if (lzma_stream_header_encode(&stream_flags, out + out_pos) + != LZMA_OK) + return LZMA_PROG_ERROR; + + out_pos += LZMA_STREAM_HEADER_SIZE; + + // Encode a Block but only if there is at least one byte of input. + lzma_block block = { + .version = 0, + .check = check, + .filters = filters, + }; + + if (in_size > 0) + return_if_error(lzma_block_buffer_encode(&block, allocator, + in, in_size, out, &out_pos, out_size)); + + // Index + { + // Create an Index. It will have one Record if there was + // at least one byte of input to encode. Otherwise the + // Index will be empty. + lzma_index *i = lzma_index_init(allocator); + if (i == NULL) + return LZMA_MEM_ERROR; + + lzma_ret ret = LZMA_OK; + + if (in_size > 0) + ret = lzma_index_append(i, allocator, + lzma_block_unpadded_size(&block), + block.uncompressed_size); + + // If adding the Record was successful, encode the Index + // and get its size which will be stored into Stream Footer. + if (ret == LZMA_OK) { + ret = lzma_index_buffer_encode( + i, out, &out_pos, out_size); + + stream_flags.backward_size = lzma_index_size(i); + } + + lzma_index_end(i, allocator); + + if (ret != LZMA_OK) + return ret; + } + + // Stream Footer. We have already reserved space for this. + if (lzma_stream_footer_encode(&stream_flags, out + out_pos) + != LZMA_OK) + return LZMA_PROG_ERROR; + + out_pos += LZMA_STREAM_HEADER_SIZE; + + // Everything went fine, make the new output position available + // to the application. + *out_pos_ptr = out_pos; + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_decoder.c b/src/liblzma/common/stream_decoder.c new file mode 100644 index 0000000000..94004b74a1 --- /dev/null +++ b/src/liblzma/common/stream_decoder.c @@ -0,0 +1,469 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_decoder.c +/// \brief Decodes .xz Streams +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "stream_decoder.h" +#include "block_decoder.h" +#include "index.h" + + +typedef struct { + enum { + SEQ_STREAM_HEADER, + SEQ_BLOCK_HEADER, + SEQ_BLOCK_INIT, + SEQ_BLOCK_RUN, + SEQ_INDEX, + SEQ_STREAM_FOOTER, + SEQ_STREAM_PADDING, + } sequence; + + /// Block decoder + lzma_next_coder block_decoder; + + /// Block options decoded by the Block Header decoder and used by + /// the Block decoder. + lzma_block block_options; + + /// Stream Flags from Stream Header + lzma_stream_flags stream_flags; + + /// Index is hashed so that it can be compared to the sizes of Blocks + /// with O(1) memory usage. + lzma_index_hash *index_hash; + + /// Memory usage limit + uint64_t memlimit; + + /// Amount of memory actually needed (only an estimate) + uint64_t memusage; + + /// If true, LZMA_NO_CHECK is returned if the Stream has + /// no integrity check. + bool tell_no_check; + + /// If true, LZMA_UNSUPPORTED_CHECK is returned if the Stream has + /// an integrity check that isn't supported by this liblzma build. + bool tell_unsupported_check; + + /// If true, LZMA_GET_CHECK is returned after decoding Stream Header. + bool tell_any_check; + + /// If true, we will tell the Block decoder to skip calculating + /// and verifying the integrity check. + bool ignore_check; + + /// If true, we will decode concatenated Streams that possibly have + /// Stream Padding between or after them. LZMA_STREAM_END is returned + /// once the application isn't giving us any new input (LZMA_FINISH), + /// and we aren't in the middle of a Stream, and possible + /// Stream Padding is a multiple of four bytes. + bool concatenated; + + /// When decoding concatenated Streams, this is true as long as we + /// are decoding the first Stream. This is needed to avoid misleading + /// LZMA_FORMAT_ERROR in case the later Streams don't have valid magic + /// bytes. + bool first_stream; + + /// Write position in buffer[] and position in Stream Padding + size_t pos; + + /// Buffer to hold Stream Header, Block Header, and Stream Footer. + /// Block Header has biggest maximum size. + uint8_t buffer[LZMA_BLOCK_HEADER_SIZE_MAX]; +} lzma_stream_coder; + + +static lzma_ret +stream_decoder_reset(lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + // Initialize the Index hash used to verify the Index. + coder->index_hash = lzma_index_hash_init(coder->index_hash, allocator); + if (coder->index_hash == NULL) + return LZMA_MEM_ERROR; + + // Reset the rest of the variables. + coder->sequence = SEQ_STREAM_HEADER; + coder->pos = 0; + + return LZMA_OK; +} + + +static lzma_ret +stream_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_stream_coder *coder = coder_ptr; + + // When decoding the actual Block, it may be able to produce more + // output even if we don't give it any new input. + while (true) + switch (coder->sequence) { + case SEQ_STREAM_HEADER: { + // Copy the Stream Header to the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + LZMA_STREAM_HEADER_SIZE); + + // Return if we didn't get the whole Stream Header yet. + if (coder->pos < LZMA_STREAM_HEADER_SIZE) + return LZMA_OK; + + coder->pos = 0; + + // Decode the Stream Header. + const lzma_ret ret = lzma_stream_header_decode( + &coder->stream_flags, coder->buffer); + if (ret != LZMA_OK) + return ret == LZMA_FORMAT_ERROR && !coder->first_stream + ? LZMA_DATA_ERROR : ret; + + // If we are decoding concatenated Streams, and the later + // Streams have invalid Header Magic Bytes, we give + // LZMA_DATA_ERROR instead of LZMA_FORMAT_ERROR. + coder->first_stream = false; + + // Copy the type of the Check so that Block Header and Block + // decoders see it. + coder->block_options.check = coder->stream_flags.check; + + // Even if we return LZMA_*_CHECK below, we want + // to continue from Block Header decoding. + coder->sequence = SEQ_BLOCK_HEADER; + + // Detect if there's no integrity check or if it is + // unsupported if those were requested by the application. + if (coder->tell_no_check && coder->stream_flags.check + == LZMA_CHECK_NONE) + return LZMA_NO_CHECK; + + if (coder->tell_unsupported_check + && !lzma_check_is_supported( + coder->stream_flags.check)) + return LZMA_UNSUPPORTED_CHECK; + + if (coder->tell_any_check) + return LZMA_GET_CHECK; + + FALLTHROUGH; + } + + case SEQ_BLOCK_HEADER: { + if (*in_pos >= in_size) + return LZMA_OK; + + if (coder->pos == 0) { + // Detect if it's Index. + if (in[*in_pos] == INDEX_INDICATOR) { + coder->sequence = SEQ_INDEX; + break; + } + + // Calculate the size of the Block Header. Note that + // Block Header decoder wants to see this byte too + // so don't advance *in_pos. + coder->block_options.header_size + = lzma_block_header_size_decode( + in[*in_pos]); + } + + // Copy the Block Header to the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + coder->block_options.header_size); + + // Return if we didn't get the whole Block Header yet. + if (coder->pos < coder->block_options.header_size) + return LZMA_OK; + + coder->pos = 0; + coder->sequence = SEQ_BLOCK_INIT; + FALLTHROUGH; + } + + case SEQ_BLOCK_INIT: { + // Checking memusage and doing the initialization needs + // its own sequence point because we need to be able to + // retry if we return LZMA_MEMLIMIT_ERROR. + + // Version 1 is needed to support the .ignore_check option. + coder->block_options.version = 1; + + // Set up a buffer to hold the filter chain. Block Header + // decoder will initialize all members of this array so + // we don't need to do it here. + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + coder->block_options.filters = filters; + + // Decode the Block Header. + return_if_error(lzma_block_header_decode(&coder->block_options, + allocator, coder->buffer)); + + // If LZMA_IGNORE_CHECK was used, this flag needs to be set. + // It has to be set after lzma_block_header_decode() because + // it always resets this to false. + coder->block_options.ignore_check = coder->ignore_check; + + // Check the memory usage limit. + const uint64_t memusage = lzma_raw_decoder_memusage(filters); + lzma_ret ret; + + if (memusage == UINT64_MAX) { + // One or more unknown Filter IDs. + ret = LZMA_OPTIONS_ERROR; + } else { + // Now we can set coder->memusage since we know that + // the filter chain is valid. We don't want + // lzma_memusage() to return UINT64_MAX in case of + // invalid filter chain. + coder->memusage = memusage; + + if (memusage > coder->memlimit) { + // The chain would need too much memory. + ret = LZMA_MEMLIMIT_ERROR; + } else { + // Memory usage is OK. + // Initialize the Block decoder. + ret = lzma_block_decoder_init( + &coder->block_decoder, + allocator, + &coder->block_options); + } + } + + // Free the allocated filter options since they are needed + // only to initialize the Block decoder. + lzma_filters_free(filters, allocator); + coder->block_options.filters = NULL; + + // Check if memory usage calculation and Block decoder + // initialization succeeded. + if (ret != LZMA_OK) + return ret; + + coder->sequence = SEQ_BLOCK_RUN; + FALLTHROUGH; + } + + case SEQ_BLOCK_RUN: { + const lzma_ret ret = coder->block_decoder.code( + coder->block_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + + if (ret != LZMA_STREAM_END) + return ret; + + // Block decoded successfully. Add the new size pair to + // the Index hash. + return_if_error(lzma_index_hash_append(coder->index_hash, + lzma_block_unpadded_size( + &coder->block_options), + coder->block_options.uncompressed_size)); + + coder->sequence = SEQ_BLOCK_HEADER; + break; + } + + case SEQ_INDEX: { + // If we don't have any input, don't call + // lzma_index_hash_decode() since it would return + // LZMA_BUF_ERROR, which we must not do here. + if (*in_pos >= in_size) + return LZMA_OK; + + // Decode the Index and compare it to the hash calculated + // from the sizes of the Blocks (if any). + const lzma_ret ret = lzma_index_hash_decode(coder->index_hash, + in, in_pos, in_size); + if (ret != LZMA_STREAM_END) + return ret; + + coder->sequence = SEQ_STREAM_FOOTER; + FALLTHROUGH; + } + + case SEQ_STREAM_FOOTER: { + // Copy the Stream Footer to the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + LZMA_STREAM_HEADER_SIZE); + + // Return if we didn't get the whole Stream Footer yet. + if (coder->pos < LZMA_STREAM_HEADER_SIZE) + return LZMA_OK; + + coder->pos = 0; + + // Decode the Stream Footer. The decoder gives + // LZMA_FORMAT_ERROR if the magic bytes don't match, + // so convert that return code to LZMA_DATA_ERROR. + lzma_stream_flags footer_flags; + const lzma_ret ret = lzma_stream_footer_decode( + &footer_flags, coder->buffer); + if (ret != LZMA_OK) + return ret == LZMA_FORMAT_ERROR + ? LZMA_DATA_ERROR : ret; + + // Check that Index Size stored in the Stream Footer matches + // the real size of the Index field. + if (lzma_index_hash_size(coder->index_hash) + != footer_flags.backward_size) + return LZMA_DATA_ERROR; + + // Compare that the Stream Flags fields are identical in + // both Stream Header and Stream Footer. + return_if_error(lzma_stream_flags_compare( + &coder->stream_flags, &footer_flags)); + + if (!coder->concatenated) + return LZMA_STREAM_END; + + coder->sequence = SEQ_STREAM_PADDING; + FALLTHROUGH; + } + + case SEQ_STREAM_PADDING: + assert(coder->concatenated); + + // Skip over possible Stream Padding. + while (true) { + if (*in_pos >= in_size) { + // Unless LZMA_FINISH was used, we cannot + // know if there's more input coming later. + if (action != LZMA_FINISH) + return LZMA_OK; + + // Stream Padding must be a multiple of + // four bytes. + return coder->pos == 0 + ? LZMA_STREAM_END + : LZMA_DATA_ERROR; + } + + // If the byte is not zero, it probably indicates + // beginning of a new Stream (or the file is corrupt). + if (in[*in_pos] != 0x00) + break; + + ++*in_pos; + coder->pos = (coder->pos + 1) & 3; + } + + // Stream Padding must be a multiple of four bytes (empty + // Stream Padding is OK). + if (coder->pos != 0) { + ++*in_pos; + return LZMA_DATA_ERROR; + } + + // Prepare to decode the next Stream. + return_if_error(stream_decoder_reset(coder, allocator)); + break; + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + // Never reached +} + + +static void +stream_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_stream_coder *coder = coder_ptr; + lzma_next_end(&coder->block_decoder, allocator); + lzma_index_hash_end(coder->index_hash, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_check +stream_decoder_get_check(const void *coder_ptr) +{ + const lzma_stream_coder *coder = coder_ptr; + return coder->stream_flags.check; +} + + +static lzma_ret +stream_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_stream_coder *coder = coder_ptr; + + *memusage = coder->memusage; + *old_memlimit = coder->memlimit; + + if (new_memlimit != 0) { + if (new_memlimit < coder->memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +extern lzma_ret +lzma_stream_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags) +{ + lzma_next_coder_init(&lzma_stream_decoder_init, next, allocator); + + if (flags & ~LZMA_SUPPORTED_FLAGS) + return LZMA_OPTIONS_ERROR; + + lzma_stream_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_stream_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &stream_decode; + next->end = &stream_decoder_end; + next->get_check = &stream_decoder_get_check; + next->memconfig = &stream_decoder_memconfig; + + coder->block_decoder = LZMA_NEXT_CODER_INIT; + coder->index_hash = NULL; + } + + coder->memlimit = my_max(1, memlimit); + coder->memusage = LZMA_MEMUSAGE_BASE; + coder->tell_no_check = (flags & LZMA_TELL_NO_CHECK) != 0; + coder->tell_unsupported_check + = (flags & LZMA_TELL_UNSUPPORTED_CHECK) != 0; + coder->tell_any_check = (flags & LZMA_TELL_ANY_CHECK) != 0; + coder->ignore_check = (flags & LZMA_IGNORE_CHECK) != 0; + coder->concatenated = (flags & LZMA_CONCATENATED) != 0; + coder->first_stream = true; + + return stream_decoder_reset(coder, allocator); +} + + +extern LZMA_API(lzma_ret) +lzma_stream_decoder(lzma_stream *strm, uint64_t memlimit, uint32_t flags) +{ + lzma_next_strm_init(lzma_stream_decoder_init, strm, memlimit, flags); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_decoder.h b/src/liblzma/common/stream_decoder.h new file mode 100644 index 0000000000..5803715374 --- /dev/null +++ b/src/liblzma/common/stream_decoder.h @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_decoder.h +/// \brief Decodes .xz Streams +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_STREAM_DECODER_H +#define LZMA_STREAM_DECODER_H + +#include "common.h" + +extern lzma_ret lzma_stream_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags); + +#endif diff --git a/src/liblzma/common/stream_decoder_mt.c b/src/liblzma/common/stream_decoder_mt.c new file mode 100644 index 0000000000..271f9b07c4 --- /dev/null +++ b/src/liblzma/common/stream_decoder_mt.c @@ -0,0 +1,2015 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_decoder_mt.c +/// \brief Multithreaded .xz Stream decoder +// +// Authors: Sebastian Andrzej Siewior +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "block_decoder.h" +#include "stream_decoder.h" +#include "index.h" +#include "outqueue.h" + + +typedef enum { + /// Waiting for work. + /// Main thread may change this to THR_RUN or THR_EXIT. + THR_IDLE, + + /// Decoding is in progress. + /// Main thread may change this to THR_IDLE or THR_EXIT. + /// The worker thread may change this to THR_IDLE. + THR_RUN, + + /// The main thread wants the thread to exit. + THR_EXIT, + +} worker_state; + + +typedef enum { + /// Partial updates (storing of worker thread progress + /// to lzma_outbuf) are disabled. + PARTIAL_DISABLED, + + /// Main thread requests partial updates to be enabled but + /// no partial update has been done by the worker thread yet. + /// + /// Changing from PARTIAL_DISABLED to PARTIAL_START requires + /// use of the worker-thread mutex. Other transitions don't + /// need a mutex. + PARTIAL_START, + + /// Partial updates are enabled and the worker thread has done + /// at least one partial update. + PARTIAL_ENABLED, + +} partial_update_mode; + + +struct worker_thread { + /// Worker state is protected with our mutex. + worker_state state; + + /// Input buffer that will contain the whole Block except Block Header. + uint8_t *in; + + /// Amount of memory allocated for "in" + size_t in_size; + + /// Number of bytes written to "in" by the main thread + size_t in_filled; + + /// Number of bytes consumed from "in" by the worker thread. + size_t in_pos; + + /// Amount of uncompressed data that has been decoded. This local + /// copy is needed because updating outbuf->pos requires locking + /// the main mutex (coder->mutex). + size_t out_pos; + + /// Pointer to the main structure is needed to (1) lock the main + /// mutex (coder->mutex) when updating outbuf->pos and (2) when + /// putting this thread back to the stack of free threads. + struct lzma_stream_coder *coder; + + /// The allocator is set by the main thread. Since a copy of the + /// pointer is kept here, the application must not change the + /// allocator before calling lzma_end(). + const lzma_allocator *allocator; + + /// Output queue buffer to which the uncompressed data is written. + lzma_outbuf *outbuf; + + /// Amount of compressed data that has already been decompressed. + /// This is updated from in_pos when our mutex is locked. + /// This is size_t, not uint64_t, because per-thread progress + /// is limited to sizes of allocated buffers. + size_t progress_in; + + /// Like progress_in but for uncompressed data. + size_t progress_out; + + /// Updating outbuf->pos requires locking the main mutex + /// (coder->mutex). Since the main thread will only read output + /// from the oldest outbuf in the queue, only the worker thread + /// that is associated with the oldest outbuf needs to update its + /// outbuf->pos. This avoids useless mutex contention that would + /// happen if all worker threads were frequently locking the main + /// mutex to update their outbuf->pos. + /// + /// Only when partial_update is something else than PARTIAL_DISABLED, + /// this worker thread will update outbuf->pos after each call to + /// the Block decoder. + partial_update_mode partial_update; + + /// Block decoder + lzma_next_coder block_decoder; + + /// Thread-specific Block options are needed because the Block + /// decoder modifies the struct given to it at initialization. + lzma_block block_options; + + /// Filter chain memory usage + uint64_t mem_filters; + + /// Next structure in the stack of free worker threads. + struct worker_thread *next; + + mythread_mutex mutex; + mythread_cond cond; + + /// The ID of this thread is used to join the thread + /// when it's not needed anymore. + mythread thread_id; +}; + + +struct lzma_stream_coder { + enum { + SEQ_STREAM_HEADER, + SEQ_BLOCK_HEADER, + SEQ_BLOCK_INIT, + SEQ_BLOCK_THR_INIT, + SEQ_BLOCK_THR_RUN, + SEQ_BLOCK_DIRECT_INIT, + SEQ_BLOCK_DIRECT_RUN, + SEQ_INDEX_WAIT_OUTPUT, + SEQ_INDEX_DECODE, + SEQ_STREAM_FOOTER, + SEQ_STREAM_PADDING, + SEQ_ERROR, + } sequence; + + /// Block decoder + lzma_next_coder block_decoder; + + /// Every Block Header will be decoded into this structure. + /// This is also used to initialize a Block decoder when in + /// direct mode. In threaded mode, a thread-specific copy will + /// be made for decoder initialization because the Block decoder + /// will modify the structure given to it. + lzma_block block_options; + + /// Buffer to hold a filter chain for Block Header decoding and + /// initialization. These are freed after successful Block decoder + /// initialization or at stream_decoder_mt_end(). The thread-specific + /// copy of block_options won't hold a pointer to filters[] after + /// initialization. + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + /// Stream Flags from Stream Header + lzma_stream_flags stream_flags; + + /// Index is hashed so that it can be compared to the sizes of Blocks + /// with O(1) memory usage. + lzma_index_hash *index_hash; + + + /// Maximum wait time if cannot use all the input and cannot + /// fill the output buffer. This is in milliseconds. + uint32_t timeout; + + + /// Error code from a worker thread. + /// + /// \note Use mutex. + lzma_ret thread_error; + + /// Error code to return after pending output has been copied out. If + /// set in read_output_and_wait(), this is a mirror of thread_error. + /// If set in stream_decode_mt() then it's, for example, error that + /// occurred when decoding Block Header. + lzma_ret pending_error; + + /// Number of threads that will be created at maximum. + uint32_t threads_max; + + /// Number of thread structures that have been initialized from + /// "threads", and thus the number of worker threads actually + /// created so far. + uint32_t threads_initialized; + + /// Array of allocated thread-specific structures. When no threads + /// are in use (direct mode) this is NULL. In threaded mode this + /// points to an array of threads_max number of worker_thread structs. + struct worker_thread *threads; + + /// Stack of free threads. When a thread finishes, it puts itself + /// back into this stack. This starts as empty because threads + /// are created only when actually needed. + /// + /// \note Use mutex. + struct worker_thread *threads_free; + + /// The most recent worker thread to which the main thread writes + /// the new input from the application. + struct worker_thread *thr; + + /// Output buffer queue for decompressed data from the worker threads + /// + /// \note Use mutex with operations that need it. + lzma_outq outq; + + mythread_mutex mutex; + mythread_cond cond; + + + /// Memory usage that will not be exceeded in multi-threaded mode. + /// Single-threaded mode can exceed this even by a large amount. + uint64_t memlimit_threading; + + /// Memory usage limit that should never be exceeded. + /// LZMA_MEMLIMIT_ERROR will be returned if decoding isn't possible + /// even in single-threaded mode without exceeding this limit. + uint64_t memlimit_stop; + + /// Amount of memory in use by the direct mode decoder + /// (coder->block_decoder). In threaded mode this is 0. + uint64_t mem_direct_mode; + + /// Amount of memory needed by the running worker threads. + /// This doesn't include the memory needed by the output buffer. + /// + /// \note Use mutex. + uint64_t mem_in_use; + + /// Amount of memory used by the idle (cached) threads. + /// + /// \note Use mutex. + uint64_t mem_cached; + + + /// Amount of memory needed for the filter chain of the next Block. + uint64_t mem_next_filters; + + /// Amount of memory needed for the thread-specific input buffer + /// for the next Block. + uint64_t mem_next_in; + + /// Amount of memory actually needed to decode the next Block + /// in threaded mode. This is + /// mem_next_filters + mem_next_in + memory needed for lzma_outbuf. + uint64_t mem_next_block; + + + /// Amount of compressed data in Stream Header + Blocks that have + /// already been finished. + /// + /// \note Use mutex. + uint64_t progress_in; + + /// Amount of uncompressed data in Blocks that have already + /// been finished. + /// + /// \note Use mutex. + uint64_t progress_out; + + + /// If true, LZMA_NO_CHECK is returned if the Stream has + /// no integrity check. + bool tell_no_check; + + /// If true, LZMA_UNSUPPORTED_CHECK is returned if the Stream has + /// an integrity check that isn't supported by this liblzma build. + bool tell_unsupported_check; + + /// If true, LZMA_GET_CHECK is returned after decoding Stream Header. + bool tell_any_check; + + /// If true, we will tell the Block decoder to skip calculating + /// and verifying the integrity check. + bool ignore_check; + + /// If true, we will decode concatenated Streams that possibly have + /// Stream Padding between or after them. LZMA_STREAM_END is returned + /// once the application isn't giving us any new input (LZMA_FINISH), + /// and we aren't in the middle of a Stream, and possible + /// Stream Padding is a multiple of four bytes. + bool concatenated; + + /// If true, we will return any errors immediately instead of first + /// producing all output before the location of the error. + bool fail_fast; + + + /// When decoding concatenated Streams, this is true as long as we + /// are decoding the first Stream. This is needed to avoid misleading + /// LZMA_FORMAT_ERROR in case the later Streams don't have valid magic + /// bytes. + bool first_stream; + + /// This is used to track if the previous call to stream_decode_mt() + /// had output space (*out_pos < out_size) and managed to fill the + /// output buffer (*out_pos == out_size). This may be set to true + /// in read_output_and_wait(). This is read and then reset to false + /// at the beginning of stream_decode_mt(). + /// + /// This is needed to support applications that call lzma_code() in + /// such a way that more input is provided only when lzma_code() + /// didn't fill the output buffer completely. Basically, this makes + /// it easier to convert such applications from single-threaded + /// decoder to multi-threaded decoder. + bool out_was_filled; + + /// Write position in buffer[] and position in Stream Padding + size_t pos; + + /// Buffer to hold Stream Header, Block Header, and Stream Footer. + /// Block Header has biggest maximum size. + uint8_t buffer[LZMA_BLOCK_HEADER_SIZE_MAX]; +}; + + +/// Enables updating of outbuf->pos. This is a callback function that is +/// used with lzma_outq_enable_partial_output(). +static void +worker_enable_partial_update(void *thr_ptr) +{ + struct worker_thread *thr = thr_ptr; + + mythread_sync(thr->mutex) { + thr->partial_update = PARTIAL_START; + mythread_cond_signal(&thr->cond); + } +} + + +static MYTHREAD_RET_TYPE +worker_decoder(void *thr_ptr) +{ + struct worker_thread *thr = thr_ptr; + size_t in_filled; + partial_update_mode partial_update; + lzma_ret ret; + +next_loop_lock: + + mythread_mutex_lock(&thr->mutex); +next_loop_unlocked: + + if (thr->state == THR_IDLE) { + mythread_cond_wait(&thr->cond, &thr->mutex); + goto next_loop_unlocked; + } + + if (thr->state == THR_EXIT) { + mythread_mutex_unlock(&thr->mutex); + + lzma_free(thr->in, thr->allocator); + lzma_next_end(&thr->block_decoder, thr->allocator); + + mythread_mutex_destroy(&thr->mutex); + mythread_cond_destroy(&thr->cond); + + return MYTHREAD_RET_VALUE; + } + + assert(thr->state == THR_RUN); + + // Update progress info for get_progress(). + thr->progress_in = thr->in_pos; + thr->progress_out = thr->out_pos; + + // If we don't have any new input, wait for a signal from the main + // thread except if partial output has just been enabled. In that + // case we will do one normal run so that the partial output info + // gets passed to the main thread. The call to block_decoder.code() + // is useless but harmless as it can occur only once per Block. + in_filled = thr->in_filled; + partial_update = thr->partial_update; + + if (in_filled == thr->in_pos && partial_update != PARTIAL_START) { + mythread_cond_wait(&thr->cond, &thr->mutex); + goto next_loop_unlocked; + } + + mythread_mutex_unlock(&thr->mutex); + + // Pass the input in small chunks to the Block decoder. + // This way we react reasonably fast if we are told to stop/exit, + // and (when partial update is enabled) we tell about our progress + // to the main thread frequently enough. + const size_t chunk_size = 16384; + if ((in_filled - thr->in_pos) > chunk_size) + in_filled = thr->in_pos + chunk_size; + + ret = thr->block_decoder.code( + thr->block_decoder.coder, thr->allocator, + thr->in, &thr->in_pos, in_filled, + thr->outbuf->buf, &thr->out_pos, + thr->outbuf->allocated, LZMA_RUN); + + if (ret == LZMA_OK) { + if (partial_update != PARTIAL_DISABLED) { + // The main thread uses thr->mutex to change from + // PARTIAL_DISABLED to PARTIAL_START. The main thread + // doesn't care about this variable after that so we + // can safely change it here to PARTIAL_ENABLED + // without a mutex. + thr->partial_update = PARTIAL_ENABLED; + + // The main thread is reading decompressed data + // from thr->outbuf. Tell the main thread about + // our progress. + // + // NOTE: It's possible that we consumed input without + // producing any new output so it's possible that + // only in_pos has changed. In case of PARTIAL_START + // it is possible that neither in_pos nor out_pos has + // changed. + mythread_sync(thr->coder->mutex) { + thr->outbuf->pos = thr->out_pos; + thr->outbuf->decoder_in_pos = thr->in_pos; + mythread_cond_signal(&thr->coder->cond); + } + } + + goto next_loop_lock; + } + + // Either we finished successfully (LZMA_STREAM_END) or an error + // occurred. + // + // The sizes are in the Block Header and the Block decoder + // checks that they match, thus we know these: + assert(ret != LZMA_STREAM_END || thr->in_pos == thr->in_size); + assert(ret != LZMA_STREAM_END + || thr->out_pos == thr->block_options.uncompressed_size); + + mythread_sync(thr->mutex) { + // Block decoder ensures this, but do a sanity check anyway + // because thr->in_filled < thr->in_size means that the main + // thread is still writing to thr->in. + if (ret == LZMA_STREAM_END && thr->in_filled != thr->in_size) { + assert(0); + ret = LZMA_PROG_ERROR; + } + + if (thr->state != THR_EXIT) + thr->state = THR_IDLE; + } + + // Free the input buffer. Don't update in_size as we need + // it later to update thr->coder->mem_in_use. + // + // This step is skipped if an error occurred because the main thread + // might still be writing to thr->in. The memory will be freed after + // threads_end() sets thr->state = THR_EXIT. + if (ret == LZMA_STREAM_END) { + lzma_free(thr->in, thr->allocator); + thr->in = NULL; + } + + mythread_sync(thr->coder->mutex) { + // Move our progress info to the main thread. + thr->coder->progress_in += thr->in_pos; + thr->coder->progress_out += thr->out_pos; + thr->progress_in = 0; + thr->progress_out = 0; + + // Mark the outbuf as finished. + thr->outbuf->pos = thr->out_pos; + thr->outbuf->decoder_in_pos = thr->in_pos; + thr->outbuf->finished = true; + thr->outbuf->finish_ret = ret; + thr->outbuf = NULL; + + // If an error occurred, tell it to the main thread. + if (ret != LZMA_STREAM_END + && thr->coder->thread_error == LZMA_OK) + thr->coder->thread_error = ret; + + // Return the worker thread to the stack of available + // threads only if no errors occurred. + if (ret == LZMA_STREAM_END) { + // Update memory usage counters. + thr->coder->mem_in_use -= thr->in_size; + thr->coder->mem_in_use -= thr->mem_filters; + thr->coder->mem_cached += thr->mem_filters; + + // Put this thread to the stack of free threads. + thr->next = thr->coder->threads_free; + thr->coder->threads_free = thr; + } + + mythread_cond_signal(&thr->coder->cond); + } + + goto next_loop_lock; +} + + +/// Tells the worker threads to exit and waits for them to terminate. +static void +threads_end(struct lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + coder->threads[i].state = THR_EXIT; + mythread_cond_signal(&coder->threads[i].cond); + } + } + + for (uint32_t i = 0; i < coder->threads_initialized; ++i) + mythread_join(coder->threads[i].thread_id); + + lzma_free(coder->threads, allocator); + coder->threads_initialized = 0; + coder->threads = NULL; + coder->threads_free = NULL; + + // The threads don't update these when they exit. Do it here. + coder->mem_in_use = 0; + coder->mem_cached = 0; + + return; +} + + +/// Tell worker threads to stop without doing any cleaning up. +/// The clean up will be done when threads_exit() is called; +/// it's not possible to reuse the threads after threads_stop(). +/// +/// This is called before returning an unrecoverable error code +/// to the application. It would be waste of processor time +/// to keep the threads running in such a situation. +static void +threads_stop(struct lzma_stream_coder *coder) +{ + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + // The threads that are in the THR_RUN state will stop + // when they check the state the next time. There's no + // need to signal coder->threads[i].cond. + mythread_sync(coder->threads[i].mutex) { + coder->threads[i].state = THR_IDLE; + } + } + + return; +} + + +/// Initialize a new worker_thread structure and create a new thread. +static lzma_ret +initialize_new_thread(struct lzma_stream_coder *coder, + const lzma_allocator *allocator) +{ + // Allocate the coder->threads array if needed. It's done here instead + // of when initializing the decoder because we don't need this if we + // use the direct mode (we may even free coder->threads in the middle + // of the file if we switch from threaded to direct mode). + if (coder->threads == NULL) { + coder->threads = lzma_alloc( + coder->threads_max * sizeof(struct worker_thread), + allocator); + + if (coder->threads == NULL) + return LZMA_MEM_ERROR; + } + + // Pick a free structure. + assert(coder->threads_initialized < coder->threads_max); + struct worker_thread *thr + = &coder->threads[coder->threads_initialized]; + + if (mythread_mutex_init(&thr->mutex)) + goto error_mutex; + + if (mythread_cond_init(&thr->cond)) + goto error_cond; + + thr->state = THR_IDLE; + thr->in = NULL; + thr->in_size = 0; + thr->allocator = allocator; + thr->coder = coder; + thr->outbuf = NULL; + thr->block_decoder = LZMA_NEXT_CODER_INIT; + thr->mem_filters = 0; + + if (mythread_create(&thr->thread_id, worker_decoder, thr)) + goto error_thread; + + ++coder->threads_initialized; + coder->thr = thr; + + return LZMA_OK; + +error_thread: + mythread_cond_destroy(&thr->cond); + +error_cond: + mythread_mutex_destroy(&thr->mutex); + +error_mutex: + return LZMA_MEM_ERROR; +} + + +static lzma_ret +get_thread(struct lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + // If there is a free structure on the stack, use it. + mythread_sync(coder->mutex) { + if (coder->threads_free != NULL) { + coder->thr = coder->threads_free; + coder->threads_free = coder->threads_free->next; + + // The thread is no longer in the cache so subtract + // it from the cached memory usage. Don't add it + // to mem_in_use though; the caller will handle it + // since it knows how much memory it will actually + // use (the filter chain might change). + coder->mem_cached -= coder->thr->mem_filters; + } + } + + if (coder->thr == NULL) { + assert(coder->threads_initialized < coder->threads_max); + + // Initialize a new thread. + return_if_error(initialize_new_thread(coder, allocator)); + } + + coder->thr->in_filled = 0; + coder->thr->in_pos = 0; + coder->thr->out_pos = 0; + + coder->thr->progress_in = 0; + coder->thr->progress_out = 0; + + coder->thr->partial_update = PARTIAL_DISABLED; + + return LZMA_OK; +} + + +static lzma_ret +read_output_and_wait(struct lzma_stream_coder *coder, + const lzma_allocator *allocator, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, + bool *input_is_possible, + bool waiting_allowed, + mythread_condtime *wait_abs, bool *has_blocked) +{ + lzma_ret ret = LZMA_OK; + + mythread_sync(coder->mutex) { + do { + // Get as much output from the queue as is possible + // without blocking. + const size_t out_start = *out_pos; + do { + ret = lzma_outq_read(&coder->outq, allocator, + out, out_pos, out_size, + NULL, NULL); + + // If a Block was finished, tell the worker + // thread of the next Block (if it is still + // running) to start telling the main thread + // when new output is available. + if (ret == LZMA_STREAM_END) + lzma_outq_enable_partial_output( + &coder->outq, + &worker_enable_partial_update); + + // Loop until a Block wasn't finished. + // It's important to loop around even if + // *out_pos == out_size because there could + // be an empty Block that will return + // LZMA_STREAM_END without needing any + // output space. + } while (ret == LZMA_STREAM_END); + + // Check if lzma_outq_read reported an error from + // the Block decoder. + if (ret != LZMA_OK) + break; + + // If the output buffer is now full but it wasn't full + // when this function was called, set out_was_filled. + // This way the next call to stream_decode_mt() knows + // that some output was produced and no output space + // remained in the previous call to stream_decode_mt(). + if (*out_pos == out_size && *out_pos != out_start) + coder->out_was_filled = true; + + // Check if any thread has indicated an error. + if (coder->thread_error != LZMA_OK) { + // If LZMA_FAIL_FAST was used, report errors + // from worker threads immediately. + if (coder->fail_fast) { + ret = coder->thread_error; + break; + } + + // Otherwise set pending_error. The value we + // set here will not actually get used other + // than working as a flag that an error has + // occurred. This is because in SEQ_ERROR + // all output before the error will be read + // first by calling this function, and once we + // reach the location of the (first) error the + // error code from the above lzma_outq_read() + // will be returned to the application. + // + // Use LZMA_PROG_ERROR since the value should + // never leak to the application. It's + // possible that pending_error has already + // been set but that doesn't matter: if we get + // here, pending_error only works as a flag. + coder->pending_error = LZMA_PROG_ERROR; + } + + // Check if decoding of the next Block can be started. + // The memusage of the active threads must be low + // enough, there must be a free buffer slot in the + // output queue, and there must be a free thread + // (that can be either created or an existing one + // reused). + // + // NOTE: This is checked after reading the output + // above because reading the output can free a slot in + // the output queue and also reduce active memusage. + // + // NOTE: If output queue is empty, then input will + // always be possible. + if (input_is_possible != NULL + && coder->memlimit_threading + - coder->mem_in_use + - coder->outq.mem_in_use + >= coder->mem_next_block + && lzma_outq_has_buf(&coder->outq) + && (coder->threads_initialized + < coder->threads_max + || coder->threads_free + != NULL)) { + *input_is_possible = true; + break; + } + + // If the caller doesn't want us to block, return now. + if (!waiting_allowed) + break; + + // This check is needed only when input_is_possible + // is NULL. We must return if we aren't waiting for + // input to become possible and there is no more + // output coming from the queue. + if (lzma_outq_is_empty(&coder->outq)) { + assert(input_is_possible == NULL); + break; + } + + // If there is more data available from the queue, + // our out buffer must be full and we need to return + // so that the application can provide more output + // space. + // + // NOTE: In general lzma_outq_is_readable() can return + // true also when there are no more bytes available. + // This can happen when a Block has finished without + // providing any new output. We know that this is not + // the case because in the beginning of this loop we + // tried to read as much as possible even when we had + // no output space left and the mutex has been locked + // all the time (so worker threads cannot have changed + // anything). Thus there must be actual pending output + // in the queue. + if (lzma_outq_is_readable(&coder->outq)) { + assert(*out_pos == out_size); + break; + } + + // If the application stops providing more input + // in the middle of a Block, there will eventually + // be one worker thread left that is stuck waiting for + // more input (that might never arrive) and a matching + // outbuf which the worker thread cannot finish due + // to lack of input. We must detect this situation, + // otherwise we would end up waiting indefinitely + // (if no timeout is in use) or keep returning + // LZMA_TIMED_OUT while making no progress. Thus, the + // application would never get LZMA_BUF_ERROR from + // lzma_code() which would tell the application that + // no more progress is possible. No LZMA_BUF_ERROR + // means that, for example, truncated .xz files could + // cause an infinite loop. + // + // A worker thread doing partial updates will + // store not only the output position in outbuf->pos + // but also the matching input position in + // outbuf->decoder_in_pos. Here we check if that + // input position matches the amount of input that + // the worker thread has been given (in_filled). + // If so, we must return and not wait as no more + // output will be coming without first getting more + // input to the worker thread. If the application + // keeps calling lzma_code() without providing more + // input, it will eventually get LZMA_BUF_ERROR. + // + // NOTE: We can read partial_update and in_filled + // without thr->mutex as only the main thread + // modifies these variables. decoder_in_pos requires + // coder->mutex which we are already holding. + if (coder->thr != NULL && coder->thr->partial_update + != PARTIAL_DISABLED) { + // There is exactly one outbuf in the queue. + assert(coder->thr->outbuf == coder->outq.head); + assert(coder->thr->outbuf == coder->outq.tail); + + if (coder->thr->outbuf->decoder_in_pos + == coder->thr->in_filled) + break; + } + + // Wait for input or output to become possible. + if (coder->timeout != 0) { + // See the comment in stream_encoder_mt.c + // about why mythread_condtime_set() is used + // like this. + // + // FIXME? + // In contrast to the encoder, this calls + // _condtime_set while the mutex is locked. + if (!*has_blocked) { + *has_blocked = true; + mythread_condtime_set(wait_abs, + &coder->cond, + coder->timeout); + } + + if (mythread_cond_timedwait(&coder->cond, + &coder->mutex, + wait_abs) != 0) { + ret = LZMA_TIMED_OUT; + break; + } + } else { + mythread_cond_wait(&coder->cond, + &coder->mutex); + } + } while (ret == LZMA_OK); + } + + // If we are returning an error, then the application cannot get + // more output from us and thus keeping the threads running is + // useless and waste of CPU time. + if (ret != LZMA_OK && ret != LZMA_TIMED_OUT) + threads_stop(coder); + + return ret; +} + + +static lzma_ret +decode_block_header(struct lzma_stream_coder *coder, + const lzma_allocator *allocator, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size) +{ + if (*in_pos >= in_size) + return LZMA_OK; + + if (coder->pos == 0) { + // Detect if it's Index. + if (in[*in_pos] == INDEX_INDICATOR) + return LZMA_INDEX_DETECTED; + + // Calculate the size of the Block Header. Note that + // Block Header decoder wants to see this byte too + // so don't advance *in_pos. + coder->block_options.header_size + = lzma_block_header_size_decode( + in[*in_pos]); + } + + // Copy the Block Header to the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + coder->block_options.header_size); + + // Return if we didn't get the whole Block Header yet. + if (coder->pos < coder->block_options.header_size) + return LZMA_OK; + + coder->pos = 0; + + // Version 1 is needed to support the .ignore_check option. + coder->block_options.version = 1; + + // Block Header decoder will initialize all members of this array + // so we don't need to do it here. + coder->block_options.filters = coder->filters; + + // Decode the Block Header. + return_if_error(lzma_block_header_decode(&coder->block_options, + allocator, coder->buffer)); + + // If LZMA_IGNORE_CHECK was used, this flag needs to be set. + // It has to be set after lzma_block_header_decode() because + // it always resets this to false. + coder->block_options.ignore_check = coder->ignore_check; + + // coder->block_options is ready now. + return LZMA_STREAM_END; +} + + +/// Get the size of the Compressed Data + Block Padding + Check. +static size_t +comp_blk_size(const struct lzma_stream_coder *coder) +{ + return vli_ceil4(coder->block_options.compressed_size) + + lzma_check_size(coder->stream_flags.check); +} + + +/// Returns true if the size (compressed or uncompressed) is such that +/// threaded decompression cannot be used. Sizes that are too big compared +/// to SIZE_MAX must be rejected to avoid integer overflows and truncations +/// when lzma_vli is assigned to a size_t. +static bool +is_direct_mode_needed(lzma_vli size) +{ + return size == LZMA_VLI_UNKNOWN || size > SIZE_MAX / 3; +} + + +static lzma_ret +stream_decoder_reset(struct lzma_stream_coder *coder, + const lzma_allocator *allocator) +{ + // Initialize the Index hash used to verify the Index. + coder->index_hash = lzma_index_hash_init(coder->index_hash, allocator); + if (coder->index_hash == NULL) + return LZMA_MEM_ERROR; + + // Reset the rest of the variables. + coder->sequence = SEQ_STREAM_HEADER; + coder->pos = 0; + + return LZMA_OK; +} + + +static lzma_ret +stream_decode_mt(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, lzma_action action) +{ + struct lzma_stream_coder *coder = coder_ptr; + + mythread_condtime wait_abs; + bool has_blocked = false; + + // Determine if in SEQ_BLOCK_HEADER and SEQ_BLOCK_THR_RUN we should + // tell read_output_and_wait() to wait until it can fill the output + // buffer (or a timeout occurs). Two conditions must be met: + // + // (1) If the caller provided no new input. The reason for this + // can be, for example, the end of the file or that there is + // a pause in the input stream and more input is available + // a little later. In this situation we should wait for output + // because otherwise we would end up in a busy-waiting loop where + // we make no progress and the application just calls us again + // without providing any new input. This would then result in + // LZMA_BUF_ERROR even though more output would be available + // once the worker threads decode more data. + // + // (2) Even if (1) is true, we will not wait if the previous call to + // this function managed to produce some output and the output + // buffer became full. This is for compatibility with applications + // that call lzma_code() in such a way that new input is provided + // only when the output buffer didn't become full. Without this + // trick such applications would have bad performance (bad + // parallelization due to decoder not getting input fast enough). + // + // NOTE: Such loops might require that timeout is disabled (0) + // if they assume that output-not-full implies that all input has + // been consumed. If and only if timeout is enabled, we may return + // when output isn't full *and* not all input has been consumed. + // + // However, if LZMA_FINISH is used, the above is ignored and we always + // wait (timeout can still cause us to return) because we know that + // we won't get any more input. This matters if the input file is + // truncated and we are doing single-shot decoding, that is, + // timeout = 0 and LZMA_FINISH is used on the first call to + // lzma_code() and the output buffer is known to be big enough + // to hold all uncompressed data: + // + // - If LZMA_FINISH wasn't handled specially, we could return + // LZMA_OK before providing all output that is possible with the + // truncated input. The rest would be available if lzma_code() was + // called again but then it's not single-shot decoding anymore. + // + // - By handling LZMA_FINISH specially here, the first call will + // produce all the output, matching the behavior of the + // single-threaded decoder. + // + // So it's a very specific corner case but also easy to avoid. Note + // that this special handling of LZMA_FINISH has no effect for + // single-shot decoding when the input file is valid (not truncated); + // premature LZMA_OK wouldn't be possible as long as timeout = 0. + const bool waiting_allowed = action == LZMA_FINISH + || (*in_pos == in_size && !coder->out_was_filled); + coder->out_was_filled = false; + + while (true) + switch (coder->sequence) { + case SEQ_STREAM_HEADER: { + // Copy the Stream Header to the internal buffer. + const size_t in_old = *in_pos; + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + LZMA_STREAM_HEADER_SIZE); + coder->progress_in += *in_pos - in_old; + + // Return if we didn't get the whole Stream Header yet. + if (coder->pos < LZMA_STREAM_HEADER_SIZE) + return LZMA_OK; + + coder->pos = 0; + + // Decode the Stream Header. + const lzma_ret ret = lzma_stream_header_decode( + &coder->stream_flags, coder->buffer); + if (ret != LZMA_OK) + return ret == LZMA_FORMAT_ERROR && !coder->first_stream + ? LZMA_DATA_ERROR : ret; + + // If we are decoding concatenated Streams, and the later + // Streams have invalid Header Magic Bytes, we give + // LZMA_DATA_ERROR instead of LZMA_FORMAT_ERROR. + coder->first_stream = false; + + // Copy the type of the Check so that Block Header and Block + // decoders see it. + coder->block_options.check = coder->stream_flags.check; + + // Even if we return LZMA_*_CHECK below, we want + // to continue from Block Header decoding. + coder->sequence = SEQ_BLOCK_HEADER; + + // Detect if there's no integrity check or if it is + // unsupported if those were requested by the application. + if (coder->tell_no_check && coder->stream_flags.check + == LZMA_CHECK_NONE) + return LZMA_NO_CHECK; + + if (coder->tell_unsupported_check + && !lzma_check_is_supported( + coder->stream_flags.check)) + return LZMA_UNSUPPORTED_CHECK; + + if (coder->tell_any_check) + return LZMA_GET_CHECK; + + FALLTHROUGH; + } + + case SEQ_BLOCK_HEADER: { + const size_t in_old = *in_pos; + const lzma_ret ret = decode_block_header(coder, allocator, + in, in_pos, in_size); + coder->progress_in += *in_pos - in_old; + + if (ret == LZMA_OK) { + // We didn't decode the whole Block Header yet. + // + // Read output from the queue before returning. This + // is important because it is possible that the + // application doesn't have any new input available + // immediately. If we didn't try to copy output from + // the output queue here, lzma_code() could end up + // returning LZMA_BUF_ERROR even though queued output + // is available. + // + // If the lzma_code() call provided at least one input + // byte, only copy as much data from the output queue + // as is available immediately. This way the + // application will be able to provide more input + // without a delay. + // + // On the other hand, if lzma_code() was called with + // an empty input buffer(*), treat it specially: try + // to fill the output buffer even if it requires + // waiting for the worker threads to provide output + // (timeout, if specified, can still cause us to + // return). + // + // - This way the application will be able to get all + // data that can be decoded from the input provided + // so far. + // + // - We avoid both premature LZMA_BUF_ERROR and + // busy-waiting where the application repeatedly + // calls lzma_code() which immediately returns + // LZMA_OK without providing new data. + // + // - If the queue becomes empty, we won't wait + // anything and will return LZMA_OK immediately + // (coder->timeout is completely ignored). + // + // (*) See the comment at the beginning of this + // function how waiting_allowed is determined + // and why there is an exception to the rule + // of "called with an empty input buffer". + assert(*in_pos == in_size); + + // If LZMA_FINISH was used we know that we won't get + // more input, so the file must be truncated if we + // get here. If worker threads don't detect any + // errors, eventually there will be no more output + // while we keep returning LZMA_OK which gets + // converted to LZMA_BUF_ERROR in lzma_code(). + // + // If fail-fast is enabled then we will return + // immediately using LZMA_DATA_ERROR instead of + // LZMA_OK or LZMA_BUF_ERROR. Rationale for the + // error code: + // + // - Worker threads may have a large amount of + // not-yet-decoded input data and we don't + // know for sure if all data is valid. Bad + // data there would result in LZMA_DATA_ERROR + // when fail-fast isn't used. + // + // - Immediate LZMA_BUF_ERROR would be a bit weird + // considering the older liblzma code. lzma_code() + // even has an assertion to prevent coders from + // returning LZMA_BUF_ERROR directly. + // + // The downside of this is that with fail-fast apps + // cannot always distinguish between corrupt and + // truncated files. + if (action == LZMA_FINISH && coder->fail_fast) { + // We won't produce any more output. Stop + // the unfinished worker threads so they + // won't waste CPU time. + threads_stop(coder); + return LZMA_DATA_ERROR; + } + + // read_output_and_wait() will call threads_stop() + // if needed so with that we can use return_if_error. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, waiting_allowed, + &wait_abs, &has_blocked)); + + if (coder->pending_error != LZMA_OK) { + coder->sequence = SEQ_ERROR; + break; + } + + return LZMA_OK; + } + + if (ret == LZMA_INDEX_DETECTED) { + coder->sequence = SEQ_INDEX_WAIT_OUTPUT; + break; + } + + // See if an error occurred. + if (ret != LZMA_STREAM_END) { + // NOTE: Here and in all other places where + // pending_error is set, it may overwrite the value + // (LZMA_PROG_ERROR) set by read_output_and_wait(). + // That function might overwrite value set here too. + // These are fine because when read_output_and_wait() + // sets pending_error, it actually works as a flag + // variable only ("some error has occurred") and the + // actual value of pending_error is not used in + // SEQ_ERROR. In such cases SEQ_ERROR will eventually + // get the correct error code from the return value of + // a later read_output_and_wait() call. + coder->pending_error = ret; + coder->sequence = SEQ_ERROR; + break; + } + + // Calculate the memory usage of the filters / Block decoder. + coder->mem_next_filters = lzma_raw_decoder_memusage( + coder->filters); + + if (coder->mem_next_filters == UINT64_MAX) { + // One or more unknown Filter IDs. + coder->pending_error = LZMA_OPTIONS_ERROR; + coder->sequence = SEQ_ERROR; + break; + } + + coder->sequence = SEQ_BLOCK_INIT; + FALLTHROUGH; + } + + case SEQ_BLOCK_INIT: { + // Check if decoding is possible at all with the current + // memlimit_stop which we must never exceed. + // + // This needs to be the first thing in SEQ_BLOCK_INIT + // to make it possible to restart decoding after increasing + // memlimit_stop with lzma_memlimit_set(). + if (coder->mem_next_filters > coder->memlimit_stop) { + // Flush pending output before returning + // LZMA_MEMLIMIT_ERROR. If the application doesn't + // want to increase the limit, at least it will get + // all the output possible so far. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + + return LZMA_MEMLIMIT_ERROR; + } + + // Check if the size information is available in Block Header. + // If it is, check if the sizes are small enough that we don't + // need to worry *too* much about integer overflows later in + // the code. If these conditions are not met, we must use the + // single-threaded direct mode. + if (is_direct_mode_needed(coder->block_options.compressed_size) + || is_direct_mode_needed( + coder->block_options.uncompressed_size)) { + coder->sequence = SEQ_BLOCK_DIRECT_INIT; + break; + } + + // Calculate the amount of memory needed for the input and + // output buffers in threaded mode. + // + // These cannot overflow because we already checked that + // the sizes are small enough using is_direct_mode_needed(). + coder->mem_next_in = comp_blk_size(coder); + const uint64_t mem_buffers = coder->mem_next_in + + lzma_outq_outbuf_memusage( + coder->block_options.uncompressed_size); + + // Add the amount needed by the filters. + // Avoid integer overflows. + if (UINT64_MAX - mem_buffers < coder->mem_next_filters) { + // Use direct mode if the memusage would overflow. + // This is a theoretical case that shouldn't happen + // in practice unless the input file is weird (broken + // or malicious). + coder->sequence = SEQ_BLOCK_DIRECT_INIT; + break; + } + + // Amount of memory needed to decode this Block in + // threaded mode: + coder->mem_next_block = coder->mem_next_filters + mem_buffers; + + // If this alone would exceed memlimit_threading, then we must + // use the single-threaded direct mode. + if (coder->mem_next_block > coder->memlimit_threading) { + coder->sequence = SEQ_BLOCK_DIRECT_INIT; + break; + } + + // Use the threaded mode. Free the direct mode decoder in + // case it has been initialized. + lzma_next_end(&coder->block_decoder, allocator); + coder->mem_direct_mode = 0; + + // Since we already know what the sizes are supposed to be, + // we can already add them to the Index hash. The Block + // decoder will verify the values while decoding. + const lzma_ret ret = lzma_index_hash_append(coder->index_hash, + lzma_block_unpadded_size( + &coder->block_options), + coder->block_options.uncompressed_size); + if (ret != LZMA_OK) { + coder->pending_error = ret; + coder->sequence = SEQ_ERROR; + break; + } + + coder->sequence = SEQ_BLOCK_THR_INIT; + FALLTHROUGH; + } + + case SEQ_BLOCK_THR_INIT: { + // We need to wait for a multiple conditions to become true + // until we can initialize the Block decoder and let a worker + // thread decode it: + // + // - Wait for the memory usage of the active threads to drop + // so that starting the decoding of this Block won't make + // us go over memlimit_threading. + // + // - Wait for at least one free output queue slot. + // + // - Wait for a free worker thread. + // + // While we wait, we must copy decompressed data to the out + // buffer and catch possible decoder errors. + // + // read_output_and_wait() does all the above. + bool block_can_start = false; + + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + &block_can_start, true, + &wait_abs, &has_blocked)); + + if (coder->pending_error != LZMA_OK) { + coder->sequence = SEQ_ERROR; + break; + } + + if (!block_can_start) { + // It's not a timeout because return_if_error handles + // it already. Output queue cannot be empty either + // because in that case block_can_start would have + // been true. Thus the output buffer must be full and + // the queue isn't empty. + assert(*out_pos == out_size); + assert(!lzma_outq_is_empty(&coder->outq)); + return LZMA_OK; + } + + // We know that we can start decoding this Block without + // exceeding memlimit_threading. However, to stay below + // memlimit_threading may require freeing some of the + // cached memory. + // + // Get a local copy of variables that require locking the + // mutex. It is fine if the worker threads modify the real + // values after we read these as those changes can only be + // towards more favorable conditions (less memory in use, + // more in cache). + // + // These are initialized to silence warnings. + uint64_t mem_in_use = 0; + uint64_t mem_cached = 0; + struct worker_thread *thr = NULL; + + mythread_sync(coder->mutex) { + mem_in_use = coder->mem_in_use; + mem_cached = coder->mem_cached; + thr = coder->threads_free; + } + + // The maximum amount of memory that can be held by other + // threads and cached buffers while allowing us to start + // decoding the next Block. + const uint64_t mem_max = coder->memlimit_threading + - coder->mem_next_block; + + // If the existing allocations are so large that starting + // to decode this Block might exceed memlimit_threads, + // try to free memory from the output queue cache first. + // + // NOTE: This math assumes the worst case. It's possible + // that the limit wouldn't be exceeded if the existing cached + // allocations are reused. + if (mem_in_use + mem_cached + coder->outq.mem_allocated + > mem_max) { + // Clear the outq cache except leave one buffer in + // the cache if its size is correct. That way we + // don't free and almost immediately reallocate + // an identical buffer. + lzma_outq_clear_cache2(&coder->outq, allocator, + coder->block_options.uncompressed_size); + } + + // If there is at least one worker_thread in the cache and + // the existing allocations are so large that starting to + // decode this Block might exceed memlimit_threads, free + // memory by freeing cached Block decoders. + // + // NOTE: The comparison is different here than above. + // Here we don't care about cached buffers in outq anymore + // and only look at memory actually in use. This is because + // if there is something in outq cache, it's a single buffer + // that can be used as is. We ensured this in the above + // if-block. + uint64_t mem_freed = 0; + if (thr != NULL && mem_in_use + mem_cached + + coder->outq.mem_in_use > mem_max) { + // Don't free the first Block decoder if its memory + // usage isn't greater than what this Block will need. + // Typically the same filter chain is used for all + // Blocks so this way the allocations can be reused + // when get_thread() picks the first worker_thread + // from the cache. + if (thr->mem_filters <= coder->mem_next_filters) + thr = thr->next; + + while (thr != NULL) { + lzma_next_end(&thr->block_decoder, allocator); + mem_freed += thr->mem_filters; + thr->mem_filters = 0; + thr = thr->next; + } + } + + // Update the memory usage counters. Note that coder->mem_* + // may have changed since we read them so we must subtract + // or add the changes. + mythread_sync(coder->mutex) { + coder->mem_cached -= mem_freed; + + // Memory needed for the filters and the input buffer. + // The output queue takes care of its own counter so + // we don't touch it here. + // + // NOTE: After this, coder->mem_in_use + + // coder->mem_cached might count the same thing twice. + // If so, this will get corrected in get_thread() when + // a worker_thread is picked from coder->free_threads + // and its memory usage is subtracted from mem_cached. + coder->mem_in_use += coder->mem_next_in + + coder->mem_next_filters; + } + + // Allocate memory for the output buffer in the output queue. + lzma_ret ret = lzma_outq_prealloc_buf( + &coder->outq, allocator, + coder->block_options.uncompressed_size); + if (ret != LZMA_OK) { + threads_stop(coder); + return ret; + } + + // Set up coder->thr. + ret = get_thread(coder, allocator); + if (ret != LZMA_OK) { + threads_stop(coder); + return ret; + } + + // The new Block decoder memory usage is already counted in + // coder->mem_in_use. Store it in the thread too. + coder->thr->mem_filters = coder->mem_next_filters; + + // Initialize the Block decoder. + coder->thr->block_options = coder->block_options; + ret = lzma_block_decoder_init( + &coder->thr->block_decoder, allocator, + &coder->thr->block_options); + + // Free the allocated filter options since they are needed + // only to initialize the Block decoder. + lzma_filters_free(coder->filters, allocator); + coder->thr->block_options.filters = NULL; + + // Check if memory usage calculation and Block encoder + // initialization succeeded. + if (ret != LZMA_OK) { + coder->pending_error = ret; + coder->sequence = SEQ_ERROR; + break; + } + + // Allocate the input buffer. + coder->thr->in_size = coder->mem_next_in; + coder->thr->in = lzma_alloc(coder->thr->in_size, allocator); + if (coder->thr->in == NULL) { + threads_stop(coder); + return LZMA_MEM_ERROR; + } + + // Get the preallocated output buffer. + coder->thr->outbuf = lzma_outq_get_buf( + &coder->outq, coder->thr); + + // Start the decoder. + mythread_sync(coder->thr->mutex) { + assert(coder->thr->state == THR_IDLE); + coder->thr->state = THR_RUN; + mythread_cond_signal(&coder->thr->cond); + } + + // Enable output from the thread that holds the oldest output + // buffer in the output queue (if such a thread exists). + mythread_sync(coder->mutex) { + lzma_outq_enable_partial_output(&coder->outq, + &worker_enable_partial_update); + } + + coder->sequence = SEQ_BLOCK_THR_RUN; + FALLTHROUGH; + } + + case SEQ_BLOCK_THR_RUN: { + if (action == LZMA_FINISH && coder->fail_fast) { + // We know that we won't get more input and that + // the caller wants fail-fast behavior. If we see + // that we don't have enough input to finish this + // Block, return LZMA_DATA_ERROR immediately. + // See SEQ_BLOCK_HEADER for the error code rationale. + const size_t in_avail = in_size - *in_pos; + const size_t in_needed = coder->thr->in_size + - coder->thr->in_filled; + if (in_avail < in_needed) { + threads_stop(coder); + return LZMA_DATA_ERROR; + } + } + + // Copy input to the worker thread. + size_t cur_in_filled = coder->thr->in_filled; + lzma_bufcpy(in, in_pos, in_size, coder->thr->in, + &cur_in_filled, coder->thr->in_size); + + // Tell the thread how much we copied. + mythread_sync(coder->thr->mutex) { + coder->thr->in_filled = cur_in_filled; + + // NOTE: Most of the time we are copying input faster + // than the thread can decode so most of the time + // calling mythread_cond_signal() is useless but + // we cannot make it conditional because thr->in_pos + // is updated without a mutex. And the overhead should + // be very much negligible anyway. + mythread_cond_signal(&coder->thr->cond); + } + + // Read output from the output queue. Just like in + // SEQ_BLOCK_HEADER, we wait to fill the output buffer + // only if waiting_allowed was set to true in the beginning + // of this function (see the comment there) and there is + // no input available. In SEQ_BLOCK_HEADER, there is never + // input available when read_output_and_wait() is called, + // but here there can be when LZMA_FINISH is used, thus we + // need to check if *in_pos == in_size. Otherwise we would + // wait here instead of using the available input to start + // a new thread. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, + waiting_allowed && *in_pos == in_size, + &wait_abs, &has_blocked)); + + if (coder->pending_error != LZMA_OK) { + coder->sequence = SEQ_ERROR; + break; + } + + // Return if the input didn't contain the whole Block. + // + // NOTE: When we updated coder->thr->in_filled a few lines + // above, the worker thread might by now have finished its + // work and returned itself back to the stack of free threads. + if (coder->thr->in_filled < coder->thr->in_size) { + assert(*in_pos == in_size); + return LZMA_OK; + } + + // The whole Block has been copied to the thread-specific + // buffer. Continue from the next Block Header or Index. + coder->thr = NULL; + coder->sequence = SEQ_BLOCK_HEADER; + break; + } + + case SEQ_BLOCK_DIRECT_INIT: { + // Wait for the threads to finish and that all decoded data + // has been copied to the output. That is, wait until the + // output queue becomes empty. + // + // NOTE: No need to check for coder->pending_error as + // we aren't consuming any input until the queue is empty + // and if there is a pending error, read_output_and_wait() + // will eventually return it before the queue is empty. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + + // Free the cached output buffers. + lzma_outq_clear_cache(&coder->outq, allocator); + + // Get rid of the worker threads, including the coder->threads + // array. + threads_end(coder, allocator); + + // Initialize the Block decoder. + const lzma_ret ret = lzma_block_decoder_init( + &coder->block_decoder, allocator, + &coder->block_options); + + // Free the allocated filter options since they are needed + // only to initialize the Block decoder. + lzma_filters_free(coder->filters, allocator); + coder->block_options.filters = NULL; + + // Check if Block decoder initialization succeeded. + if (ret != LZMA_OK) + return ret; + + // Make the memory usage visible to _memconfig(). + coder->mem_direct_mode = coder->mem_next_filters; + + coder->sequence = SEQ_BLOCK_DIRECT_RUN; + FALLTHROUGH; + } + + case SEQ_BLOCK_DIRECT_RUN: { + const size_t in_old = *in_pos; + const size_t out_old = *out_pos; + const lzma_ret ret = coder->block_decoder.code( + coder->block_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + coder->progress_in += *in_pos - in_old; + coder->progress_out += *out_pos - out_old; + + if (ret != LZMA_STREAM_END) + return ret; + + // Block decoded successfully. Add the new size pair to + // the Index hash. + return_if_error(lzma_index_hash_append(coder->index_hash, + lzma_block_unpadded_size( + &coder->block_options), + coder->block_options.uncompressed_size)); + + coder->sequence = SEQ_BLOCK_HEADER; + break; + } + + case SEQ_INDEX_WAIT_OUTPUT: + // Flush the output from all worker threads so that we can + // decode the Index without thinking about threading. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + + coder->sequence = SEQ_INDEX_DECODE; + FALLTHROUGH; + + case SEQ_INDEX_DECODE: { + // If we don't have any input, don't call + // lzma_index_hash_decode() since it would return + // LZMA_BUF_ERROR, which we must not do here. + if (*in_pos >= in_size) + return LZMA_OK; + + // Decode the Index and compare it to the hash calculated + // from the sizes of the Blocks (if any). + const size_t in_old = *in_pos; + const lzma_ret ret = lzma_index_hash_decode(coder->index_hash, + in, in_pos, in_size); + coder->progress_in += *in_pos - in_old; + if (ret != LZMA_STREAM_END) + return ret; + + coder->sequence = SEQ_STREAM_FOOTER; + FALLTHROUGH; + } + + case SEQ_STREAM_FOOTER: { + // Copy the Stream Footer to the internal buffer. + const size_t in_old = *in_pos; + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + LZMA_STREAM_HEADER_SIZE); + coder->progress_in += *in_pos - in_old; + + // Return if we didn't get the whole Stream Footer yet. + if (coder->pos < LZMA_STREAM_HEADER_SIZE) + return LZMA_OK; + + coder->pos = 0; + + // Decode the Stream Footer. The decoder gives + // LZMA_FORMAT_ERROR if the magic bytes don't match, + // so convert that return code to LZMA_DATA_ERROR. + lzma_stream_flags footer_flags; + const lzma_ret ret = lzma_stream_footer_decode( + &footer_flags, coder->buffer); + if (ret != LZMA_OK) + return ret == LZMA_FORMAT_ERROR + ? LZMA_DATA_ERROR : ret; + + // Check that Index Size stored in the Stream Footer matches + // the real size of the Index field. + if (lzma_index_hash_size(coder->index_hash) + != footer_flags.backward_size) + return LZMA_DATA_ERROR; + + // Compare that the Stream Flags fields are identical in + // both Stream Header and Stream Footer. + return_if_error(lzma_stream_flags_compare( + &coder->stream_flags, &footer_flags)); + + if (!coder->concatenated) + return LZMA_STREAM_END; + + coder->sequence = SEQ_STREAM_PADDING; + FALLTHROUGH; + } + + case SEQ_STREAM_PADDING: + assert(coder->concatenated); + + // Skip over possible Stream Padding. + while (true) { + if (*in_pos >= in_size) { + // Unless LZMA_FINISH was used, we cannot + // know if there's more input coming later. + if (action != LZMA_FINISH) + return LZMA_OK; + + // Stream Padding must be a multiple of + // four bytes. + return coder->pos == 0 + ? LZMA_STREAM_END + : LZMA_DATA_ERROR; + } + + // If the byte is not zero, it probably indicates + // beginning of a new Stream (or the file is corrupt). + if (in[*in_pos] != 0x00) + break; + + ++*in_pos; + ++coder->progress_in; + coder->pos = (coder->pos + 1) & 3; + } + + // Stream Padding must be a multiple of four bytes (empty + // Stream Padding is OK). + if (coder->pos != 0) { + ++*in_pos; + ++coder->progress_in; + return LZMA_DATA_ERROR; + } + + // Prepare to decode the next Stream. + return_if_error(stream_decoder_reset(coder, allocator)); + break; + + case SEQ_ERROR: + if (!coder->fail_fast) { + // Let the application get all data before the point + // where the error was detected. This matches the + // behavior of single-threaded use. + // + // FIXME? Some errors (LZMA_MEM_ERROR) don't get here, + // they are returned immediately. Thus in rare cases + // the output will be less than in the single-threaded + // mode. Maybe this doesn't matter much in practice. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + + // We get here only if the error happened in the main + // thread, for example, unsupported Block Header. + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + } + + // We only get here if no errors were detected by the worker + // threads. Errors from worker threads would have already been + // returned by the call to read_output_and_wait() above. + return coder->pending_error; + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + // Never reached +} + + +static void +stream_decoder_mt_end(void *coder_ptr, const lzma_allocator *allocator) +{ + struct lzma_stream_coder *coder = coder_ptr; + + threads_end(coder, allocator); + lzma_outq_end(&coder->outq, allocator); + + lzma_next_end(&coder->block_decoder, allocator); + lzma_filters_free(coder->filters, allocator); + lzma_index_hash_end(coder->index_hash, allocator); + + lzma_free(coder, allocator); + return; +} + + +static lzma_check +stream_decoder_mt_get_check(const void *coder_ptr) +{ + const struct lzma_stream_coder *coder = coder_ptr; + return coder->stream_flags.check; +} + + +static lzma_ret +stream_decoder_mt_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + // NOTE: This function gets/sets memlimit_stop. For now, + // memlimit_threading cannot be modified after initialization. + // + // *memusage will include cached memory too. Excluding cached memory + // would be misleading and it wouldn't help the applications to + // know how much memory is actually needed to decompress the file + // because the higher the number of threads and the memlimits are + // the more memory the decoder may use. + // + // Setting a new limit includes the cached memory too and too low + // limits will be rejected. Alternative could be to free the cached + // memory immediately if that helps to bring the limit down but + // the current way is the simplest. It's unlikely that limit needs + // to be lowered in the middle of a file anyway; the typical reason + // to want a new limit is to increase after LZMA_MEMLIMIT_ERROR + // and even such use isn't common. + struct lzma_stream_coder *coder = coder_ptr; + + mythread_sync(coder->mutex) { + *memusage = coder->mem_direct_mode + + coder->mem_in_use + + coder->mem_cached + + coder->outq.mem_allocated; + } + + // If no filter chains are allocated, *memusage may be zero. + // Always return at least LZMA_MEMUSAGE_BASE. + if (*memusage < LZMA_MEMUSAGE_BASE) + *memusage = LZMA_MEMUSAGE_BASE; + + *old_memlimit = coder->memlimit_stop; + + if (new_memlimit != 0) { + if (new_memlimit < *memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit_stop = new_memlimit; + } + + return LZMA_OK; +} + + +static void +stream_decoder_mt_get_progress(void *coder_ptr, + uint64_t *progress_in, uint64_t *progress_out) +{ + struct lzma_stream_coder *coder = coder_ptr; + + // Lock coder->mutex to prevent finishing threads from moving their + // progress info from the worker_thread structure to lzma_stream_coder. + mythread_sync(coder->mutex) { + *progress_in = coder->progress_in; + *progress_out = coder->progress_out; + + for (size_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + *progress_in += coder->threads[i].progress_in; + *progress_out += coder->threads[i] + .progress_out; + } + } + } + + return; +} + + +static lzma_ret +stream_decoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_mt *options) +{ + struct lzma_stream_coder *coder; + + if (options->threads == 0 || options->threads > LZMA_THREADS_MAX) + return LZMA_OPTIONS_ERROR; + + if (options->flags & ~LZMA_SUPPORTED_FLAGS) + return LZMA_OPTIONS_ERROR; + + lzma_next_coder_init(&stream_decoder_mt_init, next, allocator); + + coder = next->coder; + if (!coder) { + coder = lzma_alloc(sizeof(struct lzma_stream_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + + if (mythread_mutex_init(&coder->mutex)) { + lzma_free(coder, allocator); + return LZMA_MEM_ERROR; + } + + if (mythread_cond_init(&coder->cond)) { + mythread_mutex_destroy(&coder->mutex); + lzma_free(coder, allocator); + return LZMA_MEM_ERROR; + } + + next->code = &stream_decode_mt; + next->end = &stream_decoder_mt_end; + next->get_check = &stream_decoder_mt_get_check; + next->memconfig = &stream_decoder_mt_memconfig; + next->get_progress = &stream_decoder_mt_get_progress; + + coder->filters[0].id = LZMA_VLI_UNKNOWN; + memzero(&coder->outq, sizeof(coder->outq)); + + coder->block_decoder = LZMA_NEXT_CODER_INIT; + coder->mem_direct_mode = 0; + + coder->index_hash = NULL; + coder->threads = NULL; + coder->threads_free = NULL; + coder->threads_initialized = 0; + } + + // Cleanup old filter chain if one remains after unfinished decoding + // of a previous Stream. + lzma_filters_free(coder->filters, allocator); + + // By allocating threads from scratch we can start memory-usage + // accounting from scratch, too. Changes in filter and block sizes may + // affect number of threads. + // + // Reusing threads doesn't seem worth it. Unlike the single-threaded + // decoder, with some types of input file combinations reusing + // could leave quite a lot of memory allocated but unused (first + // file could allocate a lot, the next files could use fewer + // threads and some of the allocations from the first file would not + // get freed unless memlimit_threading forces us to clear caches). + // + // NOTE: The direct mode decoder isn't freed here if one exists. + // It will be reused or freed as needed in the main loop. + threads_end(coder, allocator); + + // All memusage counters start at 0 (including mem_direct_mode). + // The little extra that is needed for the structs in this file + // get accounted well enough by the filter chain memory usage + // which adds LZMA_MEMUSAGE_BASE for each chain. However, + // stream_decoder_mt_memconfig() has to handle this specially so that + // it will never return less than LZMA_MEMUSAGE_BASE as memory usage. + coder->mem_in_use = 0; + coder->mem_cached = 0; + coder->mem_next_block = 0; + + coder->progress_in = 0; + coder->progress_out = 0; + + coder->sequence = SEQ_STREAM_HEADER; + coder->thread_error = LZMA_OK; + coder->pending_error = LZMA_OK; + coder->thr = NULL; + + coder->timeout = options->timeout; + + coder->memlimit_threading = my_max(1, options->memlimit_threading); + coder->memlimit_stop = my_max(1, options->memlimit_stop); + if (coder->memlimit_threading > coder->memlimit_stop) + coder->memlimit_threading = coder->memlimit_stop; + + coder->tell_no_check = (options->flags & LZMA_TELL_NO_CHECK) != 0; + coder->tell_unsupported_check + = (options->flags & LZMA_TELL_UNSUPPORTED_CHECK) != 0; + coder->tell_any_check = (options->flags & LZMA_TELL_ANY_CHECK) != 0; + coder->ignore_check = (options->flags & LZMA_IGNORE_CHECK) != 0; + coder->concatenated = (options->flags & LZMA_CONCATENATED) != 0; + coder->fail_fast = (options->flags & LZMA_FAIL_FAST) != 0; + + coder->first_stream = true; + coder->out_was_filled = false; + coder->pos = 0; + + coder->threads_max = options->threads; + + return_if_error(lzma_outq_init(&coder->outq, allocator, + coder->threads_max)); + + return stream_decoder_reset(coder, allocator); +} + + +extern LZMA_API(lzma_ret) +lzma_stream_decoder_mt(lzma_stream *strm, const lzma_mt *options) +{ + lzma_next_strm_init(stream_decoder_mt_init, strm, options); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_encoder.c b/src/liblzma/common/stream_encoder.c new file mode 100644 index 0000000000..e7e5b3fce7 --- /dev/null +++ b/src/liblzma/common/stream_encoder.c @@ -0,0 +1,354 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_encoder.c +/// \brief Encodes .xz Streams +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "block_encoder.h" +#include "index_encoder.h" + + +typedef struct { + enum { + SEQ_STREAM_HEADER, + SEQ_BLOCK_INIT, + SEQ_BLOCK_HEADER, + SEQ_BLOCK_ENCODE, + SEQ_INDEX_ENCODE, + SEQ_STREAM_FOOTER, + } sequence; + + /// True if Block encoder has been initialized by + /// stream_encoder_init() or stream_encoder_update() + /// and thus doesn't need to be initialized in stream_encode(). + bool block_encoder_is_initialized; + + /// Block + lzma_next_coder block_encoder; + + /// Options for the Block encoder + lzma_block block_options; + + /// The filter chain currently in use + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + /// Index encoder. This is separate from Block encoder, because this + /// doesn't take much memory, and when encoding multiple Streams + /// with the same encoding options we avoid reallocating memory. + lzma_next_coder index_encoder; + + /// Index to hold sizes of the Blocks + lzma_index *index; + + /// Read position in buffer[] + size_t buffer_pos; + + /// Total number of bytes in buffer[] + size_t buffer_size; + + /// Buffer to hold Stream Header, Block Header, and Stream Footer. + /// Block Header has biggest maximum size. + uint8_t buffer[LZMA_BLOCK_HEADER_SIZE_MAX]; +} lzma_stream_coder; + + +static lzma_ret +block_encoder_init(lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + // Prepare the Block options. Even though Block encoder doesn't need + // compressed_size, uncompressed_size, and header_size to be + // initialized, it is a good idea to do it here, because this way + // we catch if someone gave us Filter ID that cannot be used in + // Blocks/Streams. + coder->block_options.compressed_size = LZMA_VLI_UNKNOWN; + coder->block_options.uncompressed_size = LZMA_VLI_UNKNOWN; + + return_if_error(lzma_block_header_size(&coder->block_options)); + + // Initialize the actual Block encoder. + return lzma_block_encoder_init(&coder->block_encoder, allocator, + &coder->block_options); +} + + +static lzma_ret +stream_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_stream_coder *coder = coder_ptr; + + // Main loop + while (*out_pos < out_size) + switch (coder->sequence) { + case SEQ_STREAM_HEADER: + case SEQ_BLOCK_HEADER: + case SEQ_STREAM_FOOTER: + lzma_bufcpy(coder->buffer, &coder->buffer_pos, + coder->buffer_size, out, out_pos, out_size); + if (coder->buffer_pos < coder->buffer_size) + return LZMA_OK; + + if (coder->sequence == SEQ_STREAM_FOOTER) + return LZMA_STREAM_END; + + coder->buffer_pos = 0; + ++coder->sequence; + break; + + case SEQ_BLOCK_INIT: { + if (*in_pos == in_size) { + // If we are requested to flush or finish the current + // Block, return LZMA_STREAM_END immediately since + // there's nothing to do. + if (action != LZMA_FINISH) + return action == LZMA_RUN + ? LZMA_OK : LZMA_STREAM_END; + + // The application had used LZMA_FULL_FLUSH to finish + // the previous Block, but now wants to finish without + // encoding new data, or it is simply creating an + // empty Stream with no Blocks. + // + // Initialize the Index encoder, and continue to + // actually encoding the Index. + return_if_error(lzma_index_encoder_init( + &coder->index_encoder, allocator, + coder->index)); + coder->sequence = SEQ_INDEX_ENCODE; + break; + } + + // Initialize the Block encoder unless it was already + // initialized by stream_encoder_init() or + // stream_encoder_update(). + if (!coder->block_encoder_is_initialized) + return_if_error(block_encoder_init(coder, allocator)); + + // Make it false so that we don't skip the initialization + // with the next Block. + coder->block_encoder_is_initialized = false; + + // Encode the Block Header. This shouldn't fail since we have + // already initialized the Block encoder. + if (lzma_block_header_encode(&coder->block_options, + coder->buffer) != LZMA_OK) + return LZMA_PROG_ERROR; + + coder->buffer_size = coder->block_options.header_size; + coder->sequence = SEQ_BLOCK_HEADER; + break; + } + + case SEQ_BLOCK_ENCODE: { + static const lzma_action convert[LZMA_ACTION_MAX + 1] = { + LZMA_RUN, + LZMA_SYNC_FLUSH, + LZMA_FINISH, + LZMA_FINISH, + LZMA_FINISH, + }; + + const lzma_ret ret = coder->block_encoder.code( + coder->block_encoder.coder, allocator, + in, in_pos, in_size, + out, out_pos, out_size, convert[action]); + if (ret != LZMA_STREAM_END || action == LZMA_SYNC_FLUSH) + return ret; + + // Add a new Index Record. + const lzma_vli unpadded_size = lzma_block_unpadded_size( + &coder->block_options); + assert(unpadded_size != 0); + return_if_error(lzma_index_append(coder->index, allocator, + unpadded_size, + coder->block_options.uncompressed_size)); + + coder->sequence = SEQ_BLOCK_INIT; + break; + } + + case SEQ_INDEX_ENCODE: { + // Call the Index encoder. It doesn't take any input, so + // those pointers can be NULL. + const lzma_ret ret = coder->index_encoder.code( + coder->index_encoder.coder, allocator, + NULL, NULL, 0, + out, out_pos, out_size, LZMA_RUN); + if (ret != LZMA_STREAM_END) + return ret; + + // Encode the Stream Footer into coder->buffer. + const lzma_stream_flags stream_flags = { + .version = 0, + .backward_size = lzma_index_size(coder->index), + .check = coder->block_options.check, + }; + + if (lzma_stream_footer_encode(&stream_flags, coder->buffer) + != LZMA_OK) + return LZMA_PROG_ERROR; + + coder->buffer_size = LZMA_STREAM_HEADER_SIZE; + coder->sequence = SEQ_STREAM_FOOTER; + break; + } + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + return LZMA_OK; +} + + +static void +stream_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_stream_coder *coder = coder_ptr; + + lzma_next_end(&coder->block_encoder, allocator); + lzma_next_end(&coder->index_encoder, allocator); + lzma_index_end(coder->index, allocator); + + lzma_filters_free(coder->filters, allocator); + + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +stream_encoder_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters, + const lzma_filter *reversed_filters) +{ + lzma_stream_coder *coder = coder_ptr; + lzma_ret ret; + + // Make a copy to a temporary buffer first. This way it is easier + // to keep the encoder state unchanged if an error occurs with + // lzma_filters_copy(). + lzma_filter temp[LZMA_FILTERS_MAX + 1]; + return_if_error(lzma_filters_copy(filters, temp, allocator)); + + if (coder->sequence <= SEQ_BLOCK_INIT) { + // There is no incomplete Block waiting to be finished, + // thus we can change the whole filter chain. Start by + // trying to initialize the Block encoder with the new + // chain. This way we detect if the chain is valid. + coder->block_encoder_is_initialized = false; + coder->block_options.filters = temp; + ret = block_encoder_init(coder, allocator); + coder->block_options.filters = coder->filters; + if (ret != LZMA_OK) + goto error; + + coder->block_encoder_is_initialized = true; + + } else if (coder->sequence <= SEQ_BLOCK_ENCODE) { + // We are in the middle of a Block. Try to update only + // the filter-specific options. + ret = coder->block_encoder.update( + coder->block_encoder.coder, allocator, + filters, reversed_filters); + if (ret != LZMA_OK) + goto error; + } else { + // Trying to update the filter chain when we are already + // encoding Index or Stream Footer. + ret = LZMA_PROG_ERROR; + goto error; + } + + // Free the options of the old chain. + lzma_filters_free(coder->filters, allocator); + + // Copy the new filter chain in place. + memcpy(coder->filters, temp, sizeof(temp)); + + return LZMA_OK; + +error: + lzma_filters_free(temp, allocator); + return ret; +} + + +static lzma_ret +stream_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter *filters, lzma_check check) +{ + lzma_next_coder_init(&stream_encoder_init, next, allocator); + + if (filters == NULL) + return LZMA_PROG_ERROR; + + lzma_stream_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_stream_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &stream_encode; + next->end = &stream_encoder_end; + next->update = &stream_encoder_update; + + coder->filters[0].id = LZMA_VLI_UNKNOWN; + coder->block_encoder = LZMA_NEXT_CODER_INIT; + coder->index_encoder = LZMA_NEXT_CODER_INIT; + coder->index = NULL; + } + + // Basic initializations + coder->sequence = SEQ_STREAM_HEADER; + coder->block_options.version = 0; + coder->block_options.check = check; + + // Initialize the Index + lzma_index_end(coder->index, allocator); + coder->index = lzma_index_init(allocator); + if (coder->index == NULL) + return LZMA_MEM_ERROR; + + // Encode the Stream Header + lzma_stream_flags stream_flags = { + .version = 0, + .check = check, + }; + return_if_error(lzma_stream_header_encode( + &stream_flags, coder->buffer)); + + coder->buffer_pos = 0; + coder->buffer_size = LZMA_STREAM_HEADER_SIZE; + + // Initialize the Block encoder. This way we detect unsupported + // filter chains when initializing the Stream encoder instead of + // giving an error after Stream Header has already been written out. + return stream_encoder_update(coder, allocator, filters, NULL); +} + + +extern LZMA_API(lzma_ret) +lzma_stream_encoder(lzma_stream *strm, + const lzma_filter *filters, lzma_check check) +{ + lzma_next_strm_init(stream_encoder_init, strm, filters, check); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_SYNC_FLUSH] = true; + strm->internal->supported_actions[LZMA_FULL_FLUSH] = true; + strm->internal->supported_actions[LZMA_FULL_BARRIER] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_encoder_mt.c b/src/liblzma/common/stream_encoder_mt.c new file mode 100644 index 0000000000..fd0eb98df6 --- /dev/null +++ b/src/liblzma/common/stream_encoder_mt.c @@ -0,0 +1,1278 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_encoder_mt.c +/// \brief Multithreaded .xz Stream encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_encoder.h" +#include "easy_preset.h" +#include "block_encoder.h" +#include "block_buffer_encoder.h" +#include "index_encoder.h" +#include "outqueue.h" + + +/// Maximum supported block size. This makes it simpler to prevent integer +/// overflows if we are given unusually large block size. +#define BLOCK_SIZE_MAX (UINT64_MAX / LZMA_THREADS_MAX) + + +typedef enum { + /// Waiting for work. + THR_IDLE, + + /// Encoding is in progress. + THR_RUN, + + /// Encoding is in progress but no more input data will + /// be read. + THR_FINISH, + + /// The main thread wants the thread to stop whatever it was doing + /// but not exit. + THR_STOP, + + /// The main thread wants the thread to exit. We could use + /// cancellation but since there's stopped anyway, this is lazier. + THR_EXIT, + +} worker_state; + +typedef struct lzma_stream_coder_s lzma_stream_coder; + +typedef struct worker_thread_s worker_thread; +struct worker_thread_s { + worker_state state; + + /// Input buffer of coder->block_size bytes. The main thread will + /// put new input into this and update in_size accordingly. Once + /// no more input is coming, state will be set to THR_FINISH. + uint8_t *in; + + /// Amount of data available in the input buffer. This is modified + /// only by the main thread. + size_t in_size; + + /// Output buffer for this thread. This is set by the main + /// thread every time a new Block is started with this thread + /// structure. + lzma_outbuf *outbuf; + + /// Pointer to the main structure is needed when putting this + /// thread back to the stack of free threads. + lzma_stream_coder *coder; + + /// The allocator is set by the main thread. Since a copy of the + /// pointer is kept here, the application must not change the + /// allocator before calling lzma_end(). + const lzma_allocator *allocator; + + /// Amount of uncompressed data that has already been compressed. + uint64_t progress_in; + + /// Amount of compressed data that is ready. + uint64_t progress_out; + + /// Block encoder + lzma_next_coder block_encoder; + + /// Compression options for this Block + lzma_block block_options; + + /// Filter chain for this thread. By copying the filters array + /// to each thread it is possible to change the filter chain + /// between Blocks using lzma_filters_update(). + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + /// Next structure in the stack of free worker threads. + worker_thread *next; + + mythread_mutex mutex; + mythread_cond cond; + + /// The ID of this thread is used to join the thread + /// when it's not needed anymore. + mythread thread_id; +}; + + +struct lzma_stream_coder_s { + enum { + SEQ_STREAM_HEADER, + SEQ_BLOCK, + SEQ_INDEX, + SEQ_STREAM_FOOTER, + } sequence; + + /// Start a new Block every block_size bytes of input unless + /// LZMA_FULL_FLUSH or LZMA_FULL_BARRIER is used earlier. + size_t block_size; + + /// The filter chain to use for the next Block. + /// This can be updated using lzma_filters_update() + /// after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH. + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + /// A copy of filters[] will be put here when attempting to get + /// a new worker thread. This will be copied to a worker thread + /// when a thread becomes free and then this cache is marked as + /// empty by setting [0].id = LZMA_VLI_UNKNOWN. Without this cache + /// the filter options from filters[] would get uselessly copied + /// multiple times (allocated and freed) when waiting for a new free + /// worker thread. + /// + /// This is freed if filters[] is updated via lzma_filters_update(). + lzma_filter filters_cache[LZMA_FILTERS_MAX + 1]; + + + /// Index to hold sizes of the Blocks + lzma_index *index; + + /// Index encoder + lzma_next_coder index_encoder; + + + /// Stream Flags for encoding the Stream Header and Stream Footer. + lzma_stream_flags stream_flags; + + /// Buffer to hold Stream Header and Stream Footer. + uint8_t header[LZMA_STREAM_HEADER_SIZE]; + + /// Read position in header[] + size_t header_pos; + + + /// Output buffer queue for compressed data + lzma_outq outq; + + /// How much memory to allocate for each lzma_outbuf.buf + size_t outbuf_alloc_size; + + + /// Maximum wait time if cannot use all the input and cannot + /// fill the output buffer. This is in milliseconds. + uint32_t timeout; + + + /// Error code from a worker thread + lzma_ret thread_error; + + /// Array of allocated thread-specific structures + worker_thread *threads; + + /// Number of structures in "threads" above. This is also the + /// number of threads that will be created at maximum. + uint32_t threads_max; + + /// Number of thread structures that have been initialized, and + /// thus the number of worker threads actually created so far. + uint32_t threads_initialized; + + /// Stack of free threads. When a thread finishes, it puts itself + /// back into this stack. This starts as empty because threads + /// are created only when actually needed. + worker_thread *threads_free; + + /// The most recent worker thread to which the main thread writes + /// the new input from the application. + worker_thread *thr; + + + /// Amount of uncompressed data in Blocks that have already + /// been finished. + uint64_t progress_in; + + /// Amount of compressed data in Stream Header + Blocks that + /// have already been finished. + uint64_t progress_out; + + + mythread_mutex mutex; + mythread_cond cond; +}; + + +/// Tell the main thread that something has gone wrong. +static void +worker_error(worker_thread *thr, lzma_ret ret) +{ + assert(ret != LZMA_OK); + assert(ret != LZMA_STREAM_END); + + mythread_sync(thr->coder->mutex) { + if (thr->coder->thread_error == LZMA_OK) + thr->coder->thread_error = ret; + + mythread_cond_signal(&thr->coder->cond); + } + + return; +} + + +static worker_state +worker_encode(worker_thread *thr, size_t *out_pos, worker_state state) +{ + assert(thr->progress_in == 0); + assert(thr->progress_out == 0); + + // Set the Block options. + thr->block_options = (lzma_block){ + .version = 0, + .check = thr->coder->stream_flags.check, + .compressed_size = thr->outbuf->allocated, + .uncompressed_size = thr->coder->block_size, + .filters = thr->filters, + }; + + // Calculate maximum size of the Block Header. This amount is + // reserved in the beginning of the buffer so that Block Header + // along with Compressed Size and Uncompressed Size can be + // written there. + lzma_ret ret = lzma_block_header_size(&thr->block_options); + if (ret != LZMA_OK) { + worker_error(thr, ret); + return THR_STOP; + } + + // Initialize the Block encoder. + ret = lzma_block_encoder_init(&thr->block_encoder, + thr->allocator, &thr->block_options); + if (ret != LZMA_OK) { + worker_error(thr, ret); + return THR_STOP; + } + + size_t in_pos = 0; + size_t in_size = 0; + + *out_pos = thr->block_options.header_size; + const size_t out_size = thr->outbuf->allocated; + + do { + mythread_sync(thr->mutex) { + // Store in_pos and *out_pos into *thr so that + // an application may read them via + // lzma_get_progress() to get progress information. + // + // NOTE: These aren't updated when the encoding + // finishes. Instead, the final values are taken + // later from thr->outbuf. + thr->progress_in = in_pos; + thr->progress_out = *out_pos; + + while (in_size == thr->in_size + && thr->state == THR_RUN) + mythread_cond_wait(&thr->cond, &thr->mutex); + + state = thr->state; + in_size = thr->in_size; + } + + // Return if we were asked to stop or exit. + if (state >= THR_STOP) + return state; + + lzma_action action = state == THR_FINISH + ? LZMA_FINISH : LZMA_RUN; + + // Limit the amount of input given to the Block encoder + // at once. This way this thread can react fairly quickly + // if the main thread wants us to stop or exit. + static const size_t in_chunk_max = 16384; + size_t in_limit = in_size; + if (in_size - in_pos > in_chunk_max) { + in_limit = in_pos + in_chunk_max; + action = LZMA_RUN; + } + + ret = thr->block_encoder.code( + thr->block_encoder.coder, thr->allocator, + thr->in, &in_pos, in_limit, thr->outbuf->buf, + out_pos, out_size, action); + } while (ret == LZMA_OK && *out_pos < out_size); + + switch (ret) { + case LZMA_STREAM_END: + assert(state == THR_FINISH); + + // Encode the Block Header. By doing it after + // the compression, we can store the Compressed Size + // and Uncompressed Size fields. + ret = lzma_block_header_encode(&thr->block_options, + thr->outbuf->buf); + if (ret != LZMA_OK) { + worker_error(thr, ret); + return THR_STOP; + } + + break; + + case LZMA_OK: + // The data was incompressible. Encode it using uncompressed + // LZMA2 chunks. + // + // First wait that we have gotten all the input. + mythread_sync(thr->mutex) { + while (thr->state == THR_RUN) + mythread_cond_wait(&thr->cond, &thr->mutex); + + state = thr->state; + in_size = thr->in_size; + } + + if (state >= THR_STOP) + return state; + + // Do the encoding. This takes care of the Block Header too. + *out_pos = 0; + ret = lzma_block_uncomp_encode(&thr->block_options, + thr->in, in_size, thr->outbuf->buf, + out_pos, out_size); + + // It shouldn't fail. + if (ret != LZMA_OK) { + worker_error(thr, LZMA_PROG_ERROR); + return THR_STOP; + } + + break; + + default: + worker_error(thr, ret); + return THR_STOP; + } + + // Set the size information that will be read by the main thread + // to write the Index field. + thr->outbuf->unpadded_size + = lzma_block_unpadded_size(&thr->block_options); + assert(thr->outbuf->unpadded_size != 0); + thr->outbuf->uncompressed_size = thr->block_options.uncompressed_size; + + return THR_FINISH; +} + + +static MYTHREAD_RET_TYPE +worker_start(void *thr_ptr) +{ + worker_thread *thr = thr_ptr; + worker_state state = THR_IDLE; // Init to silence a warning + + while (true) { + // Wait for work. + mythread_sync(thr->mutex) { + while (true) { + // The thread is already idle so if we are + // requested to stop, just set the state. + if (thr->state == THR_STOP) { + thr->state = THR_IDLE; + mythread_cond_signal(&thr->cond); + } + + state = thr->state; + if (state != THR_IDLE) + break; + + mythread_cond_wait(&thr->cond, &thr->mutex); + } + } + + size_t out_pos = 0; + + assert(state != THR_IDLE); + assert(state != THR_STOP); + + if (state <= THR_FINISH) + state = worker_encode(thr, &out_pos, state); + + if (state == THR_EXIT) + break; + + // Mark the thread as idle unless the main thread has + // told us to exit. Signal is needed for the case + // where the main thread is waiting for the threads to stop. + mythread_sync(thr->mutex) { + if (thr->state != THR_EXIT) { + thr->state = THR_IDLE; + mythread_cond_signal(&thr->cond); + } + } + + mythread_sync(thr->coder->mutex) { + // If no errors occurred, make the encoded data + // available to be copied out. + if (state == THR_FINISH) { + thr->outbuf->pos = out_pos; + thr->outbuf->finished = true; + } + + // Update the main progress info. + thr->coder->progress_in + += thr->outbuf->uncompressed_size; + thr->coder->progress_out += out_pos; + thr->progress_in = 0; + thr->progress_out = 0; + + // Return this thread to the stack of free threads. + thr->next = thr->coder->threads_free; + thr->coder->threads_free = thr; + + mythread_cond_signal(&thr->coder->cond); + } + } + + // Exiting, free the resources. + lzma_filters_free(thr->filters, thr->allocator); + + mythread_mutex_destroy(&thr->mutex); + mythread_cond_destroy(&thr->cond); + + lzma_next_end(&thr->block_encoder, thr->allocator); + lzma_free(thr->in, thr->allocator); + return MYTHREAD_RET_VALUE; +} + + +/// Make the threads stop but not exit. Optionally wait for them to stop. +static void +threads_stop(lzma_stream_coder *coder, bool wait_for_threads) +{ + // Tell the threads to stop. + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + coder->threads[i].state = THR_STOP; + mythread_cond_signal(&coder->threads[i].cond); + } + } + + if (!wait_for_threads) + return; + + // Wait for the threads to settle in the idle state. + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + while (coder->threads[i].state != THR_IDLE) + mythread_cond_wait(&coder->threads[i].cond, + &coder->threads[i].mutex); + } + } + + return; +} + + +/// Stop the threads and free the resources associated with them. +/// Wait until the threads have exited. +static void +threads_end(lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + coder->threads[i].state = THR_EXIT; + mythread_cond_signal(&coder->threads[i].cond); + } + } + + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + int ret = mythread_join(coder->threads[i].thread_id); + assert(ret == 0); + (void)ret; + } + + lzma_free(coder->threads, allocator); + return; +} + + +/// Initialize a new worker_thread structure and create a new thread. +static lzma_ret +initialize_new_thread(lzma_stream_coder *coder, + const lzma_allocator *allocator) +{ + worker_thread *thr = &coder->threads[coder->threads_initialized]; + + thr->in = lzma_alloc(coder->block_size, allocator); + if (thr->in == NULL) + return LZMA_MEM_ERROR; + + if (mythread_mutex_init(&thr->mutex)) + goto error_mutex; + + if (mythread_cond_init(&thr->cond)) + goto error_cond; + + thr->state = THR_IDLE; + thr->allocator = allocator; + thr->coder = coder; + thr->progress_in = 0; + thr->progress_out = 0; + thr->block_encoder = LZMA_NEXT_CODER_INIT; + thr->filters[0].id = LZMA_VLI_UNKNOWN; + + if (mythread_create(&thr->thread_id, &worker_start, thr)) + goto error_thread; + + ++coder->threads_initialized; + coder->thr = thr; + + return LZMA_OK; + +error_thread: + mythread_cond_destroy(&thr->cond); + +error_cond: + mythread_mutex_destroy(&thr->mutex); + +error_mutex: + lzma_free(thr->in, allocator); + return LZMA_MEM_ERROR; +} + + +static lzma_ret +get_thread(lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + // If there are no free output subqueues, there is no + // point to try getting a thread. + if (!lzma_outq_has_buf(&coder->outq)) + return LZMA_OK; + + // That's also true if we cannot allocate memory for the output + // buffer in the output queue. + return_if_error(lzma_outq_prealloc_buf(&coder->outq, allocator, + coder->outbuf_alloc_size)); + + // Make a thread-specific copy of the filter chain. Put it in + // the cache array first so that if we cannot get a new thread yet, + // the allocation is ready when we try again. + if (coder->filters_cache[0].id == LZMA_VLI_UNKNOWN) + return_if_error(lzma_filters_copy( + coder->filters, coder->filters_cache, allocator)); + + // If there is a free structure on the stack, use it. + mythread_sync(coder->mutex) { + if (coder->threads_free != NULL) { + coder->thr = coder->threads_free; + coder->threads_free = coder->threads_free->next; + } + } + + if (coder->thr == NULL) { + // If there are no uninitialized structures left, return. + if (coder->threads_initialized == coder->threads_max) + return LZMA_OK; + + // Initialize a new thread. + return_if_error(initialize_new_thread(coder, allocator)); + } + + // Reset the parts of the thread state that have to be done + // in the main thread. + mythread_sync(coder->thr->mutex) { + coder->thr->state = THR_RUN; + coder->thr->in_size = 0; + coder->thr->outbuf = lzma_outq_get_buf(&coder->outq, NULL); + + // Free the old thread-specific filter options and replace + // them with the already-allocated new options from + // coder->filters_cache[]. Then mark the cache as empty. + lzma_filters_free(coder->thr->filters, allocator); + memcpy(coder->thr->filters, coder->filters_cache, + sizeof(coder->filters_cache)); + coder->filters_cache[0].id = LZMA_VLI_UNKNOWN; + + mythread_cond_signal(&coder->thr->cond); + } + + return LZMA_OK; +} + + +static lzma_ret +stream_encode_in(lzma_stream_coder *coder, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, lzma_action action) +{ + while (*in_pos < in_size + || (coder->thr != NULL && action != LZMA_RUN)) { + if (coder->thr == NULL) { + // Get a new thread. + const lzma_ret ret = get_thread(coder, allocator); + if (coder->thr == NULL) + return ret; + } + + // Copy the input data to thread's buffer. + size_t thr_in_size = coder->thr->in_size; + lzma_bufcpy(in, in_pos, in_size, coder->thr->in, + &thr_in_size, coder->block_size); + + // Tell the Block encoder to finish if + // - it has got block_size bytes of input; or + // - all input was used and LZMA_FINISH, LZMA_FULL_FLUSH, + // or LZMA_FULL_BARRIER was used. + // + // TODO: LZMA_SYNC_FLUSH and LZMA_SYNC_BARRIER. + const bool finish = thr_in_size == coder->block_size + || (*in_pos == in_size && action != LZMA_RUN); + + bool block_error = false; + + mythread_sync(coder->thr->mutex) { + if (coder->thr->state == THR_IDLE) { + // Something has gone wrong with the Block + // encoder. It has set coder->thread_error + // which we will read a few lines later. + block_error = true; + } else { + // Tell the Block encoder its new amount + // of input and update the state if needed. + coder->thr->in_size = thr_in_size; + + if (finish) + coder->thr->state = THR_FINISH; + + mythread_cond_signal(&coder->thr->cond); + } + } + + if (block_error) { + lzma_ret ret = LZMA_OK; // Init to silence a warning. + + mythread_sync(coder->mutex) { + ret = coder->thread_error; + } + + return ret; + } + + if (finish) + coder->thr = NULL; + } + + return LZMA_OK; +} + + +/// Wait until more input can be consumed, more output can be read, or +/// an optional timeout is reached. +static bool +wait_for_work(lzma_stream_coder *coder, mythread_condtime *wait_abs, + bool *has_blocked, bool has_input) +{ + if (coder->timeout != 0 && !*has_blocked) { + // Every time when stream_encode_mt() is called via + // lzma_code(), *has_blocked starts as false. We set it + // to true here and calculate the absolute time when + // we must return if there's nothing to do. + // + // This way if we block multiple times for short moments + // less than "timeout" milliseconds, we will return once + // "timeout" amount of time has passed since the *first* + // blocking occurred. If the absolute time was calculated + // again every time we block, "timeout" would effectively + // be meaningless if we never consecutively block longer + // than "timeout" ms. + *has_blocked = true; + mythread_condtime_set(wait_abs, &coder->cond, coder->timeout); + } + + bool timed_out = false; + + mythread_sync(coder->mutex) { + // There are four things that we wait. If one of them + // becomes possible, we return. + // - If there is input left, we need to get a free + // worker thread and an output buffer for it. + // - Data ready to be read from the output queue. + // - A worker thread indicates an error. + // - Time out occurs. + while ((!has_input || coder->threads_free == NULL + || !lzma_outq_has_buf(&coder->outq)) + && !lzma_outq_is_readable(&coder->outq) + && coder->thread_error == LZMA_OK + && !timed_out) { + if (coder->timeout != 0) + timed_out = mythread_cond_timedwait( + &coder->cond, &coder->mutex, + wait_abs) != 0; + else + mythread_cond_wait(&coder->cond, + &coder->mutex); + } + } + + return timed_out; +} + + +static lzma_ret +stream_encode_mt(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_stream_coder *coder = coder_ptr; + + switch (coder->sequence) { + case SEQ_STREAM_HEADER: + lzma_bufcpy(coder->header, &coder->header_pos, + sizeof(coder->header), + out, out_pos, out_size); + if (coder->header_pos < sizeof(coder->header)) + return LZMA_OK; + + coder->header_pos = 0; + coder->sequence = SEQ_BLOCK; + FALLTHROUGH; + + case SEQ_BLOCK: { + // Initialized to silence warnings. + lzma_vli unpadded_size = 0; + lzma_vli uncompressed_size = 0; + lzma_ret ret = LZMA_OK; + + // These are for wait_for_work(). + bool has_blocked = false; + mythread_condtime wait_abs = { 0 }; + + while (true) { + mythread_sync(coder->mutex) { + // Check for Block encoder errors. + ret = coder->thread_error; + if (ret != LZMA_OK) { + assert(ret != LZMA_STREAM_END); + break; // Break out of mythread_sync. + } + + // Try to read compressed data to out[]. + ret = lzma_outq_read(&coder->outq, allocator, + out, out_pos, out_size, + &unpadded_size, + &uncompressed_size); + } + + if (ret == LZMA_STREAM_END) { + // End of Block. Add it to the Index. + ret = lzma_index_append(coder->index, + allocator, unpadded_size, + uncompressed_size); + if (ret != LZMA_OK) { + threads_stop(coder, false); + return ret; + } + + // If we didn't fill the output buffer yet, + // try to read more data. Maybe the next + // outbuf has been finished already too. + if (*out_pos < out_size) + continue; + } + + if (ret != LZMA_OK) { + // coder->thread_error was set. + threads_stop(coder, false); + return ret; + } + + // Try to give uncompressed data to a worker thread. + ret = stream_encode_in(coder, allocator, + in, in_pos, in_size, action); + if (ret != LZMA_OK) { + threads_stop(coder, false); + return ret; + } + + // See if we should wait or return. + // + // TODO: LZMA_SYNC_FLUSH and LZMA_SYNC_BARRIER. + if (*in_pos == in_size) { + // LZMA_RUN: More data is probably coming + // so return to let the caller fill the + // input buffer. + if (action == LZMA_RUN) + return LZMA_OK; + + // LZMA_FULL_BARRIER: The same as with + // LZMA_RUN but tell the caller that the + // barrier was completed. + if (action == LZMA_FULL_BARRIER) + return LZMA_STREAM_END; + + // Finishing or flushing isn't completed until + // all input data has been encoded and copied + // to the output buffer. + if (lzma_outq_is_empty(&coder->outq)) { + // LZMA_FINISH: Continue to encode + // the Index field. + if (action == LZMA_FINISH) + break; + + // LZMA_FULL_FLUSH: Return to tell + // the caller that flushing was + // completed. + if (action == LZMA_FULL_FLUSH) + return LZMA_STREAM_END; + } + } + + // Return if there is no output space left. + // This check must be done after testing the input + // buffer, because we might want to use a different + // return code. + if (*out_pos == out_size) + return LZMA_OK; + + // Neither in nor out has been used completely. + // Wait until there's something we can do. + if (wait_for_work(coder, &wait_abs, &has_blocked, + *in_pos < in_size)) + return LZMA_TIMED_OUT; + } + + // All Blocks have been encoded and the threads have stopped. + // Prepare to encode the Index field. + return_if_error(lzma_index_encoder_init( + &coder->index_encoder, allocator, + coder->index)); + coder->sequence = SEQ_INDEX; + + // Update the progress info to take the Index and + // Stream Footer into account. Those are very fast to encode + // so in terms of progress information they can be thought + // to be ready to be copied out. + coder->progress_out += lzma_index_size(coder->index) + + LZMA_STREAM_HEADER_SIZE; + + FALLTHROUGH; + } + + case SEQ_INDEX: { + // Call the Index encoder. It doesn't take any input, so + // those pointers can be NULL. + const lzma_ret ret = coder->index_encoder.code( + coder->index_encoder.coder, allocator, + NULL, NULL, 0, + out, out_pos, out_size, LZMA_RUN); + if (ret != LZMA_STREAM_END) + return ret; + + // Encode the Stream Footer into coder->buffer. + coder->stream_flags.backward_size + = lzma_index_size(coder->index); + if (lzma_stream_footer_encode(&coder->stream_flags, + coder->header) != LZMA_OK) + return LZMA_PROG_ERROR; + + coder->sequence = SEQ_STREAM_FOOTER; + FALLTHROUGH; + } + + case SEQ_STREAM_FOOTER: + lzma_bufcpy(coder->header, &coder->header_pos, + sizeof(coder->header), + out, out_pos, out_size); + return coder->header_pos < sizeof(coder->header) + ? LZMA_OK : LZMA_STREAM_END; + } + + assert(0); + return LZMA_PROG_ERROR; +} + + +static void +stream_encoder_mt_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_stream_coder *coder = coder_ptr; + + // Threads must be killed before the output queue can be freed. + threads_end(coder, allocator); + lzma_outq_end(&coder->outq, allocator); + + lzma_filters_free(coder->filters, allocator); + lzma_filters_free(coder->filters_cache, allocator); + + lzma_next_end(&coder->index_encoder, allocator); + lzma_index_end(coder->index, allocator); + + mythread_cond_destroy(&coder->cond); + mythread_mutex_destroy(&coder->mutex); + + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +stream_encoder_mt_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters, + const lzma_filter *reversed_filters + lzma_attribute((__unused__))) +{ + lzma_stream_coder *coder = coder_ptr; + + // Applications shouldn't attempt to change the options when + // we are already encoding the Index or Stream Footer. + if (coder->sequence > SEQ_BLOCK) + return LZMA_PROG_ERROR; + + // For now the threaded encoder doesn't support changing + // the options in the middle of a Block. + if (coder->thr != NULL) + return LZMA_PROG_ERROR; + + // Check if the filter chain seems mostly valid. See the comment + // in stream_encoder_mt_init(). + if (lzma_raw_encoder_memusage(filters) == UINT64_MAX) + return LZMA_OPTIONS_ERROR; + + // Make a copy to a temporary buffer first. This way the encoder + // state stays unchanged if an error occurs in lzma_filters_copy(). + lzma_filter temp[LZMA_FILTERS_MAX + 1]; + return_if_error(lzma_filters_copy(filters, temp, allocator)); + + // Free the options of the old chain as well as the cache. + lzma_filters_free(coder->filters, allocator); + lzma_filters_free(coder->filters_cache, allocator); + + // Copy the new filter chain in place. + memcpy(coder->filters, temp, sizeof(temp)); + + return LZMA_OK; +} + + +/// Options handling for lzma_stream_encoder_mt_init() and +/// lzma_stream_encoder_mt_memusage() +static lzma_ret +get_options(const lzma_mt *options, lzma_options_easy *opt_easy, + const lzma_filter **filters, uint64_t *block_size, + uint64_t *outbuf_size_max) +{ + // Validate some of the options. + if (options == NULL) + return LZMA_PROG_ERROR; + + if (options->flags != 0 || options->threads == 0 + || options->threads > LZMA_THREADS_MAX) + return LZMA_OPTIONS_ERROR; + + if (options->filters != NULL) { + // Filter chain was given, use it as is. + *filters = options->filters; + } else { + // Use a preset. + if (lzma_easy_preset(opt_easy, options->preset)) + return LZMA_OPTIONS_ERROR; + + *filters = opt_easy->filters; + } + + // If the Block size is not set, determine it from the filter chain. + if (options->block_size > 0) + *block_size = options->block_size; + else + *block_size = lzma_mt_block_size(*filters); + + // UINT64_MAX > BLOCK_SIZE_MAX, so the second condition + // should be optimized out by any reasonable compiler. + // The second condition should be there in the unlikely event that + // the macros change and UINT64_MAX < BLOCK_SIZE_MAX. + if (*block_size > BLOCK_SIZE_MAX || *block_size == UINT64_MAX) + return LZMA_OPTIONS_ERROR; + + // Calculate the maximum amount output that a single output buffer + // may need to hold. This is the same as the maximum total size of + // a Block. + *outbuf_size_max = lzma_block_buffer_bound64(*block_size); + if (*outbuf_size_max == 0) + return LZMA_MEM_ERROR; + + return LZMA_OK; +} + + +static void +get_progress(void *coder_ptr, uint64_t *progress_in, uint64_t *progress_out) +{ + lzma_stream_coder *coder = coder_ptr; + + // Lock coder->mutex to prevent finishing threads from moving their + // progress info from the worker_thread structure to lzma_stream_coder. + mythread_sync(coder->mutex) { + *progress_in = coder->progress_in; + *progress_out = coder->progress_out; + + for (size_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + *progress_in += coder->threads[i].progress_in; + *progress_out += coder->threads[i] + .progress_out; + } + } + } + + return; +} + + +static lzma_ret +stream_encoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_mt *options) +{ + lzma_next_coder_init(&stream_encoder_mt_init, next, allocator); + + // Get the filter chain. + lzma_options_easy easy; + const lzma_filter *filters; + uint64_t block_size; + uint64_t outbuf_size_max; + return_if_error(get_options(options, &easy, &filters, + &block_size, &outbuf_size_max)); + +#if SIZE_MAX < UINT64_MAX + if (block_size > SIZE_MAX || outbuf_size_max > SIZE_MAX) + return LZMA_MEM_ERROR; +#endif + + // Validate the filter chain so that we can give an error in this + // function instead of delaying it to the first call to lzma_code(). + // The memory usage calculation verifies the filter chain as + // a side effect so we take advantage of that. It's not a perfect + // check though as raw encoder allows LZMA1 too but such problems + // will be caught eventually with Block Header encoder. + if (lzma_raw_encoder_memusage(filters) == UINT64_MAX) + return LZMA_OPTIONS_ERROR; + + // Validate the Check ID. + if ((unsigned int)(options->check) > LZMA_CHECK_ID_MAX) + return LZMA_PROG_ERROR; + + if (!lzma_check_is_supported(options->check)) + return LZMA_UNSUPPORTED_CHECK; + + // Allocate and initialize the base structure if needed. + lzma_stream_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_stream_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + + // For the mutex and condition variable initializations + // the error handling has to be done here because + // stream_encoder_mt_end() doesn't know if they have + // already been initialized or not. + if (mythread_mutex_init(&coder->mutex)) { + lzma_free(coder, allocator); + next->coder = NULL; + return LZMA_MEM_ERROR; + } + + if (mythread_cond_init(&coder->cond)) { + mythread_mutex_destroy(&coder->mutex); + lzma_free(coder, allocator); + next->coder = NULL; + return LZMA_MEM_ERROR; + } + + next->code = &stream_encode_mt; + next->end = &stream_encoder_mt_end; + next->get_progress = &get_progress; + next->update = &stream_encoder_mt_update; + + coder->filters[0].id = LZMA_VLI_UNKNOWN; + coder->filters_cache[0].id = LZMA_VLI_UNKNOWN; + coder->index_encoder = LZMA_NEXT_CODER_INIT; + coder->index = NULL; + memzero(&coder->outq, sizeof(coder->outq)); + coder->threads = NULL; + coder->threads_max = 0; + coder->threads_initialized = 0; + } + + // Basic initializations + coder->sequence = SEQ_STREAM_HEADER; + coder->block_size = (size_t)(block_size); + coder->outbuf_alloc_size = (size_t)(outbuf_size_max); + coder->thread_error = LZMA_OK; + coder->thr = NULL; + + // Allocate the thread-specific base structures. + assert(options->threads > 0); + if (coder->threads_max != options->threads) { + threads_end(coder, allocator); + + coder->threads = NULL; + coder->threads_max = 0; + + coder->threads_initialized = 0; + coder->threads_free = NULL; + + coder->threads = lzma_alloc( + options->threads * sizeof(worker_thread), + allocator); + if (coder->threads == NULL) + return LZMA_MEM_ERROR; + + coder->threads_max = options->threads; + } else { + // Reuse the old structures and threads. Tell the running + // threads to stop and wait until they have stopped. + threads_stop(coder, true); + } + + // Output queue + return_if_error(lzma_outq_init(&coder->outq, allocator, + options->threads)); + + // Timeout + coder->timeout = options->timeout; + + // Free the old filter chain and the cache. + lzma_filters_free(coder->filters, allocator); + lzma_filters_free(coder->filters_cache, allocator); + + // Copy the new filter chain. + return_if_error(lzma_filters_copy( + filters, coder->filters, allocator)); + + // Index + lzma_index_end(coder->index, allocator); + coder->index = lzma_index_init(allocator); + if (coder->index == NULL) + return LZMA_MEM_ERROR; + + // Stream Header + coder->stream_flags.version = 0; + coder->stream_flags.check = options->check; + return_if_error(lzma_stream_header_encode( + &coder->stream_flags, coder->header)); + + coder->header_pos = 0; + + // Progress info + coder->progress_in = 0; + coder->progress_out = LZMA_STREAM_HEADER_SIZE; + + return LZMA_OK; +} + + +#ifdef HAVE_SYMBOL_VERSIONS_LINUX +// These are for compatibility with binaries linked against liblzma that +// has been patched with xz-5.2.2-compat-libs.patch from RHEL/CentOS 7. +// Actually that patch didn't create lzma_stream_encoder_mt@XZ_5.2.2 +// but it has been added here anyway since someone might misread the +// RHEL patch and think both @XZ_5.1.2alpha and @XZ_5.2.2 exist. +LZMA_SYMVER_API("lzma_stream_encoder_mt@XZ_5.1.2alpha", + lzma_ret, lzma_stream_encoder_mt_512a)( + lzma_stream *strm, const lzma_mt *options) + lzma_nothrow lzma_attr_warn_unused_result + __attribute__((__alias__("lzma_stream_encoder_mt_52"))); + +LZMA_SYMVER_API("lzma_stream_encoder_mt@XZ_5.2.2", + lzma_ret, lzma_stream_encoder_mt_522)( + lzma_stream *strm, const lzma_mt *options) + lzma_nothrow lzma_attr_warn_unused_result + __attribute__((__alias__("lzma_stream_encoder_mt_52"))); + +LZMA_SYMVER_API("lzma_stream_encoder_mt@@XZ_5.2", + lzma_ret, lzma_stream_encoder_mt_52)( + lzma_stream *strm, const lzma_mt *options) + lzma_nothrow lzma_attr_warn_unused_result; + +#define lzma_stream_encoder_mt lzma_stream_encoder_mt_52 +#endif +extern LZMA_API(lzma_ret) +lzma_stream_encoder_mt(lzma_stream *strm, const lzma_mt *options) +{ + lzma_next_strm_init(stream_encoder_mt_init, strm, options); + + strm->internal->supported_actions[LZMA_RUN] = true; +// strm->internal->supported_actions[LZMA_SYNC_FLUSH] = true; + strm->internal->supported_actions[LZMA_FULL_FLUSH] = true; + strm->internal->supported_actions[LZMA_FULL_BARRIER] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} + + +#ifdef HAVE_SYMBOL_VERSIONS_LINUX +LZMA_SYMVER_API("lzma_stream_encoder_mt_memusage@XZ_5.1.2alpha", + uint64_t, lzma_stream_encoder_mt_memusage_512a)( + const lzma_mt *options) lzma_nothrow lzma_attr_pure + __attribute__((__alias__("lzma_stream_encoder_mt_memusage_52"))); + +LZMA_SYMVER_API("lzma_stream_encoder_mt_memusage@XZ_5.2.2", + uint64_t, lzma_stream_encoder_mt_memusage_522)( + const lzma_mt *options) lzma_nothrow lzma_attr_pure + __attribute__((__alias__("lzma_stream_encoder_mt_memusage_52"))); + +LZMA_SYMVER_API("lzma_stream_encoder_mt_memusage@@XZ_5.2", + uint64_t, lzma_stream_encoder_mt_memusage_52)( + const lzma_mt *options) lzma_nothrow lzma_attr_pure; + +#define lzma_stream_encoder_mt_memusage lzma_stream_encoder_mt_memusage_52 +#endif +// This function name is a monster but it's consistent with the older +// monster names. :-( 31 chars is the max that C99 requires so in that +// sense it's not too long. ;-) +extern LZMA_API(uint64_t) +lzma_stream_encoder_mt_memusage(const lzma_mt *options) +{ + lzma_options_easy easy; + const lzma_filter *filters; + uint64_t block_size; + uint64_t outbuf_size_max; + + if (get_options(options, &easy, &filters, &block_size, + &outbuf_size_max) != LZMA_OK) + return UINT64_MAX; + + // Memory usage of the input buffers + const uint64_t inbuf_memusage = options->threads * block_size; + + // Memory usage of the filter encoders + uint64_t filters_memusage = lzma_raw_encoder_memusage(filters); + if (filters_memusage == UINT64_MAX) + return UINT64_MAX; + + filters_memusage *= options->threads; + + // Memory usage of the output queue + const uint64_t outq_memusage = lzma_outq_memusage( + outbuf_size_max, options->threads); + if (outq_memusage == UINT64_MAX) + return UINT64_MAX; + + // Sum them with overflow checking. + uint64_t total_memusage = LZMA_MEMUSAGE_BASE + + sizeof(lzma_stream_coder) + + options->threads * sizeof(worker_thread); + + if (UINT64_MAX - total_memusage < inbuf_memusage) + return UINT64_MAX; + + total_memusage += inbuf_memusage; + + if (UINT64_MAX - total_memusage < filters_memusage) + return UINT64_MAX; + + total_memusage += filters_memusage; + + if (UINT64_MAX - total_memusage < outq_memusage) + return UINT64_MAX; + + return total_memusage + outq_memusage; +} diff --git a/src/liblzma/common/stream_flags_common.c b/src/liblzma/common/stream_flags_common.c new file mode 100644 index 0000000000..41b8dcb70d --- /dev/null +++ b/src/liblzma/common/stream_flags_common.c @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_flags_common.c +/// \brief Common stuff for Stream flags coders +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "stream_flags_common.h" + + +const uint8_t lzma_header_magic[6] = { 0xFD, 0x37, 0x7A, 0x58, 0x5A, 0x00 }; +const uint8_t lzma_footer_magic[2] = { 0x59, 0x5A }; + + +extern LZMA_API(lzma_ret) +lzma_stream_flags_compare( + const lzma_stream_flags *a, const lzma_stream_flags *b) +{ + // We can compare only version 0 structures. + if (a->version != 0 || b->version != 0) + return LZMA_OPTIONS_ERROR; + + // Check type + if ((unsigned int)(a->check) > LZMA_CHECK_ID_MAX + || (unsigned int)(b->check) > LZMA_CHECK_ID_MAX) + return LZMA_PROG_ERROR; + + if (a->check != b->check) + return LZMA_DATA_ERROR; + + // Backward Sizes are compared only if they are known in both. + if (a->backward_size != LZMA_VLI_UNKNOWN + && b->backward_size != LZMA_VLI_UNKNOWN) { + if (!is_backward_size_valid(a) || !is_backward_size_valid(b)) + return LZMA_PROG_ERROR; + + if (a->backward_size != b->backward_size) + return LZMA_DATA_ERROR; + } + + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_flags_common.h b/src/liblzma/common/stream_flags_common.h new file mode 100644 index 0000000000..28729dbcb6 --- /dev/null +++ b/src/liblzma/common/stream_flags_common.h @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_flags_common.h +/// \brief Common stuff for Stream flags coders +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_STREAM_FLAGS_COMMON_H +#define LZMA_STREAM_FLAGS_COMMON_H + +#include "common.h" + +/// Size of the Stream Flags field +#define LZMA_STREAM_FLAGS_SIZE 2 + +lzma_attr_visibility_hidden +extern const uint8_t lzma_header_magic[6]; + +lzma_attr_visibility_hidden +extern const uint8_t lzma_footer_magic[2]; + + +static inline bool +is_backward_size_valid(const lzma_stream_flags *options) +{ + return options->backward_size >= LZMA_BACKWARD_SIZE_MIN + && options->backward_size <= LZMA_BACKWARD_SIZE_MAX + && (options->backward_size & 3) == 0; +} + +#endif diff --git a/src/liblzma/common/stream_flags_decoder.c b/src/liblzma/common/stream_flags_decoder.c new file mode 100644 index 0000000000..522c98b6fd --- /dev/null +++ b/src/liblzma/common/stream_flags_decoder.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_flags_decoder.c +/// \brief Decodes Stream Header and Stream Footer from .xz files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "stream_flags_common.h" + + +static bool +stream_flags_decode(lzma_stream_flags *options, const uint8_t *in) +{ + // Reserved bits must be unset. + if (in[0] != 0x00 || (in[1] & 0xF0)) + return true; + + options->version = 0; + options->check = in[1] & 0x0F; + + return false; +} + + +extern LZMA_API(lzma_ret) +lzma_stream_header_decode(lzma_stream_flags *options, const uint8_t *in) +{ + // Magic + if (memcmp(in, lzma_header_magic, sizeof(lzma_header_magic)) != 0) + return LZMA_FORMAT_ERROR; + + // Verify the CRC32 so we can distinguish between corrupt + // and unsupported files. + const uint32_t crc = lzma_crc32(in + sizeof(lzma_header_magic), + LZMA_STREAM_FLAGS_SIZE, 0); + if (crc != read32le(in + sizeof(lzma_header_magic) + + LZMA_STREAM_FLAGS_SIZE)) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + return LZMA_DATA_ERROR; +#endif + } + + // Stream Flags + if (stream_flags_decode(options, in + sizeof(lzma_header_magic))) + return LZMA_OPTIONS_ERROR; + + // Set Backward Size to indicate unknown value. That way + // lzma_stream_flags_compare() can be used to compare Stream Header + // and Stream Footer while keeping it useful also for comparing + // two Stream Footers. + options->backward_size = LZMA_VLI_UNKNOWN; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_stream_footer_decode(lzma_stream_flags *options, const uint8_t *in) +{ + // Magic + if (memcmp(in + sizeof(uint32_t) * 2 + LZMA_STREAM_FLAGS_SIZE, + lzma_footer_magic, sizeof(lzma_footer_magic)) != 0) + return LZMA_FORMAT_ERROR; + + // CRC32 + const uint32_t crc = lzma_crc32(in + sizeof(uint32_t), + sizeof(uint32_t) + LZMA_STREAM_FLAGS_SIZE, 0); + if (crc != read32le(in)) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + return LZMA_DATA_ERROR; +#endif + } + + // Stream Flags + if (stream_flags_decode(options, in + sizeof(uint32_t) * 2)) + return LZMA_OPTIONS_ERROR; + + // Backward Size + options->backward_size = read32le(in + sizeof(uint32_t)); + options->backward_size = (options->backward_size + 1) * 4; + + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_flags_encoder.c b/src/liblzma/common/stream_flags_encoder.c new file mode 100644 index 0000000000..f94b5cd0a2 --- /dev/null +++ b/src/liblzma/common/stream_flags_encoder.c @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_flags_encoder.c +/// \brief Encodes Stream Header and Stream Footer for .xz files +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "stream_flags_common.h" + + +static bool +stream_flags_encode(const lzma_stream_flags *options, uint8_t *out) +{ + if ((unsigned int)(options->check) > LZMA_CHECK_ID_MAX) + return true; + + out[0] = 0x00; + out[1] = options->check; + + return false; +} + + +extern LZMA_API(lzma_ret) +lzma_stream_header_encode(const lzma_stream_flags *options, uint8_t *out) +{ + assert(sizeof(lzma_header_magic) + LZMA_STREAM_FLAGS_SIZE + + 4 == LZMA_STREAM_HEADER_SIZE); + + if (options->version != 0) + return LZMA_OPTIONS_ERROR; + + // Magic + memcpy(out, lzma_header_magic, sizeof(lzma_header_magic)); + + // Stream Flags + if (stream_flags_encode(options, out + sizeof(lzma_header_magic))) + return LZMA_PROG_ERROR; + + // CRC32 of the Stream Header + const uint32_t crc = lzma_crc32(out + sizeof(lzma_header_magic), + LZMA_STREAM_FLAGS_SIZE, 0); + + write32le(out + sizeof(lzma_header_magic) + LZMA_STREAM_FLAGS_SIZE, + crc); + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_stream_footer_encode(const lzma_stream_flags *options, uint8_t *out) +{ + assert(2 * 4 + LZMA_STREAM_FLAGS_SIZE + sizeof(lzma_footer_magic) + == LZMA_STREAM_HEADER_SIZE); + + if (options->version != 0) + return LZMA_OPTIONS_ERROR; + + // Backward Size + if (!is_backward_size_valid(options)) + return LZMA_PROG_ERROR; + + write32le(out + 4, options->backward_size / 4 - 1); + + // Stream Flags + if (stream_flags_encode(options, out + 2 * 4)) + return LZMA_PROG_ERROR; + + // CRC32 + const uint32_t crc = lzma_crc32( + out + 4, 4 + LZMA_STREAM_FLAGS_SIZE, 0); + + write32le(out, crc); + + // Magic + memcpy(out + 2 * 4 + LZMA_STREAM_FLAGS_SIZE, + lzma_footer_magic, sizeof(lzma_footer_magic)); + + return LZMA_OK; +} diff --git a/src/liblzma/common/string_conversion.c b/src/liblzma/common/string_conversion.c new file mode 100644 index 0000000000..015acf2258 --- /dev/null +++ b/src/liblzma/common/string_conversion.c @@ -0,0 +1,1363 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file string_conversion.c +/// \brief Conversion of strings to filter chain and vice versa +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_common.h" + + +// liblzma itself doesn't use gettext to translate messages. +// Mark the strings still so that xz can translate them. +#define N_(msgid) msgid + + +///////////////////// +// String building // +///////////////////// + +/// How much memory to allocate for strings. For now, no realloc is used +/// so this needs to be big enough even though there of course is +/// an overflow check still. +/// +/// FIXME? Using a fixed size is wasteful if the application doesn't free +/// the string fairly quickly but this can be improved later if needed. +#define STR_ALLOC_SIZE 800 + + +typedef struct { + char *buf; + size_t pos; +} lzma_str; + + +static lzma_ret +str_init(lzma_str *str, const lzma_allocator *allocator) +{ + str->buf = lzma_alloc(STR_ALLOC_SIZE, allocator); + if (str->buf == NULL) + return LZMA_MEM_ERROR; + + str->pos = 0; + return LZMA_OK; +} + + +static void +str_free(lzma_str *str, const lzma_allocator *allocator) +{ + lzma_free(str->buf, allocator); + return; +} + + +static bool +str_is_full(const lzma_str *str) +{ + return str->pos == STR_ALLOC_SIZE - 1; +} + + +static lzma_ret +str_finish(char **dest, lzma_str *str, const lzma_allocator *allocator) +{ + if (str_is_full(str)) { + // The preallocated buffer was too small. + // This shouldn't happen as STR_ALLOC_SIZE should + // be adjusted if new filters are added. + lzma_free(str->buf, allocator); + *dest = NULL; + assert(0); + return LZMA_PROG_ERROR; + } + + str->buf[str->pos] = '\0'; + *dest = str->buf; + return LZMA_OK; +} + + +static void +str_append_str(lzma_str *str, const char *s) +{ + const size_t len = strlen(s); + const size_t limit = STR_ALLOC_SIZE - 1 - str->pos; + const size_t copy_size = my_min(len, limit); + + memcpy(str->buf + str->pos, s, copy_size); + str->pos += copy_size; + return; +} + + +static void +str_append_u32(lzma_str *str, uint32_t v, bool use_byte_suffix) +{ + if (v == 0) { + str_append_str(str, "0"); + } else { + // NOTE: Don't use plain "B" because xz and the parser in this + // file don't support it and at glance it may look like 8 + // (there cannot be a space before the suffix). + static const char suffixes[4][4] = { "", "KiB", "MiB", "GiB" }; + + size_t suf = 0; + if (use_byte_suffix) { + while ((v & 1023) == 0 + && suf < ARRAY_SIZE(suffixes) - 1) { + v >>= 10; + ++suf; + } + } + + // UINT32_MAX in base 10 would need 10 + 1 bytes. Remember + // that initializing to "" initializes all elements to + // zero so '\0'-termination gets handled by this. + char buf[16] = ""; + size_t pos = sizeof(buf) - 1; + + do { + buf[--pos] = '0' + (v % 10); + v /= 10; + } while (v != 0); + + str_append_str(str, buf + pos); + str_append_str(str, suffixes[suf]); + } + + return; +} + + +////////////////////////////////////////////// +// Parsing and stringification declarations // +////////////////////////////////////////////// + +/// Maximum length for filter and option names. +/// 11 chars + terminating '\0' + sizeof(uint32_t) = 16 bytes +#define NAME_LEN_MAX 11 + + +/// For option_map.flags: Use .u.map to do convert the input value +/// to an integer. Without this flag, .u.range.{min,max} are used +/// as the allowed range for the integer. +#define OPTMAP_USE_NAME_VALUE_MAP 0x01 + +/// For option_map.flags: Allow KiB/MiB/GiB in input string and use them in +/// the stringified output if the value is an exact multiple of these. +/// This is used e.g. for LZMA1/2 dictionary size. +#define OPTMAP_USE_BYTE_SUFFIX 0x02 + +/// For option_map.flags: If the integer value is zero then this option +/// won't be included in the stringified output. It's used e.g. for +/// BCJ filter start offset which usually is zero. +#define OPTMAP_NO_STRFY_ZERO 0x04 + +/// Possible values for option_map.type. Since OPTMAP_TYPE_UINT32 is 0, +/// it doesn't need to be specified in the initializers as it is +/// the implicit value. +enum { + OPTMAP_TYPE_UINT32, + OPTMAP_TYPE_LZMA_MODE, + OPTMAP_TYPE_LZMA_MATCH_FINDER, + OPTMAP_TYPE_LZMA_PRESET, +}; + + +/// This is for mapping string values in options to integers. +/// The last element of an array must have "" as the name. +/// It's used e.g. for match finder names in LZMA1/2. +typedef struct { + const char name[NAME_LEN_MAX + 1]; + const uint32_t value; +} name_value_map; + + +/// Each filter that has options needs an array of option_map structures. +/// The array doesn't need to be terminated as the functions take the +/// length of the array as an argument. +/// +/// When converting a string to filter options structure, option values +/// will be handled in a few different ways: +/// +/// (1) If .type equals OPTMAP_TYPE_LZMA_PRESET then LZMA1/2 preset string +/// is handled specially. +/// +/// (2) If .flags has OPTMAP_USE_NAME_VALUE_MAP set then the string is +/// converted to an integer using the name_value_map pointed by .u.map. +/// The last element in .u.map must have .name = "" as the terminator. +/// +/// (3) Otherwise the string is treated as a non-negative unsigned decimal +/// integer which must be in the range set in .u.range. If .flags has +/// OPTMAP_USE_BYTE_SUFFIX then KiB, MiB, and GiB suffixes are allowed. +/// +/// The integer value from (2) or (3) is then stored to filter_options +/// at the offset specified in .offset using the type specified in .type +/// (default is uint32_t). +/// +/// Stringifying a filter is done by processing a given number of options +/// in order from the beginning of an option_map array. The integer is +/// read from filter_options at .offset using the type from .type. +/// +/// If the integer is zero and .flags has OPTMAP_NO_STRFY_ZERO then the +/// option is skipped. +/// +/// If .flags has OPTMAP_USE_NAME_VALUE_MAP set then .u.map will be used +/// to convert the option to a string. If the map doesn't contain a string +/// for the integer value then "UNKNOWN" is used. +/// +/// If .flags doesn't have OPTMAP_USE_NAME_VALUE_MAP set then the integer is +/// converted to a decimal value. If OPTMAP_USE_BYTE_SUFFIX is used then KiB, +/// MiB, or GiB suffix is used if the value is an exact multiple of these. +/// Plain "B" suffix is never used. +typedef struct { + char name[NAME_LEN_MAX + 1]; + uint8_t type; + uint8_t flags; + uint16_t offset; + + union { + // NVHPC has problems with unions that contain pointers that + // are not the first members, so keep "map" at the top. + const name_value_map *map; + + struct { + uint32_t min; + uint32_t max; + } range; + } u; +} option_map; + + +static const char *parse_options(const char **const str, const char *str_end, + void *filter_options, + const option_map *const optmap, const size_t optmap_size); + + +///////// +// BCJ // +///////// + +#if defined(HAVE_ENCODER_X86) \ + || defined(HAVE_DECODER_X86) \ + || defined(HAVE_ENCODER_ARM) \ + || defined(HAVE_DECODER_ARM) \ + || defined(HAVE_ENCODER_ARMTHUMB) \ + || defined(HAVE_DECODER_ARMTHUMB) \ + || defined(HAVE_ENCODER_ARM64) \ + || defined(HAVE_DECODER_ARM64) \ + || defined(HAVE_ENCODER_POWERPC) \ + || defined(HAVE_DECODER_POWERPC) \ + || defined(HAVE_ENCODER_IA64) \ + || defined(HAVE_DECODER_IA64) \ + || defined(HAVE_ENCODER_SPARC) \ + || defined(HAVE_DECODER_SPARC) \ + || defined(HAVE_ENCODER_RISCV) \ + || defined(HAVE_DECODER_RISCV) +static const option_map bcj_optmap[] = { + { + .name = "start", + .flags = OPTMAP_NO_STRFY_ZERO | OPTMAP_USE_BYTE_SUFFIX, + .offset = offsetof(lzma_options_bcj, start_offset), + .u.range.min = 0, + .u.range.max = UINT32_MAX, + } +}; + + +static const char * +parse_bcj(const char **const str, const char *str_end, void *filter_options) +{ + // filter_options was zeroed on allocation and that is enough + // for the default value. + return parse_options(str, str_end, filter_options, + bcj_optmap, ARRAY_SIZE(bcj_optmap)); +} +#endif + + +/////////// +// Delta // +/////////// + +#if defined(HAVE_ENCODER_DELTA) || defined(HAVE_DECODER_DELTA) +static const option_map delta_optmap[] = { + { + .name = "dist", + .offset = offsetof(lzma_options_delta, dist), + .u.range.min = LZMA_DELTA_DIST_MIN, + .u.range.max = LZMA_DELTA_DIST_MAX, + } +}; + + +static const char * +parse_delta(const char **const str, const char *str_end, void *filter_options) +{ + lzma_options_delta *opts = filter_options; + opts->type = LZMA_DELTA_TYPE_BYTE; + opts->dist = LZMA_DELTA_DIST_MIN; + + return parse_options(str, str_end, filter_options, + delta_optmap, ARRAY_SIZE(delta_optmap)); +} +#endif + + +/////////////////// +// LZMA1 & LZMA2 // +/////////////////// + +/// Help string for presets +#define LZMA12_PRESET_STR "0-9[e]" + + +static const char * +parse_lzma12_preset(const char **const str, const char *str_end, + uint32_t *preset) +{ + assert(*str < str_end); + + if (!(**str >= '0' && **str <= '9')) + return N_("Unsupported preset"); + + *preset = (uint32_t)(**str - '0'); + + // NOTE: Remember to update LZMA12_PRESET_STR if this is modified! + while (++*str < str_end) { + switch (**str) { + case 'e': + *preset |= LZMA_PRESET_EXTREME; + break; + + default: + return N_("Unsupported flag in the preset"); + } + } + + return NULL; +} + + +static const char * +set_lzma12_preset(const char **const str, const char *str_end, + void *filter_options) +{ + uint32_t preset; + const char *errmsg = parse_lzma12_preset(str, str_end, &preset); + if (errmsg != NULL) + return errmsg; + + lzma_options_lzma *opts = filter_options; + if (lzma_lzma_preset(opts, preset)) + return N_("Unsupported preset"); + + return NULL; +} + + +static const name_value_map lzma12_mode_map[] = { + { "fast", LZMA_MODE_FAST }, + { "normal", LZMA_MODE_NORMAL }, + { "", 0 } +}; + + +static const name_value_map lzma12_mf_map[] = { + { "hc3", LZMA_MF_HC3 }, + { "hc4", LZMA_MF_HC4 }, + { "bt2", LZMA_MF_BT2 }, + { "bt3", LZMA_MF_BT3 }, + { "bt4", LZMA_MF_BT4 }, + { "", 0 } +}; + + +static const option_map lzma12_optmap[] = { + { + .name = "preset", + .type = OPTMAP_TYPE_LZMA_PRESET, + }, { + .name = "dict", + .flags = OPTMAP_USE_BYTE_SUFFIX, + .offset = offsetof(lzma_options_lzma, dict_size), + .u.range.min = LZMA_DICT_SIZE_MIN, + // FIXME? The max is really max for encoding but decoding + // would allow 4 GiB - 1 B. + .u.range.max = (UINT32_C(1) << 30) + (UINT32_C(1) << 29), + }, { + .name = "lc", + .offset = offsetof(lzma_options_lzma, lc), + .u.range.min = LZMA_LCLP_MIN, + .u.range.max = LZMA_LCLP_MAX, + }, { + .name = "lp", + .offset = offsetof(lzma_options_lzma, lp), + .u.range.min = LZMA_LCLP_MIN, + .u.range.max = LZMA_LCLP_MAX, + }, { + .name = "pb", + .offset = offsetof(lzma_options_lzma, pb), + .u.range.min = LZMA_PB_MIN, + .u.range.max = LZMA_PB_MAX, + }, { + .name = "mode", + .type = OPTMAP_TYPE_LZMA_MODE, + .flags = OPTMAP_USE_NAME_VALUE_MAP, + .offset = offsetof(lzma_options_lzma, mode), + .u.map = lzma12_mode_map, + }, { + .name = "nice", + .offset = offsetof(lzma_options_lzma, nice_len), + .u.range.min = 2, + .u.range.max = 273, + }, { + .name = "mf", + .type = OPTMAP_TYPE_LZMA_MATCH_FINDER, + .flags = OPTMAP_USE_NAME_VALUE_MAP, + .offset = offsetof(lzma_options_lzma, mf), + .u.map = lzma12_mf_map, + }, { + .name = "depth", + .offset = offsetof(lzma_options_lzma, depth), + .u.range.min = 0, + .u.range.max = UINT32_MAX, + } +}; + + +static const char * +parse_lzma12(const char **const str, const char *str_end, void *filter_options) +{ + lzma_options_lzma *opts = filter_options; + + // It cannot fail. + const bool preset_ret = lzma_lzma_preset(opts, LZMA_PRESET_DEFAULT); + assert(!preset_ret); + (void)preset_ret; + + const char *errmsg = parse_options(str, str_end, filter_options, + lzma12_optmap, ARRAY_SIZE(lzma12_optmap)); + if (errmsg != NULL) + return errmsg; + + if (opts->lc + opts->lp > LZMA_LCLP_MAX) + return N_("The sum of lc and lp must not exceed 4"); + + return NULL; +} + + +///////////////////////////////////////// +// Generic parsing and stringification // +///////////////////////////////////////// + +static const struct { + /// Name of the filter + char name[NAME_LEN_MAX + 1]; + + /// For lzma_str_to_filters: + /// Size of the filter-specific options structure. + uint32_t opts_size; + + /// Filter ID + lzma_vli id; + + /// For lzma_str_to_filters: + /// Function to parse the filter-specific options. The filter_options + /// will already have been allocated using lzma_alloc_zero(). + const char *(*parse)(const char **str, const char *str_end, + void *filter_options); + + /// For lzma_str_from_filters: + /// If the flag LZMA_STR_ENCODER is used then the first + /// strfy_encoder elements of optmap are stringified. + /// With LZMA_STR_DECODER strfy_decoder is used. + /// Currently encoders use all options that decoders do but if + /// that changes then this needs to be changed too, for example, + /// add a new OPTMAP flag to skip printing some decoder-only options. + const option_map *optmap; + uint8_t strfy_encoder; + uint8_t strfy_decoder; + + /// For lzma_str_from_filters: + /// If true, lzma_filter.options is allowed to be NULL. In that case, + /// only the filter name is printed without any options. + bool allow_null; + +} filter_name_map[] = { +#if defined (HAVE_ENCODER_LZMA1) || defined(HAVE_DECODER_LZMA1) + { "lzma1", sizeof(lzma_options_lzma), LZMA_FILTER_LZMA1, + &parse_lzma12, lzma12_optmap, 9, 5, false }, +#endif + +#if defined(HAVE_ENCODER_LZMA2) || defined(HAVE_DECODER_LZMA2) + { "lzma2", sizeof(lzma_options_lzma), LZMA_FILTER_LZMA2, + &parse_lzma12, lzma12_optmap, 9, 2, false }, +#endif + +#if defined(HAVE_ENCODER_X86) || defined(HAVE_DECODER_X86) + { "x86", sizeof(lzma_options_bcj), LZMA_FILTER_X86, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_ARM) || defined(HAVE_DECODER_ARM) + { "arm", sizeof(lzma_options_bcj), LZMA_FILTER_ARM, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_ARMTHUMB) || defined(HAVE_DECODER_ARMTHUMB) + { "armthumb", sizeof(lzma_options_bcj), LZMA_FILTER_ARMTHUMB, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_ARM64) || defined(HAVE_DECODER_ARM64) + { "arm64", sizeof(lzma_options_bcj), LZMA_FILTER_ARM64, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_RISCV) || defined(HAVE_DECODER_RISCV) + { "riscv", sizeof(lzma_options_bcj), LZMA_FILTER_RISCV, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_POWERPC) || defined(HAVE_DECODER_POWERPC) + { "powerpc", sizeof(lzma_options_bcj), LZMA_FILTER_POWERPC, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_IA64) || defined(HAVE_DECODER_IA64) + { "ia64", sizeof(lzma_options_bcj), LZMA_FILTER_IA64, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_SPARC) || defined(HAVE_DECODER_SPARC) + { "sparc", sizeof(lzma_options_bcj), LZMA_FILTER_SPARC, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_DELTA) || defined(HAVE_DECODER_DELTA) + { "delta", sizeof(lzma_options_delta), LZMA_FILTER_DELTA, + &parse_delta, delta_optmap, 1, 1, false }, +#endif +}; + + +/// Decodes options from a string for one filter (name1=value1,name2=value2). +/// Caller must have allocated memory for filter_options already and set +/// the initial default values. This is called from the filter-specific +/// parse_* functions. +/// +/// The input string starts at *str and the address in str_end is the first +/// char that is not part of the string anymore. So no '\0' terminator is +/// used. *str is advanced every time something has been decoded successfully. +static const char * +parse_options(const char **const str, const char *str_end, + void *filter_options, + const option_map *const optmap, const size_t optmap_size) +{ + while (*str < str_end && **str != '\0') { + // Each option is of the form name=value. + // Commas (',') separate options. Extra commas are ignored. + // Ignoring extra commas makes it simpler if an optional + // option stored in a shell variable which can be empty. + if (**str == ',') { + ++*str; + continue; + } + + // Find where the next name=value ends. + const size_t str_len = (size_t)(str_end - *str); + const char *name_eq_value_end = memchr(*str, ',', str_len); + if (name_eq_value_end == NULL) + name_eq_value_end = str_end; + + const char *equals_sign = memchr(*str, '=', + (size_t)(name_eq_value_end - *str)); + + // Fail if the '=' wasn't found or the option name is missing + // (the first char is '='). + if (equals_sign == NULL || **str == '=') + return N_("Options must be 'name=value' pairs " + "separated with commas"); + + // Reject a too long option name so that the memcmp() + // in the loop below won't read past the end of the + // string in optmap[i].name. + const size_t name_len = (size_t)(equals_sign - *str); + if (name_len > NAME_LEN_MAX) + return N_("Unknown option name"); + + // Find the option name from optmap[]. + size_t i = 0; + while (true) { + if (i == optmap_size) + return N_("Unknown option name"); + + if (memcmp(*str, optmap[i].name, name_len) == 0 + && optmap[i].name[name_len] == '\0') + break; + + ++i; + } + + // The input string is good at least until the start of + // the option value. + *str = equals_sign + 1; + + // The code assumes that the option value isn't an empty + // string so check it here. + const size_t value_len = (size_t)(name_eq_value_end - *str); + if (value_len == 0) + return N_("Option value cannot be empty"); + + // LZMA1/2 preset has its own parsing function. + if (optmap[i].type == OPTMAP_TYPE_LZMA_PRESET) { + const char *errmsg = set_lzma12_preset(str, + name_eq_value_end, filter_options); + if (errmsg != NULL) + return errmsg; + + continue; + } + + // It's an integer value. + uint32_t v; + if (optmap[i].flags & OPTMAP_USE_NAME_VALUE_MAP) { + // The integer is picked from a string-to-integer map. + // + // Reject a too long value string so that the memcmp() + // in the loop below won't read past the end of the + // string in optmap[i].u.map[j].name. + if (value_len > NAME_LEN_MAX) + return N_("Invalid option value"); + + const name_value_map *map = optmap[i].u.map; + size_t j = 0; + while (true) { + // The array is terminated with an empty name. + if (map[j].name[0] == '\0') + return N_("Invalid option value"); + + if (memcmp(*str, map[j].name, value_len) == 0 + && map[j].name[value_len] + == '\0') { + v = map[j].value; + break; + } + + ++j; + } + } else if (**str < '0' || **str > '9') { + // Note that "max" isn't supported while it is + // supported in xz. It's not useful here. + return N_("Value is not a non-negative " + "decimal integer"); + } else { + // strtoul() has locale-specific behavior so it cannot + // be relied on to get reproducible results since we + // cannot change the locate in a thread-safe library. + // It also needs '\0'-termination. + // + // Use a temporary pointer so that *str will point + // to the beginning of the value string in case + // an error occurs. + const char *p = *str; + v = 0; + do { + if (v > UINT32_MAX / 10) + return N_("Value out of range"); + + v *= 10; + + const uint32_t add = (uint32_t)(*p - '0'); + if (UINT32_MAX - add < v) + return N_("Value out of range"); + + v += add; + ++p; + } while (p < name_eq_value_end + && *p >= '0' && *p <= '9'); + + if (p < name_eq_value_end) { + // Remember this position so that it can be + // used for error messages that are + // specifically about the suffix. (Out of + // range values are about the whole value + // and those error messages point to the + // beginning of the number part, + // not to the suffix.) + const char *multiplier_start = p; + + // If multiplier suffix shouldn't be used + // then don't allow them even if the value + // would stay within limits. This is a somewhat + // unnecessary check but it rejects silly + // things like lzma2:pb=0MiB which xz allows. + if ((optmap[i].flags & OPTMAP_USE_BYTE_SUFFIX) + == 0) { + *str = multiplier_start; + return N_("This option does not " + "support any multiplier " + "suffixes"); + } + + uint32_t shift; + + switch (*p) { + case 'k': + case 'K': + shift = 10; + break; + + case 'm': + case 'M': + shift = 20; + break; + + case 'g': + case 'G': + shift = 30; + break; + + default: + *str = multiplier_start; + + // TRANSLATORS: Don't translate the + // suffixes "KiB", "MiB", or "GiB" + // because a user can only specify + // untranslated suffixes. + return N_("Invalid multiplier suffix " + "(KiB, MiB, or GiB)"); + } + + ++p; + + // Allow "M", "Mi", "MB", "MiB" and the same + // for the other five characters from the + // switch-statement above. All are handled + // as base-2 (perhaps a mistake, perhaps not). + // Note that 'i' and 'B' are case sensitive. + if (p < name_eq_value_end && *p == 'i') + ++p; + + if (p < name_eq_value_end && *p == 'B') + ++p; + + // Now we must have no chars remaining. + if (p < name_eq_value_end) { + *str = multiplier_start; + return N_("Invalid multiplier suffix " + "(KiB, MiB, or GiB)"); + } + + if (v > (UINT32_MAX >> shift)) + return N_("Value out of range"); + + v <<= shift; + } + + if (v < optmap[i].u.range.min + || v > optmap[i].u.range.max) + return N_("Value out of range"); + } + + // Set the value in filter_options. Enums are handled + // specially since the underlying type isn't the same + // as uint32_t on all systems. + void *ptr = (char *)filter_options + optmap[i].offset; + switch (optmap[i].type) { + case OPTMAP_TYPE_LZMA_MODE: + *(lzma_mode *)ptr = (lzma_mode)v; + break; + + case OPTMAP_TYPE_LZMA_MATCH_FINDER: + *(lzma_match_finder *)ptr = (lzma_match_finder)v; + break; + + default: + *(uint32_t *)ptr = v; + break; + } + + // This option has been successfully handled. + *str = name_eq_value_end; + } + + // No errors. + return NULL; +} + + +/// Finds the name of the filter at the beginning of the string and +/// calls filter_name_map[i].parse() to decode the filter-specific options. +/// The caller must have set str_end so that exactly one filter and its +/// options are present without any trailing characters. +static const char * +parse_filter(const char **const str, const char *str_end, lzma_filter *filter, + const lzma_allocator *allocator, bool only_xz) +{ + // Search for a colon or equals sign that would separate the filter + // name from filter options. If neither is found, then the input + // string only contains a filter name and there are no options. + // + // First assume that a colon or equals sign won't be found: + const char *name_end = str_end; + const char *opts_start = str_end; + + for (const char *p = *str; p < str_end; ++p) { + if (*p == ':' || *p == '=') { + name_end = p; + + // Filter options (name1=value1,name2=value2,...) + // begin after the colon or equals sign. + opts_start = p + 1; + break; + } + } + + // Reject a too long filter name so that the memcmp() + // in the loop below won't read past the end of the + // string in filter_name_map[i].name. + const size_t name_len = (size_t)(name_end - *str); + if (name_len > NAME_LEN_MAX) + return N_("Unknown filter name"); + + for (size_t i = 0; i < ARRAY_SIZE(filter_name_map); ++i) { + if (memcmp(*str, filter_name_map[i].name, name_len) == 0 + && filter_name_map[i].name[name_len] == '\0') { + if (only_xz && filter_name_map[i].id + >= LZMA_FILTER_RESERVED_START) + return N_("This filter cannot be used in " + "the .xz format"); + + // Allocate the filter-specific options and + // initialize the memory with zeros. + void *options = lzma_alloc_zero( + filter_name_map[i].opts_size, + allocator); + if (options == NULL) + return N_("Memory allocation failed"); + + // Filter name was found so the input string is good + // at least this far. + *str = opts_start; + + const char *errmsg = filter_name_map[i].parse( + str, str_end, options); + if (errmsg != NULL) { + lzma_free(options, allocator); + return errmsg; + } + + // *filter is modified only when parsing is successful. + filter->id = filter_name_map[i].id; + filter->options = options; + return NULL; + } + } + + return N_("Unknown filter name"); +} + + +/// Converts the string to a filter chain (array of lzma_filter structures). +/// +/// *str is advanced every time something has been decoded successfully. +/// This way the caller knows where in the string a possible error occurred. +static const char * +str_to_filters(const char **const str, lzma_filter *filters, uint32_t flags, + const lzma_allocator *allocator) +{ + const char *errmsg; + + // Skip leading spaces. + while (**str == ' ') + ++*str; + + if (**str == '\0') + return N_("Empty string is not allowed, " + "try '6' if a default value is needed"); + + // Detect the type of the string. + // + // A string beginning with a digit or a string beginning with + // one dash and a digit are treated as presets. Trailing spaces + // will be ignored too (leading spaces were already ignored above). + // + // For example, "6", "7 ", "-9e", or " -3 " are treated as presets. + // Strings like "-" or "- " aren't preset. +#define MY_IS_DIGIT(c) ((c) >= '0' && (c) <= '9') + if (MY_IS_DIGIT(**str) || (**str == '-' && MY_IS_DIGIT((*str)[1]))) { + if (**str == '-') + ++*str; + + // Ignore trailing spaces. + const size_t str_len = strlen(*str); + const char *str_end = memchr(*str, ' ', str_len); + if (str_end != NULL) { + // There is at least one trailing space. Check that + // there are no chars other than spaces. + for (size_t i = 1; str_end[i] != '\0'; ++i) + if (str_end[i] != ' ') + return N_("Unsupported preset"); + } else { + // There are no trailing spaces. Use the whole string. + str_end = *str + str_len; + } + + uint32_t preset; + errmsg = parse_lzma12_preset(str, str_end, &preset); + if (errmsg != NULL) + return errmsg; + + lzma_options_lzma *opts = lzma_alloc(sizeof(*opts), allocator); + if (opts == NULL) + return N_("Memory allocation failed"); + + if (lzma_lzma_preset(opts, preset)) { + lzma_free(opts, allocator); + return N_("Unsupported preset"); + } + + filters[0].id = LZMA_FILTER_LZMA2; + filters[0].options = opts; + filters[1].id = LZMA_VLI_UNKNOWN; + filters[1].options = NULL; + + return NULL; + } + + // Not a preset so it must be a filter chain. + // + // If LZMA_STR_ALL_FILTERS isn't used we allow only filters that + // can be used in .xz. + const bool only_xz = (flags & LZMA_STR_ALL_FILTERS) == 0; + + // Use a temporary array so that we don't modify the caller-supplied + // one until we know that no errors occurred. + lzma_filter temp_filters[LZMA_FILTERS_MAX + 1]; + + size_t i = 0; + do { + if (i == LZMA_FILTERS_MAX) { + errmsg = N_("The maximum number of filters is four"); + goto error; + } + + // Skip "--" if present. + if ((*str)[0] == '-' && (*str)[1] == '-') + *str += 2; + + // Locate the end of "filter:name1=value1,name2=value2", + // stopping at the first "--" or a single space. + const char *filter_end = *str; + while (filter_end[0] != '\0') { + if ((filter_end[0] == '-' && filter_end[1] == '-') + || filter_end[0] == ' ') + break; + + ++filter_end; + } + + // Inputs that have "--" at the end or "-- " in the middle + // will result in an empty filter name. + if (filter_end == *str) { + errmsg = N_("Filter name is missing"); + goto error; + } + + errmsg = parse_filter(str, filter_end, &temp_filters[i], + allocator, only_xz); + if (errmsg != NULL) + goto error; + + // Skip trailing spaces. + while (**str == ' ') + ++*str; + + ++i; + } while (**str != '\0'); + + // Seems to be good, terminate the array so that + // basic validation can be done. + temp_filters[i].id = LZMA_VLI_UNKNOWN; + temp_filters[i].options = NULL; + + // Do basic validation if the application didn't prohibit it. + if ((flags & LZMA_STR_NO_VALIDATION) == 0) { + size_t dummy; + const lzma_ret ret = lzma_validate_chain(temp_filters, &dummy); + assert(ret == LZMA_OK || ret == LZMA_OPTIONS_ERROR); + if (ret != LZMA_OK) { + errmsg = N_("Invalid filter chain " + "('lzma2' missing at the end?)"); + goto error; + } + } + + // All good. Copy the filters to the application supplied array. + memcpy(filters, temp_filters, (i + 1) * sizeof(lzma_filter)); + return NULL; + +error: + // Free the filter options that were successfully decoded. + while (i-- > 0) + lzma_free(temp_filters[i].options, allocator); + + return errmsg; +} + + +extern LZMA_API(const char *) +lzma_str_to_filters(const char *str, int *error_pos, lzma_filter *filters, + uint32_t flags, const lzma_allocator *allocator) +{ + // If error_pos isn't NULL, *error_pos must always be set. + // liblzma <= 5.4.6 and <= 5.6.1 have a bug and don't do this + // when str == NULL or filters == NULL or flags are unsupported. + if (error_pos != NULL) + *error_pos = 0; + + if (str == NULL || filters == NULL) { + // Don't translate this because it's only shown in case of + // a programming error. + return "Unexpected NULL pointer argument(s) " + "to lzma_str_to_filters()"; + } + + // Validate the flags. + const uint32_t supported_flags + = LZMA_STR_ALL_FILTERS + | LZMA_STR_NO_VALIDATION; + + if (flags & ~supported_flags) { + // This message is possible only if the caller uses flags + // that are only supported in a newer liblzma version (or + // the flags are simply buggy). Don't translate this at least + // when liblzma itself doesn't use gettext; xz and liblzma + // are usually upgraded at the same time. + return "Unsupported flags to lzma_str_to_filters()"; + } + + const char *used = str; + const char *errmsg = str_to_filters(&used, filters, flags, allocator); + + if (error_pos != NULL) { + const size_t n = (size_t)(used - str); + *error_pos = n > INT_MAX ? INT_MAX : (int)n; + } + + return errmsg; +} + + +/// Converts options of one filter to a string. +/// +/// The caller must have already put the filter name in the destination +/// string. Since it is possible that no options will be needed, the caller +/// won't have put a delimiter character (':' or '=') in the string yet. +/// We will add it if at least one option will be added to the string. +static void +strfy_filter(lzma_str *dest, const char *delimiter, + const option_map *optmap, size_t optmap_count, + const void *filter_options) +{ + for (size_t i = 0; i < optmap_count; ++i) { + // No attempt is made to reverse LZMA1/2 preset. + if (optmap[i].type == OPTMAP_TYPE_LZMA_PRESET) + continue; + + // All options have integer values, some just are mapped + // to a string with a name_value_map. LZMA1/2 preset + // isn't reversed back to preset=PRESET form. + uint32_t v; + const void *ptr + = (const char *)filter_options + optmap[i].offset; + switch (optmap[i].type) { + case OPTMAP_TYPE_LZMA_MODE: + v = *(const lzma_mode *)ptr; + break; + + case OPTMAP_TYPE_LZMA_MATCH_FINDER: + v = *(const lzma_match_finder *)ptr; + break; + + default: + v = *(const uint32_t *)ptr; + break; + } + + // Skip this if this option should be omitted from + // the string when the value is zero. + if (v == 0 && (optmap[i].flags & OPTMAP_NO_STRFY_ZERO)) + continue; + + // Before the first option we add whatever delimiter + // the caller gave us. For later options a comma is used. + str_append_str(dest, delimiter); + delimiter = ","; + + // Add the option name and equals sign. + str_append_str(dest, optmap[i].name); + str_append_str(dest, "="); + + if (optmap[i].flags & OPTMAP_USE_NAME_VALUE_MAP) { + const name_value_map *map = optmap[i].u.map; + size_t j = 0; + while (true) { + if (map[j].name[0] == '\0') { + str_append_str(dest, "UNKNOWN"); + break; + } + + if (map[j].value == v) { + str_append_str(dest, map[j].name); + break; + } + + ++j; + } + } else { + str_append_u32(dest, v, + optmap[i].flags & OPTMAP_USE_BYTE_SUFFIX); + } + } + + return; +} + + +extern LZMA_API(lzma_ret) +lzma_str_from_filters(char **output_str, const lzma_filter *filters, + uint32_t flags, const lzma_allocator *allocator) +{ + // On error *output_str is always set to NULL. + // Do it as the very first step. + if (output_str == NULL) + return LZMA_PROG_ERROR; + + *output_str = NULL; + + if (filters == NULL) + return LZMA_PROG_ERROR; + + // Validate the flags. + const uint32_t supported_flags + = LZMA_STR_ENCODER + | LZMA_STR_DECODER + | LZMA_STR_GETOPT_LONG + | LZMA_STR_NO_SPACES; + + if (flags & ~supported_flags) + return LZMA_OPTIONS_ERROR; + + // There must be at least one filter. + if (filters[0].id == LZMA_VLI_UNKNOWN) + return LZMA_OPTIONS_ERROR; + + // Allocate memory for the output string. + lzma_str dest; + return_if_error(str_init(&dest, allocator)); + + const bool show_opts = (flags & (LZMA_STR_ENCODER | LZMA_STR_DECODER)); + + const char *opt_delim = (flags & LZMA_STR_GETOPT_LONG) ? "=" : ":"; + + for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + // If we reach LZMA_FILTERS_MAX, then the filters array + // is too large since the ID cannot be LZMA_VLI_UNKNOWN here. + if (i == LZMA_FILTERS_MAX) { + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + // Don't add a space between filters if the caller + // doesn't want them. + if (i > 0 && !(flags & LZMA_STR_NO_SPACES)) + str_append_str(&dest, " "); + + // Use dashes for xz getopt_long() compatible syntax but also + // use dashes to separate filters when spaces weren't wanted. + if ((flags & LZMA_STR_GETOPT_LONG) + || (i > 0 && (flags & LZMA_STR_NO_SPACES))) + str_append_str(&dest, "--"); + + size_t j = 0; + while (true) { + if (j == ARRAY_SIZE(filter_name_map)) { + // Filter ID in filters[i].id isn't supported. + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + if (filter_name_map[j].id == filters[i].id) { + // Add the filter name. + str_append_str(&dest, filter_name_map[j].name); + + // If only the filter names were wanted then + // skip to the next filter. In this case + // .options is ignored and may be NULL even + // when the filter doesn't allow NULL options. + if (!show_opts) + break; + + if (filters[i].options == NULL) { + if (!filter_name_map[j].allow_null) { + // Filter-specific options + // are missing but with + // this filter the options + // structure is mandatory. + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + // .options is allowed to be NULL. + // There is no need to add any + // options to the string. + break; + } + + // Options structure is available. Add + // the filter options to the string. + const size_t optmap_count + = (flags & LZMA_STR_ENCODER) + ? filter_name_map[j].strfy_encoder + : filter_name_map[j].strfy_decoder; + strfy_filter(&dest, opt_delim, + filter_name_map[j].optmap, + optmap_count, + filters[i].options); + break; + } + + ++j; + } + } + + return str_finish(output_str, &dest, allocator); +} + + +extern LZMA_API(lzma_ret) +lzma_str_list_filters(char **output_str, lzma_vli filter_id, uint32_t flags, + const lzma_allocator *allocator) +{ + // On error *output_str is always set to NULL. + // Do it as the very first step. + if (output_str == NULL) + return LZMA_PROG_ERROR; + + *output_str = NULL; + + // Validate the flags. + const uint32_t supported_flags + = LZMA_STR_ALL_FILTERS + | LZMA_STR_ENCODER + | LZMA_STR_DECODER + | LZMA_STR_GETOPT_LONG; + + if (flags & ~supported_flags) + return LZMA_OPTIONS_ERROR; + + // Allocate memory for the output string. + lzma_str dest; + return_if_error(str_init(&dest, allocator)); + + // If only listing the filter names then separate them with spaces. + // Otherwise use newlines. + const bool show_opts = (flags & (LZMA_STR_ENCODER | LZMA_STR_DECODER)); + const char *filter_delim = show_opts ? "\n" : " "; + + const char *opt_delim = (flags & LZMA_STR_GETOPT_LONG) ? "=" : ":"; + bool first_filter_printed = false; + + for (size_t i = 0; i < ARRAY_SIZE(filter_name_map); ++i) { + // If we are printing only one filter then skip others. + if (filter_id != LZMA_VLI_UNKNOWN + && filter_id != filter_name_map[i].id) + continue; + + // If we are printing only .xz filters then skip the others. + if (filter_name_map[i].id >= LZMA_FILTER_RESERVED_START + && (flags & LZMA_STR_ALL_FILTERS) == 0 + && filter_id == LZMA_VLI_UNKNOWN) + continue; + + // Add a new line if this isn't the first filter being + // written to the string. + if (first_filter_printed) + str_append_str(&dest, filter_delim); + + first_filter_printed = true; + + if (flags & LZMA_STR_GETOPT_LONG) + str_append_str(&dest, "--"); + + str_append_str(&dest, filter_name_map[i].name); + + // If only the filter names were wanted then continue + // to the next filter. + if (!show_opts) + continue; + + const option_map *optmap = filter_name_map[i].optmap; + const char *d = opt_delim; + + const size_t end = (flags & LZMA_STR_ENCODER) + ? filter_name_map[i].strfy_encoder + : filter_name_map[i].strfy_decoder; + + for (size_t j = 0; j < end; ++j) { + // The first option is delimited from the filter + // name using "=" or ":" and the rest of the options + // are separated with ",". + str_append_str(&dest, d); + d = ","; + + // optname= + str_append_str(&dest, optmap[j].name); + str_append_str(&dest, "=<"); + + if (optmap[j].type == OPTMAP_TYPE_LZMA_PRESET) { + // LZMA1/2 preset has its custom help string. + str_append_str(&dest, LZMA12_PRESET_STR); + } else if (optmap[j].flags + & OPTMAP_USE_NAME_VALUE_MAP) { + // Separate the possible option values by "|". + const name_value_map *m = optmap[j].u.map; + for (size_t k = 0; m[k].name[0] != '\0'; ++k) { + if (k > 0) + str_append_str(&dest, "|"); + + str_append_str(&dest, m[k].name); + } + } else { + // Integer range is shown as min-max. + const bool use_byte_suffix = optmap[j].flags + & OPTMAP_USE_BYTE_SUFFIX; + str_append_u32(&dest, optmap[j].u.range.min, + use_byte_suffix); + str_append_str(&dest, "-"); + str_append_u32(&dest, optmap[j].u.range.max, + use_byte_suffix); + } + + str_append_str(&dest, ">"); + } + } + + // If no filters were added to the string then it must be because + // the caller provided an unsupported Filter ID. + if (!first_filter_printed) { + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + return str_finish(output_str, &dest, allocator); +} diff --git a/src/liblzma/common/vli_decoder.c b/src/liblzma/common/vli_decoder.c new file mode 100644 index 0000000000..3254ccc35b --- /dev/null +++ b/src/liblzma/common/vli_decoder.c @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file vli_decoder.c +/// \brief Decodes variable-length integers +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + + +extern LZMA_API(lzma_ret) +lzma_vli_decode(lzma_vli *restrict vli, size_t *vli_pos, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size) +{ + // If we haven't been given vli_pos, work in single-call mode. + size_t vli_pos_internal = 0; + if (vli_pos == NULL) { + vli_pos = &vli_pos_internal; + *vli = 0; + + // If there's no input, use LZMA_DATA_ERROR. This way it is + // easy to decode VLIs from buffers that have known size, + // and get the correct error code in case the buffer is + // too short. + if (*in_pos >= in_size) + return LZMA_DATA_ERROR; + + } else { + // Initialize *vli when starting to decode a new integer. + if (*vli_pos == 0) + *vli = 0; + + // Validate the arguments. + if (*vli_pos >= LZMA_VLI_BYTES_MAX + || (*vli >> (*vli_pos * 7)) != 0) + return LZMA_PROG_ERROR;; + + if (*in_pos >= in_size) + return LZMA_BUF_ERROR; + } + + do { + // Read the next byte. Use a temporary variable so that we + // can update *in_pos immediately. + const uint8_t byte = in[*in_pos]; + ++*in_pos; + + // Add the newly read byte to *vli. + *vli += (lzma_vli)(byte & 0x7F) << (*vli_pos * 7); + ++*vli_pos; + + // Check if this is the last byte of a multibyte integer. + if ((byte & 0x80) == 0) { + // We don't allow using variable-length integers as + // padding i.e. the encoding must use the most the + // compact form. + if (byte == 0x00 && *vli_pos > 1) + return LZMA_DATA_ERROR; + + return vli_pos == &vli_pos_internal + ? LZMA_OK : LZMA_STREAM_END; + } + + // There is at least one more byte coming. If we have already + // read maximum number of bytes, the integer is considered + // corrupt. + // + // If we need bigger integers in future, old versions liblzma + // will confusingly indicate the file being corrupt instead of + // unsupported. I suppose it's still better this way, because + // in the foreseeable future (writing this in 2008) the only + // reason why files would appear having over 63-bit integers + // is that the files are simply corrupt. + if (*vli_pos == LZMA_VLI_BYTES_MAX) + return LZMA_DATA_ERROR; + + } while (*in_pos < in_size); + + return vli_pos == &vli_pos_internal ? LZMA_DATA_ERROR : LZMA_OK; +} diff --git a/src/liblzma/common/vli_encoder.c b/src/liblzma/common/vli_encoder.c new file mode 100644 index 0000000000..3859006a94 --- /dev/null +++ b/src/liblzma/common/vli_encoder.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file vli_encoder.c +/// \brief Encodes variable-length integers +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + + +extern LZMA_API(lzma_ret) +lzma_vli_encode(lzma_vli vli, size_t *vli_pos, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size) +{ + // If we haven't been given vli_pos, work in single-call mode. + size_t vli_pos_internal = 0; + if (vli_pos == NULL) { + vli_pos = &vli_pos_internal; + + // In single-call mode, we expect that the caller has + // reserved enough output space. + if (*out_pos >= out_size) + return LZMA_PROG_ERROR; + } else { + // This never happens when we are called by liblzma, but + // may happen if called directly from an application. + if (*out_pos >= out_size) + return LZMA_BUF_ERROR; + } + + // Validate the arguments. + if (*vli_pos >= LZMA_VLI_BYTES_MAX || vli > LZMA_VLI_MAX) + return LZMA_PROG_ERROR; + + // Shift vli so that the next bits to encode are the lowest. In + // single-call mode this never changes vli since *vli_pos is zero. + vli >>= *vli_pos * 7; + + // Write the non-last bytes in a loop. + while (vli >= 0x80) { + // We don't need *vli_pos during this function call anymore, + // but update it here so that it is ready if we need to + // return before the whole integer has been decoded. + ++*vli_pos; + assert(*vli_pos < LZMA_VLI_BYTES_MAX); + + // Write the next byte. + out[*out_pos] = (uint8_t)(vli) | 0x80; + vli >>= 7; + + if (++*out_pos == out_size) + return vli_pos == &vli_pos_internal + ? LZMA_PROG_ERROR : LZMA_OK; + } + + // Write the last byte. + out[*out_pos] = (uint8_t)(vli); + ++*out_pos; + ++*vli_pos; + + return vli_pos == &vli_pos_internal ? LZMA_OK : LZMA_STREAM_END; + +} diff --git a/src/liblzma/common/vli_size.c b/src/liblzma/common/vli_size.c new file mode 100644 index 0000000000..c8cb2ec10a --- /dev/null +++ b/src/liblzma/common/vli_size.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file vli_size.c +/// \brief Calculates the encoded size of a variable-length integer +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + + +extern LZMA_API(uint32_t) +lzma_vli_size(lzma_vli vli) +{ + if (vli > LZMA_VLI_MAX) + return 0; + + uint32_t i = 0; + do { + vli >>= 7; + ++i; + } while (vli != 0); + + assert(i <= LZMA_VLI_BYTES_MAX); + return i; +} diff --git a/src/liblzma/delta/Makefile.inc b/src/liblzma/delta/Makefile.inc new file mode 100644 index 0000000000..e5b919e0ed --- /dev/null +++ b/src/liblzma/delta/Makefile.inc @@ -0,0 +1,19 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +liblzma_la_SOURCES += \ + delta/delta_common.c \ + delta/delta_common.h \ + delta/delta_private.h + +if COND_ENCODER_DELTA +liblzma_la_SOURCES += \ + delta/delta_encoder.c \ + delta/delta_encoder.h +endif + +if COND_DECODER_DELTA +liblzma_la_SOURCES += \ + delta/delta_decoder.c \ + delta/delta_decoder.h +endif diff --git a/src/liblzma/delta/delta_common.c b/src/liblzma/delta/delta_common.c new file mode 100644 index 0000000000..5dbe253b4b --- /dev/null +++ b/src/liblzma/delta/delta_common.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_common.c +/// \brief Common stuff for Delta encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "delta_common.h" +#include "delta_private.h" + + +static void +delta_coder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_delta_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder, allocator); + return; +} + + +extern lzma_ret +lzma_delta_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + // Allocate memory for the decoder if needed. + lzma_delta_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_delta_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + + // End function is the same for encoder and decoder. + next->end = &delta_coder_end; + coder->next = LZMA_NEXT_CODER_INIT; + } + + // Validate the options. + if (lzma_delta_coder_memusage(filters[0].options) == UINT64_MAX) + return LZMA_OPTIONS_ERROR; + + // Set the delta distance. + const lzma_options_delta *opt = filters[0].options; + coder->distance = opt->dist; + + // Initialize the rest of the variables. + coder->pos = 0; + memzero(coder->history, LZMA_DELTA_DIST_MAX); + + // Initialize the next decoder in the chain, if any. + return lzma_next_filter_init(&coder->next, allocator, filters + 1); +} + + +extern uint64_t +lzma_delta_coder_memusage(const void *options) +{ + const lzma_options_delta *opt = options; + + if (opt == NULL || opt->type != LZMA_DELTA_TYPE_BYTE + || opt->dist < LZMA_DELTA_DIST_MIN + || opt->dist > LZMA_DELTA_DIST_MAX) + return UINT64_MAX; + + return sizeof(lzma_delta_coder); +} diff --git a/src/liblzma/delta/delta_common.h b/src/liblzma/delta/delta_common.h new file mode 100644 index 0000000000..bd09127697 --- /dev/null +++ b/src/liblzma/delta/delta_common.h @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_common.h +/// \brief Common stuff for Delta encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_DELTA_COMMON_H +#define LZMA_DELTA_COMMON_H + +#include "common.h" + +extern uint64_t lzma_delta_coder_memusage(const void *options); + +#endif diff --git a/src/liblzma/delta/delta_decoder.c b/src/liblzma/delta/delta_decoder.c new file mode 100644 index 0000000000..9f0d49ca41 --- /dev/null +++ b/src/liblzma/delta/delta_decoder.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_decoder.c +/// \brief Delta filter decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "delta_decoder.h" +#include "delta_private.h" + + +static void +decode_buffer(lzma_delta_coder *coder, uint8_t *buffer, size_t size) +{ + const size_t distance = coder->distance; + + for (size_t i = 0; i < size; ++i) { + buffer[i] += coder->history[(distance + coder->pos) & 0xFF]; + coder->history[coder->pos-- & 0xFF] = buffer[i]; + } +} + + +// For an unknown reason NVIDIA HPC Compiler needs this pragma +// to produce working code. +#ifdef __NVCOMPILER +# pragma routine novector +#endif +static lzma_ret +delta_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_delta_coder *coder = coder_ptr; + + assert(coder->next.code != NULL); + + const size_t out_start = *out_pos; + + const lzma_ret ret = coder->next.code(coder->next.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + + // out might be NULL. In that case size == 0. Null pointer + 0 is + // undefined behavior so skip the call in that case as it would + // do nothing anyway. + const size_t size = *out_pos - out_start; + if (size > 0) + decode_buffer(coder, out + out_start, size); + + return ret; +} + + +extern lzma_ret +lzma_delta_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + next->code = &delta_decode; + return lzma_delta_coder_init(next, allocator, filters); +} + + +extern lzma_ret +lzma_delta_props_decode(void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size) +{ + if (props_size != 1) + return LZMA_OPTIONS_ERROR; + + lzma_options_delta *opt + = lzma_alloc(sizeof(lzma_options_delta), allocator); + if (opt == NULL) + return LZMA_MEM_ERROR; + + opt->type = LZMA_DELTA_TYPE_BYTE; + opt->dist = props[0] + 1U; + + *options = opt; + + return LZMA_OK; +} diff --git a/src/liblzma/delta/delta_decoder.h b/src/liblzma/delta/delta_decoder.h new file mode 100644 index 0000000000..e2268ed44e --- /dev/null +++ b/src/liblzma/delta/delta_decoder.h @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_decoder.h +/// \brief Delta filter decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_DELTA_DECODER_H +#define LZMA_DELTA_DECODER_H + +#include "delta_common.h" + +extern lzma_ret lzma_delta_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_delta_props_decode( + void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size); + +#endif diff --git a/src/liblzma/delta/delta_encoder.c b/src/liblzma/delta/delta_encoder.c new file mode 100644 index 0000000000..ba4a50b1f4 --- /dev/null +++ b/src/liblzma/delta/delta_encoder.c @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_encoder.c +/// \brief Delta filter encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "delta_encoder.h" +#include "delta_private.h" + + +/// Copies and encodes the data at the same time. This is used when Delta +/// is the first filter in the chain (and thus the last filter in the +/// encoder's filter stack). +static void +copy_and_encode(lzma_delta_coder *coder, + const uint8_t *restrict in, uint8_t *restrict out, size_t size) +{ + const size_t distance = coder->distance; + + for (size_t i = 0; i < size; ++i) { + const uint8_t tmp = coder->history[ + (distance + coder->pos) & 0xFF]; + coder->history[coder->pos-- & 0xFF] = in[i]; + out[i] = in[i] - tmp; + } +} + + +/// Encodes the data in place. This is used when we are the last filter +/// in the chain (and thus non-last filter in the encoder's filter stack). +static void +encode_in_place(lzma_delta_coder *coder, uint8_t *buffer, size_t size) +{ + const size_t distance = coder->distance; + + for (size_t i = 0; i < size; ++i) { + const uint8_t tmp = coder->history[ + (distance + coder->pos) & 0xFF]; + coder->history[coder->pos-- & 0xFF] = buffer[i]; + buffer[i] -= tmp; + } +} + + +static lzma_ret +delta_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_delta_coder *coder = coder_ptr; + + lzma_ret ret; + + if (coder->next.code == NULL) { + const size_t in_avail = in_size - *in_pos; + const size_t out_avail = out_size - *out_pos; + const size_t size = my_min(in_avail, out_avail); + + // in and out might be NULL. In such cases size == 0. + // Null pointer + 0 is undefined behavior so skip + // the call in that case as it would do nothing anyway. + if (size > 0) + copy_and_encode(coder, in + *in_pos, out + *out_pos, + size); + + *in_pos += size; + *out_pos += size; + + ret = action != LZMA_RUN && *in_pos == in_size + ? LZMA_STREAM_END : LZMA_OK; + + } else { + const size_t out_start = *out_pos; + + ret = coder->next.code(coder->next.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + + // Like above, avoid null pointer + 0. + const size_t size = *out_pos - out_start; + if (size > 0) + encode_in_place(coder, out + out_start, size); + } + + return ret; +} + + +static lzma_ret +delta_encoder_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters_null lzma_attribute((__unused__)), + const lzma_filter *reversed_filters) +{ + lzma_delta_coder *coder = coder_ptr; + + // Delta doesn't and will never support changing the options in + // the middle of encoding. If the app tries to change them, we + // simply ignore them. + return lzma_next_filter_update( + &coder->next, allocator, reversed_filters + 1); +} + + +extern lzma_ret +lzma_delta_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + next->code = &delta_encode; + next->update = &delta_encoder_update; + return lzma_delta_coder_init(next, allocator, filters); +} + + +extern lzma_ret +lzma_delta_props_encode(const void *options, uint8_t *out) +{ + // The caller must have already validated the options, so it's + // LZMA_PROG_ERROR if they are invalid. + if (lzma_delta_coder_memusage(options) == UINT64_MAX) + return LZMA_PROG_ERROR; + + const lzma_options_delta *opt = options; + out[0] = opt->dist - LZMA_DELTA_DIST_MIN; + + return LZMA_OK; +} diff --git a/src/liblzma/delta/delta_encoder.h b/src/liblzma/delta/delta_encoder.h new file mode 100644 index 0000000000..735f0ed009 --- /dev/null +++ b/src/liblzma/delta/delta_encoder.h @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_encoder.h +/// \brief Delta filter encoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_DELTA_ENCODER_H +#define LZMA_DELTA_ENCODER_H + +#include "delta_common.h" + +extern lzma_ret lzma_delta_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_delta_props_encode(const void *options, uint8_t *out); + +#endif diff --git a/src/liblzma/delta/delta_private.h b/src/liblzma/delta/delta_private.h new file mode 100644 index 0000000000..e54721a846 --- /dev/null +++ b/src/liblzma/delta/delta_private.h @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file delta_private.h +/// \brief Private common stuff for Delta encoder and decoder +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_DELTA_PRIVATE_H +#define LZMA_DELTA_PRIVATE_H + +#include "delta_common.h" + +typedef struct { + /// Next coder in the chain + lzma_next_coder next; + + /// Delta distance + size_t distance; + + /// Position in history[] + uint8_t pos; + + /// Buffer to hold history of the original data + uint8_t history[LZMA_DELTA_DIST_MAX]; +} lzma_delta_coder; + + +extern lzma_ret lzma_delta_coder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters); + +#endif diff --git a/src/liblzma/liblzma.pc.in b/src/liblzma/liblzma.pc.in new file mode 100644 index 0000000000..a432992b70 --- /dev/null +++ b/src/liblzma/liblzma.pc.in @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: 0BSD +# Author: Lasse Collin + +prefix=@prefix@ +exec_prefix=@exec_prefix@ +libdir=@libdir@ +includedir=@includedir@ + +Name: liblzma +Description: General purpose data compression library +URL: @PACKAGE_URL@ +Version: @PACKAGE_VERSION@ +Cflags: -I${includedir} +Cflags.private: -DLZMA_API_STATIC +Libs: -L${libdir} -llzma +Libs.private: @PTHREAD_CFLAGS@ @LIBS@ diff --git a/src/liblzma/liblzma_generic.map b/src/liblzma/liblzma_generic.map new file mode 100644 index 0000000000..2bef27a8f7 --- /dev/null +++ b/src/liblzma/liblzma_generic.map @@ -0,0 +1,138 @@ +/* SPDX-License-Identifier: 0BSD */ + +XZ_5.0 { +global: + lzma_alone_decoder; + lzma_alone_encoder; + lzma_auto_decoder; + lzma_block_buffer_bound; + lzma_block_buffer_decode; + lzma_block_buffer_encode; + lzma_block_compressed_size; + lzma_block_decoder; + lzma_block_encoder; + lzma_block_header_decode; + lzma_block_header_encode; + lzma_block_header_size; + lzma_block_total_size; + lzma_block_unpadded_size; + lzma_check_is_supported; + lzma_check_size; + lzma_code; + lzma_crc32; + lzma_crc64; + lzma_easy_buffer_encode; + lzma_easy_decoder_memusage; + lzma_easy_encoder; + lzma_easy_encoder_memusage; + lzma_end; + lzma_filter_decoder_is_supported; + lzma_filter_encoder_is_supported; + lzma_filter_flags_decode; + lzma_filter_flags_encode; + lzma_filter_flags_size; + lzma_filters_copy; + lzma_filters_update; + lzma_get_check; + lzma_index_append; + lzma_index_block_count; + lzma_index_buffer_decode; + lzma_index_buffer_encode; + lzma_index_cat; + lzma_index_checks; + lzma_index_decoder; + lzma_index_dup; + lzma_index_encoder; + lzma_index_end; + lzma_index_file_size; + lzma_index_hash_append; + lzma_index_hash_decode; + lzma_index_hash_end; + lzma_index_hash_init; + lzma_index_hash_size; + lzma_index_init; + lzma_index_iter_init; + lzma_index_iter_locate; + lzma_index_iter_next; + lzma_index_iter_rewind; + lzma_index_memusage; + lzma_index_memused; + lzma_index_size; + lzma_index_stream_count; + lzma_index_stream_flags; + lzma_index_stream_padding; + lzma_index_stream_size; + lzma_index_total_size; + lzma_index_uncompressed_size; + lzma_lzma_preset; + lzma_memlimit_get; + lzma_memlimit_set; + lzma_memusage; + lzma_mf_is_supported; + lzma_mode_is_supported; + lzma_physmem; + lzma_properties_decode; + lzma_properties_encode; + lzma_properties_size; + lzma_raw_buffer_decode; + lzma_raw_buffer_encode; + lzma_raw_decoder; + lzma_raw_decoder_memusage; + lzma_raw_encoder; + lzma_raw_encoder_memusage; + lzma_stream_buffer_bound; + lzma_stream_buffer_decode; + lzma_stream_buffer_encode; + lzma_stream_decoder; + lzma_stream_encoder; + lzma_stream_flags_compare; + lzma_stream_footer_decode; + lzma_stream_footer_encode; + lzma_stream_header_decode; + lzma_stream_header_encode; + lzma_version_number; + lzma_version_string; + lzma_vli_decode; + lzma_vli_encode; + lzma_vli_size; + +local: + *; +}; + +XZ_5.2 { +global: + lzma_block_uncomp_encode; + lzma_cputhreads; + lzma_get_progress; + lzma_stream_encoder_mt; + lzma_stream_encoder_mt_memusage; +} XZ_5.0; + +XZ_5.4 { +global: + lzma_file_info_decoder; + lzma_filters_free; + lzma_lzip_decoder; + lzma_microlzma_decoder; + lzma_microlzma_encoder; + lzma_stream_decoder_mt; + lzma_str_from_filters; + lzma_str_list_filters; + lzma_str_to_filters; +} XZ_5.2; + +XZ_5.6.0 { +global: + lzma_mt_block_size; +} XZ_5.4; + +XZ_5.8 { +global: + lzma_bcj_arm64_encode; + lzma_bcj_arm64_decode; + lzma_bcj_riscv_encode; + lzma_bcj_riscv_decode; + lzma_bcj_x86_encode; + lzma_bcj_x86_decode; +} XZ_5.6.0; diff --git a/src/liblzma/liblzma_linux.map b/src/liblzma/liblzma_linux.map new file mode 100644 index 0000000000..50f1571de2 --- /dev/null +++ b/src/liblzma/liblzma_linux.map @@ -0,0 +1,153 @@ +/* SPDX-License-Identifier: 0BSD */ + +XZ_5.0 { +global: + lzma_alone_decoder; + lzma_alone_encoder; + lzma_auto_decoder; + lzma_block_buffer_bound; + lzma_block_buffer_decode; + lzma_block_buffer_encode; + lzma_block_compressed_size; + lzma_block_decoder; + lzma_block_encoder; + lzma_block_header_decode; + lzma_block_header_encode; + lzma_block_header_size; + lzma_block_total_size; + lzma_block_unpadded_size; + lzma_check_is_supported; + lzma_check_size; + lzma_code; + lzma_crc32; + lzma_crc64; + lzma_easy_buffer_encode; + lzma_easy_decoder_memusage; + lzma_easy_encoder; + lzma_easy_encoder_memusage; + lzma_end; + lzma_filter_decoder_is_supported; + lzma_filter_encoder_is_supported; + lzma_filter_flags_decode; + lzma_filter_flags_encode; + lzma_filter_flags_size; + lzma_filters_copy; + lzma_filters_update; + lzma_get_check; + lzma_index_append; + lzma_index_block_count; + lzma_index_buffer_decode; + lzma_index_buffer_encode; + lzma_index_cat; + lzma_index_checks; + lzma_index_decoder; + lzma_index_dup; + lzma_index_encoder; + lzma_index_end; + lzma_index_file_size; + lzma_index_hash_append; + lzma_index_hash_decode; + lzma_index_hash_end; + lzma_index_hash_init; + lzma_index_hash_size; + lzma_index_init; + lzma_index_iter_init; + lzma_index_iter_locate; + lzma_index_iter_next; + lzma_index_iter_rewind; + lzma_index_memusage; + lzma_index_memused; + lzma_index_size; + lzma_index_stream_count; + lzma_index_stream_flags; + lzma_index_stream_padding; + lzma_index_stream_size; + lzma_index_total_size; + lzma_index_uncompressed_size; + lzma_lzma_preset; + lzma_memlimit_get; + lzma_memlimit_set; + lzma_memusage; + lzma_mf_is_supported; + lzma_mode_is_supported; + lzma_physmem; + lzma_properties_decode; + lzma_properties_encode; + lzma_properties_size; + lzma_raw_buffer_decode; + lzma_raw_buffer_encode; + lzma_raw_decoder; + lzma_raw_decoder_memusage; + lzma_raw_encoder; + lzma_raw_encoder_memusage; + lzma_stream_buffer_bound; + lzma_stream_buffer_decode; + lzma_stream_buffer_encode; + lzma_stream_decoder; + lzma_stream_encoder; + lzma_stream_flags_compare; + lzma_stream_footer_decode; + lzma_stream_footer_encode; + lzma_stream_header_decode; + lzma_stream_header_encode; + lzma_version_number; + lzma_version_string; + lzma_vli_decode; + lzma_vli_encode; + lzma_vli_size; + +local: + *; +}; + +XZ_5.2 { +global: + lzma_block_uncomp_encode; + lzma_cputhreads; + lzma_get_progress; + lzma_stream_encoder_mt; + lzma_stream_encoder_mt_memusage; +} XZ_5.0; + +XZ_5.1.2alpha { +global: + lzma_stream_encoder_mt; + lzma_stream_encoder_mt_memusage; +} XZ_5.0; + +XZ_5.2.2 { +global: + lzma_block_uncomp_encode; + lzma_cputhreads; + lzma_get_progress; + lzma_stream_encoder_mt; + lzma_stream_encoder_mt_memusage; +} XZ_5.1.2alpha; + +XZ_5.4 { +global: + lzma_file_info_decoder; + lzma_filters_free; + lzma_lzip_decoder; + lzma_microlzma_decoder; + lzma_microlzma_encoder; + lzma_stream_decoder_mt; + lzma_str_from_filters; + lzma_str_list_filters; + lzma_str_to_filters; +} XZ_5.2; + +XZ_5.6.0 { +global: + lzma_mt_block_size; +} XZ_5.4; + +XZ_5.8 { +global: + lzma_bcj_arm64_encode; + lzma_bcj_arm64_decode; + lzma_bcj_riscv_encode; + lzma_bcj_riscv_decode; + lzma_bcj_x86_encode; + lzma_bcj_x86_decode; +} XZ_5.6.0; diff --git a/src/liblzma/liblzma_w32res.rc b/src/liblzma/liblzma_w32res.rc new file mode 100644 index 0000000000..19507b6e96 --- /dev/null +++ b/src/liblzma/liblzma_w32res.rc @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: 0BSD */ + +/* + * Author: Lasse Collin + */ + +#define MY_TYPE VFT_DLL + +#if defined(__MSYS__) +# define MY_NAME "msys-lzma-5" +#elif defined(__CYGWIN__) +# define MY_NAME "cyglzma-5" +#else +# define MY_NAME "liblzma" +#endif + +#define MY_SUFFIX ".dll" +#define MY_DESC "liblzma data compression library" +#include "common_w32res.rc" diff --git a/src/liblzma/lz/Makefile.inc b/src/liblzma/lz/Makefile.inc new file mode 100644 index 0000000000..15235d7d10 --- /dev/null +++ b/src/liblzma/lz/Makefile.inc @@ -0,0 +1,18 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +if COND_ENCODER_LZ +liblzma_la_SOURCES += \ + lz/lz_encoder.c \ + lz/lz_encoder.h \ + lz/lz_encoder_hash.h \ + lz/lz_encoder_hash_table.h \ + lz/lz_encoder_mf.c +endif + + +if COND_DECODER_LZ +liblzma_la_SOURCES += \ + lz/lz_decoder.c \ + lz/lz_decoder.h +endif diff --git a/src/liblzma/lz/lz_decoder.c b/src/liblzma/lz/lz_decoder.c new file mode 100644 index 0000000000..1cb120ab3b --- /dev/null +++ b/src/liblzma/lz/lz_decoder.c @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lz_decoder.c +/// \brief LZ out window +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +// liblzma supports multiple LZ77-based filters. The LZ part is shared +// between these filters. The LZ code takes care of dictionary handling +// and passing the data between filters in the chain. The filter-specific +// part decodes from the input buffer to the dictionary. + + +#include "lz_decoder.h" + + +typedef struct { + /// Dictionary (history buffer) + lzma_dict dict; + + /// The actual LZ-based decoder e.g. LZMA + lzma_lz_decoder lz; + + /// Next filter in the chain, if any. Note that LZMA and LZMA2 are + /// only allowed as the last filter, but the long-range filter in + /// future can be in the middle of the chain. + lzma_next_coder next; + + /// True if the next filter in the chain has returned LZMA_STREAM_END. + bool next_finished; + + /// True if the LZ decoder (e.g. LZMA) has detected end of payload + /// marker. This may become true before next_finished becomes true. + bool this_finished; + + /// Temporary buffer needed when the LZ-based filter is not the last + /// filter in the chain. The output of the next filter is first + /// decoded into buffer[], which is then used as input for the actual + /// LZ-based decoder. + struct { + size_t pos; + size_t size; + uint8_t buffer[LZMA_BUFFER_SIZE]; + } temp; +} lzma_coder; + + +static void +lz_decoder_reset(lzma_coder *coder) +{ + coder->dict.pos = LZ_DICT_INIT_POS; + coder->dict.full = 0; + coder->dict.buf[LZ_DICT_INIT_POS - 1] = '\0'; + coder->dict.has_wrapped = false; + coder->dict.need_reset = false; + return; +} + + +static lzma_ret +decode_buffer(lzma_coder *coder, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size) +{ + while (true) { + // Wrap the dictionary if needed. + if (coder->dict.pos == coder->dict.size) { + // See the comment of #define LZ_DICT_REPEAT_MAX. + coder->dict.pos = LZ_DICT_REPEAT_MAX; + coder->dict.has_wrapped = true; + memcpy(coder->dict.buf, coder->dict.buf + + coder->dict.size + - LZ_DICT_REPEAT_MAX, + LZ_DICT_REPEAT_MAX); + } + + // Store the current dictionary position. It is needed to know + // where to start copying to the out[] buffer. + const size_t dict_start = coder->dict.pos; + + // Calculate how much we allow coder->lz.code() to decode. + // It must not decode past the end of the dictionary + // buffer, and we don't want it to decode more than is + // actually needed to fill the out[] buffer. + coder->dict.limit = coder->dict.pos + + my_min(out_size - *out_pos, + coder->dict.size - coder->dict.pos); + + // Call the coder->lz.code() to do the actual decoding. + const lzma_ret ret = coder->lz.code( + coder->lz.coder, &coder->dict, + in, in_pos, in_size); + + // Copy the decoded data from the dictionary to the out[] + // buffer. Do it conditionally because out can be NULL + // (in which case copy_size is always 0). Calling memcpy() + // with a null-pointer is undefined even if the third + // argument is 0. + const size_t copy_size = coder->dict.pos - dict_start; + assert(copy_size <= out_size - *out_pos); + + if (copy_size > 0) + memcpy(out + *out_pos, coder->dict.buf + dict_start, + copy_size); + + *out_pos += copy_size; + + // Reset the dictionary if so requested by coder->lz.code(). + if (coder->dict.need_reset) { + lz_decoder_reset(coder); + + // Since we reset dictionary, we don't check if + // dictionary became full. + if (ret != LZMA_OK || *out_pos == out_size) + return ret; + } else { + // Return if everything got decoded or an error + // occurred, or if there's no more data to decode. + // + // Note that detecting if there's something to decode + // is done by looking if dictionary become full + // instead of looking if *in_pos == in_size. This + // is because it is possible that all the input was + // consumed already but some data is pending to be + // written to the dictionary. + if (ret != LZMA_OK || *out_pos == out_size + || coder->dict.pos < coder->dict.size) + return ret; + } + } +} + + +static lzma_ret +lz_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, + lzma_action action) +{ + lzma_coder *coder = coder_ptr; + + if (coder->next.code == NULL) + return decode_buffer(coder, in, in_pos, in_size, + out, out_pos, out_size); + + // We aren't the last coder in the chain, we need to decode + // our input to a temporary buffer. + while (*out_pos < out_size) { + // Fill the temporary buffer if it is empty. + if (!coder->next_finished + && coder->temp.pos == coder->temp.size) { + coder->temp.pos = 0; + coder->temp.size = 0; + + const lzma_ret ret = coder->next.code( + coder->next.coder, + allocator, in, in_pos, in_size, + coder->temp.buffer, &coder->temp.size, + LZMA_BUFFER_SIZE, action); + + if (ret == LZMA_STREAM_END) + coder->next_finished = true; + else if (ret != LZMA_OK || coder->temp.size == 0) + return ret; + } + + if (coder->this_finished) { + if (coder->temp.size != 0) + return LZMA_DATA_ERROR; + + if (coder->next_finished) + return LZMA_STREAM_END; + + return LZMA_OK; + } + + const lzma_ret ret = decode_buffer(coder, coder->temp.buffer, + &coder->temp.pos, coder->temp.size, + out, out_pos, out_size); + + if (ret == LZMA_STREAM_END) + coder->this_finished = true; + else if (ret != LZMA_OK) + return ret; + else if (coder->next_finished && *out_pos < out_size) + return LZMA_DATA_ERROR; + } + + return LZMA_OK; +} + + +static void +lz_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_coder *coder = coder_ptr; + + lzma_next_end(&coder->next, allocator); + lzma_free(coder->dict.buf, allocator); + + if (coder->lz.end != NULL) + coder->lz.end(coder->lz.coder, allocator); + else + lzma_free(coder->lz.coder, allocator); + + lzma_free(coder, allocator); + return; +} + + +extern lzma_ret +lzma_lz_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, + lzma_ret (*lz_init)(lzma_lz_decoder *lz, + const lzma_allocator *allocator, + lzma_vli id, const void *options, + lzma_lz_options *lz_options)) +{ + // Allocate the base structure if it isn't already allocated. + lzma_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &lz_decode; + next->end = &lz_decoder_end; + + coder->dict.buf = NULL; + coder->dict.size = 0; + coder->lz = LZMA_LZ_DECODER_INIT; + coder->next = LZMA_NEXT_CODER_INIT; + } + + // Allocate and initialize the LZ-based decoder. It will also give + // us the dictionary size. + lzma_lz_options lz_options; + return_if_error(lz_init(&coder->lz, allocator, + filters[0].id, filters[0].options, &lz_options)); + + // If the dictionary size is very small, increase it to 4096 bytes. + // This is to prevent constant wrapping of the dictionary, which + // would slow things down. The downside is that since we don't check + // separately for the real dictionary size, we may happily accept + // corrupt files. + if (lz_options.dict_size < 4096) + lz_options.dict_size = 4096; + + // Make dictionary size a multiple of 16. Some LZ-based decoders like + // LZMA use the lowest bits lzma_dict.pos to know the alignment of the + // data. Aligned buffer is also good when memcpying from the + // dictionary to the output buffer, since applications are + // recommended to give aligned buffers to liblzma. + // + // Reserve 2 * LZ_DICT_REPEAT_MAX bytes of extra space which is + // needed for alloc_size. Reserve also LZ_DICT_EXTRA bytes of extra + // space which is *not* counted in alloc_size or coder->dict.size. + // + // Avoid integer overflow. + if (lz_options.dict_size > SIZE_MAX - 15 - 2 * LZ_DICT_REPEAT_MAX + - LZ_DICT_EXTRA) + return LZMA_MEM_ERROR; + + lz_options.dict_size = (lz_options.dict_size + 15) & ~((size_t)(15)); + + // Reserve extra space as explained in the comment + // of #define LZ_DICT_REPEAT_MAX. + const size_t alloc_size + = lz_options.dict_size + 2 * LZ_DICT_REPEAT_MAX; + + // Allocate and initialize the dictionary. + if (coder->dict.size != alloc_size) { + lzma_free(coder->dict.buf, allocator); + + // The LZ_DICT_EXTRA bytes at the end of the buffer aren't + // included in alloc_size. These extra bytes allow + // dict_repeat() to read and write more data than requested. + // Otherwise this extra space is ignored. + coder->dict.buf = lzma_alloc(alloc_size + LZ_DICT_EXTRA, + allocator); + if (coder->dict.buf == NULL) + return LZMA_MEM_ERROR; + + // NOTE: Yes, alloc_size, not lz_options.dict_size. The way + // coder->dict.full is updated will take care that we will + // still reject distances larger than lz_options.dict_size. + coder->dict.size = alloc_size; + } + + lz_decoder_reset(next->coder); + + // Use the preset dictionary if it was given to us. + if (lz_options.preset_dict != NULL + && lz_options.preset_dict_size > 0) { + // If the preset dictionary is bigger than the actual + // dictionary, copy only the tail. + const size_t copy_size = my_min(lz_options.preset_dict_size, + lz_options.dict_size); + const size_t offset = lz_options.preset_dict_size - copy_size; + memcpy(coder->dict.buf + coder->dict.pos, + lz_options.preset_dict + offset, + copy_size); + + // dict.pos isn't zero after lz_decoder_reset(). + coder->dict.pos += copy_size; + coder->dict.full = copy_size; + } + + // Miscellaneous initializations + coder->next_finished = false; + coder->this_finished = false; + coder->temp.pos = 0; + coder->temp.size = 0; + + // Initialize the next filter in the chain, if any. + return lzma_next_filter_init(&coder->next, allocator, filters + 1); +} + + +extern uint64_t +lzma_lz_decoder_memusage(size_t dictionary_size) +{ + return sizeof(lzma_coder) + (uint64_t)(dictionary_size) + + 2 * LZ_DICT_REPEAT_MAX + LZ_DICT_EXTRA; +} diff --git a/src/liblzma/lz/lz_decoder.h b/src/liblzma/lz/lz_decoder.h new file mode 100644 index 0000000000..2698e0167f --- /dev/null +++ b/src/liblzma/lz/lz_decoder.h @@ -0,0 +1,325 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lz_decoder.h +/// \brief LZ out window +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZ_DECODER_H +#define LZMA_LZ_DECODER_H + +#include "common.h" + +#ifdef HAVE_IMMINTRIN_H +# include +#endif + + +// dict_repeat() implementation variant: +// 0 = Byte-by-byte copying only. +// 1 = Use memcpy() for non-overlapping copies. +// 2 = Use x86 SSE2 for non-overlapping copies. +#ifndef LZMA_LZ_DECODER_CONFIG +# if defined(TUKLIB_FAST_UNALIGNED_ACCESS) \ + && defined(HAVE_IMMINTRIN_H) \ + && (defined(__SSE2__) || defined(_M_X64) \ + || (defined(_M_IX86_FP) && _M_IX86_FP >= 2)) +# define LZMA_LZ_DECODER_CONFIG 2 +# else +# define LZMA_LZ_DECODER_CONFIG 1 +# endif +#endif + +/// Byte-by-byte and memcpy() copy exactly the amount needed. Other methods +/// can copy up to LZ_DICT_EXTRA bytes more than requested, and this amount +/// of extra space is needed at the end of the allocated dictionary buffer. +/// +/// NOTE: If this is increased, update LZMA_DICT_REPEAT_MAX too. +#if LZMA_LZ_DECODER_CONFIG >= 2 +# define LZ_DICT_EXTRA 32 +#else +# define LZ_DICT_EXTRA 0 +#endif + +/// Maximum number of bytes that dict_repeat() may copy. The allocated +/// dictionary buffer will be 2 * LZ_DICT_REPEAT_MAX + LZMA_DICT_EXTRA bytes +/// larger than the actual dictionary size: +/// +/// (1) Every time the decoder reaches the end of the dictionary buffer, +/// the last LZ_DICT_REPEAT_MAX bytes will be copied to the beginning. +/// This way dict_repeat() will only need to copy from one place, +/// never from both the end and beginning of the buffer. +/// +/// (2) The other LZ_DICT_REPEAT_MAX bytes is kept as a buffer between +/// the oldest byte still in the dictionary and the current write +/// position. This way dict_repeat() with the maximum valid distance +/// won't need memmove() as the copying cannot overlap. +/// +/// (3) LZ_DICT_EXTRA bytes are required at the end of the dictionary buffer +/// so that extra copying done by dict_repeat() won't write or read past +/// the end of the allocated buffer. This amount is *not* counted as part +/// of lzma_dict.size. +/// +/// Note that memcpy() still cannot be used if distance < len. +/// +/// LZMA's longest match length is 273 bytes. The LZMA decoder looks at +/// the lowest four bits of the dictionary position, thus 273 must be +/// rounded up to the next multiple of 16 (288). In addition, optimized +/// dict_repeat() copies 32 bytes at a time, thus this must also be +/// a multiple of 32. +#define LZ_DICT_REPEAT_MAX 288 + +/// Initial position in lzma_dict.buf when the dictionary is empty. +#define LZ_DICT_INIT_POS (2 * LZ_DICT_REPEAT_MAX) + + +typedef struct { + /// Pointer to the dictionary buffer. + uint8_t *buf; + + /// Write position in dictionary. The next byte will be written to + /// buf[pos]. + size_t pos; + + /// Indicates how full the dictionary is. This is used by + /// dict_is_distance_valid() to detect corrupt files that would + /// read beyond the beginning of the dictionary. + size_t full; + + /// Write limit + size_t limit; + + /// Allocated size of buf. This is 2 * LZ_DICT_REPEAT_MAX bytes + /// larger than the actual dictionary size. This is enforced by + /// how the value for "full" is set; it can be at most + /// "size - 2 * LZ_DICT_REPEAT_MAX". + size_t size; + + /// True once the dictionary has become full and the writing position + /// has been wrapped in decode_buffer() in lz_decoder.c. + bool has_wrapped; + + /// True when dictionary should be reset before decoding more data. + bool need_reset; + +} lzma_dict; + + +typedef struct { + size_t dict_size; + const uint8_t *preset_dict; + size_t preset_dict_size; +} lzma_lz_options; + + +typedef struct { + /// Data specific to the LZ-based decoder + void *coder; + + /// Function to decode from in[] to *dict + lzma_ret (*code)(void *coder, + lzma_dict *restrict dict, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size); + + void (*reset)(void *coder, const void *options); + + /// Set the uncompressed size. If uncompressed_size == LZMA_VLI_UNKNOWN + /// then allow_eopm will always be true. + void (*set_uncompressed)(void *coder, lzma_vli uncompressed_size, + bool allow_eopm); + + /// Free allocated resources + void (*end)(void *coder, const lzma_allocator *allocator); + +} lzma_lz_decoder; + + +#define LZMA_LZ_DECODER_INIT \ + (lzma_lz_decoder){ \ + .coder = NULL, \ + .code = NULL, \ + .reset = NULL, \ + .set_uncompressed = NULL, \ + .end = NULL, \ + } + + +extern lzma_ret lzma_lz_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters, + lzma_ret (*lz_init)(lzma_lz_decoder *lz, + const lzma_allocator *allocator, + lzma_vli id, const void *options, + lzma_lz_options *lz_options)); + +extern uint64_t lzma_lz_decoder_memusage(size_t dictionary_size); + + +////////////////////// +// Inline functions // +////////////////////// + +/// Get a byte from the history buffer. +static inline uint8_t +dict_get(const lzma_dict *const dict, const uint32_t distance) +{ + return dict->buf[dict->pos - distance - 1 + + (distance < dict->pos + ? 0 : dict->size - LZ_DICT_REPEAT_MAX)]; +} + + +/// Optimized version of dict_get(dict, 0) +static inline uint8_t +dict_get0(const lzma_dict *const dict) +{ + return dict->buf[dict->pos - 1]; +} + + +/// Test if dictionary is empty. +static inline bool +dict_is_empty(const lzma_dict *const dict) +{ + return dict->full == 0; +} + + +/// Validate the match distance +static inline bool +dict_is_distance_valid(const lzma_dict *const dict, const size_t distance) +{ + return dict->full > distance; +} + + +/// Repeat *len bytes at distance. +static inline bool +dict_repeat(lzma_dict *restrict dict, + uint32_t distance, uint32_t *restrict len) +{ + // Don't write past the end of the dictionary. + const size_t dict_avail = dict->limit - dict->pos; + uint32_t left = my_min(dict_avail, *len); + *len -= left; + + size_t back = dict->pos - distance - 1; + if (distance >= dict->pos) + back += dict->size - LZ_DICT_REPEAT_MAX; + +#if LZMA_LZ_DECODER_CONFIG == 0 + // Minimal byte-by-byte method. This might be the least bad choice + // if memcpy() isn't fast and there's no replacement for it below. + while (left-- > 0) { + dict->buf[dict->pos++] = dict->buf[back++]; + } + +#else + // Because memcpy() or a similar method can be faster than copying + // byte by byte in a loop, the copying process is split into + // two cases. + if (distance < left) { + // Source and target areas overlap, thus we can't use + // memcpy() nor even memmove() safely. + do { + dict->buf[dict->pos++] = dict->buf[back++]; + } while (--left > 0); + } else { +# if LZMA_LZ_DECODER_CONFIG == 1 + memcpy(dict->buf + dict->pos, dict->buf + back, left); + dict->pos += left; + +# elif LZMA_LZ_DECODER_CONFIG == 2 + // This can copy up to 32 bytes more than required. + // (If left == 0, we still copy 32 bytes.) + size_t pos = dict->pos; + dict->pos += left; + do { + const __m128i x0 = _mm_loadu_si128( + (__m128i *)(dict->buf + back)); + const __m128i x1 = _mm_loadu_si128( + (__m128i *)(dict->buf + back + 16)); + back += 32; + _mm_storeu_si128( + (__m128i *)(dict->buf + pos), x0); + _mm_storeu_si128( + (__m128i *)(dict->buf + pos + 16), x1); + pos += 32; + } while (pos < dict->pos); + +# else +# error "Invalid LZMA_LZ_DECODER_CONFIG value" +# endif + } +#endif + + // Update how full the dictionary is. + if (!dict->has_wrapped) + dict->full = dict->pos - LZ_DICT_INIT_POS; + + return *len != 0; +} + + +static inline void +dict_put(lzma_dict *restrict dict, uint8_t byte) +{ + dict->buf[dict->pos++] = byte; + + if (!dict->has_wrapped) + dict->full = dict->pos - LZ_DICT_INIT_POS; +} + + +/// Puts one byte into the dictionary. Returns true if the dictionary was +/// already full and the byte couldn't be added. +static inline bool +dict_put_safe(lzma_dict *restrict dict, uint8_t byte) +{ + if (unlikely(dict->pos == dict->limit)) + return true; + + dict_put(dict, byte); + return false; +} + + +/// Copies arbitrary amount of data into the dictionary. +static inline void +dict_write(lzma_dict *restrict dict, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size, + size_t *restrict left) +{ + // NOTE: If we are being given more data than the size of the + // dictionary, it could be possible to optimize the LZ decoder + // so that not everything needs to go through the dictionary. + // This shouldn't be very common thing in practice though, and + // the slowdown of one extra memcpy() isn't bad compared to how + // much time it would have taken if the data were compressed. + + if (in_size - *in_pos > *left) + in_size = *in_pos + *left; + + *left -= lzma_bufcpy(in, in_pos, in_size, + dict->buf, &dict->pos, dict->limit); + + if (!dict->has_wrapped) + dict->full = dict->pos - LZ_DICT_INIT_POS; + + return; +} + + +static inline void +dict_reset(lzma_dict *dict) +{ + dict->need_reset = true; + return; +} + +#endif diff --git a/src/liblzma/lz/lz_encoder.c b/src/liblzma/lz/lz_encoder.c new file mode 100644 index 0000000000..e5c4057dca --- /dev/null +++ b/src/liblzma/lz/lz_encoder.c @@ -0,0 +1,632 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lz_encoder.c +/// \brief LZ in window +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lz_encoder.h" +#include "lz_encoder_hash.h" + +// See lz_encoder_hash.h. This is a bit hackish but avoids making +// endianness a conditional in makefiles. +#ifdef LZMA_LZ_HASH_TABLE_IS_NEEDED +# include "lz_encoder_hash_table.h" +#endif + +#include "memcmplen.h" + + +typedef struct { + /// LZ-based encoder e.g. LZMA + lzma_lz_encoder lz; + + /// History buffer and match finder + lzma_mf mf; + + /// Next coder in the chain + lzma_next_coder next; +} lzma_coder; + + +/// \brief Moves the data in the input window to free space for new data +/// +/// mf->buffer is a sliding input window, which keeps mf->keep_size_before +/// bytes of input history available all the time. Now and then we need to +/// "slide" the buffer to make space for the new data to the end of the +/// buffer. At the same time, data older than keep_size_before is dropped. +/// +static void +move_window(lzma_mf *mf) +{ + // Align the move to a multiple of 16 bytes. Some LZ-based encoders + // like LZMA use the lowest bits of mf->read_pos to know the + // alignment of the uncompressed data. We also get better speed + // for memmove() with aligned buffers. + assert(mf->read_pos > mf->keep_size_before); + const uint32_t move_offset + = (mf->read_pos - mf->keep_size_before) & ~UINT32_C(15); + + assert(mf->write_pos > move_offset); + const size_t move_size = mf->write_pos - move_offset; + + assert(move_offset + move_size <= mf->size); + + memmove(mf->buffer, mf->buffer + move_offset, move_size); + + mf->offset += move_offset; + mf->read_pos -= move_offset; + mf->read_limit -= move_offset; + mf->write_pos -= move_offset; + + return; +} + + +/// \brief Tries to fill the input window (mf->buffer) +/// +/// If we are the last encoder in the chain, our input data is in in[]. +/// Otherwise we call the next filter in the chain to process in[] and +/// write its output to mf->buffer. +/// +/// This function must not be called once it has returned LZMA_STREAM_END. +/// +static lzma_ret +fill_window(lzma_coder *coder, const lzma_allocator *allocator, + const uint8_t *in, size_t *in_pos, size_t in_size, + lzma_action action) +{ + assert(coder->mf.read_pos <= coder->mf.write_pos); + + // Move the sliding window if needed. + if (coder->mf.read_pos >= coder->mf.size - coder->mf.keep_size_after) + move_window(&coder->mf); + + // Maybe this is ugly, but lzma_mf uses uint32_t for most things + // (which I find cleanest), but we need size_t here when filling + // the history window. + size_t write_pos = coder->mf.write_pos; + lzma_ret ret; + if (coder->next.code == NULL) { + // Not using a filter, simply memcpy() as much as possible. + lzma_bufcpy(in, in_pos, in_size, coder->mf.buffer, + &write_pos, coder->mf.size); + + ret = action != LZMA_RUN && *in_pos == in_size + ? LZMA_STREAM_END : LZMA_OK; + + } else { + ret = coder->next.code(coder->next.coder, allocator, + in, in_pos, in_size, + coder->mf.buffer, &write_pos, + coder->mf.size, action); + } + + coder->mf.write_pos = write_pos; + + // Silence Valgrind. lzma_memcmplen() can read extra bytes + // and Valgrind will give warnings if those bytes are uninitialized + // because Valgrind cannot see that the values of the uninitialized + // bytes are eventually ignored. + memzero(coder->mf.buffer + write_pos, LZMA_MEMCMPLEN_EXTRA); + + // If end of stream has been reached or flushing completed, we allow + // the encoder to process all the input (that is, read_pos is allowed + // to reach write_pos). Otherwise we keep keep_size_after bytes + // available as prebuffer. + if (ret == LZMA_STREAM_END) { + assert(*in_pos == in_size); + ret = LZMA_OK; + coder->mf.action = action; + coder->mf.read_limit = coder->mf.write_pos; + + } else if (coder->mf.write_pos > coder->mf.keep_size_after) { + // This needs to be done conditionally, because if we got + // only little new input, there may be too little input + // to do any encoding yet. + coder->mf.read_limit = coder->mf.write_pos + - coder->mf.keep_size_after; + } + + // Restart the match finder after finished LZMA_SYNC_FLUSH. + if (coder->mf.pending > 0 + && coder->mf.read_pos < coder->mf.read_limit) { + // Match finder may update coder->pending and expects it to + // start from zero, so use a temporary variable. + const uint32_t pending = coder->mf.pending; + coder->mf.pending = 0; + + // Rewind read_pos so that the match finder can hash + // the pending bytes. + assert(coder->mf.read_pos >= pending); + coder->mf.read_pos -= pending; + + // Call the skip function directly instead of using + // mf_skip(), since we don't want to touch mf->read_ahead. + coder->mf.skip(&coder->mf, pending); + } + + return ret; +} + + +static lzma_ret +lz_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, lzma_action action) +{ + lzma_coder *coder = coder_ptr; + + while (*out_pos < out_size + && (*in_pos < in_size || action != LZMA_RUN)) { + // Read more data to coder->mf.buffer if needed. + if (coder->mf.action == LZMA_RUN && coder->mf.read_pos + >= coder->mf.read_limit) + return_if_error(fill_window(coder, allocator, + in, in_pos, in_size, action)); + + // Encode + const lzma_ret ret = coder->lz.code(coder->lz.coder, + &coder->mf, out, out_pos, out_size); + if (ret != LZMA_OK) { + // Setting this to LZMA_RUN for cases when we are + // flushing. It doesn't matter when finishing or if + // an error occurred. + coder->mf.action = LZMA_RUN; + return ret; + } + } + + return LZMA_OK; +} + + +static bool +lz_encoder_prepare(lzma_mf *mf, const lzma_allocator *allocator, + const lzma_lz_options *lz_options) +{ + // For now, the dictionary size is limited to 1.5 GiB. This may grow + // in the future if needed, but it needs a little more work than just + // changing this check. + if (!IS_ENC_DICT_SIZE_VALID(lz_options->dict_size) + || lz_options->nice_len > lz_options->match_len_max) + return true; + + mf->keep_size_before = lz_options->before_size + lz_options->dict_size; + + mf->keep_size_after = lz_options->after_size + + lz_options->match_len_max; + + // To avoid constant memmove()s, allocate some extra space. Since + // memmove()s become more expensive when the size of the buffer + // increases, we reserve more space when a large dictionary is + // used to make the memmove() calls rarer. + // + // This works with dictionaries up to about 3 GiB. If bigger + // dictionary is wanted, some extra work is needed: + // - Several variables in lzma_mf have to be changed from uint32_t + // to size_t. + // - Memory usage calculation needs something too, e.g. use uint64_t + // for mf->size. + uint32_t reserve = lz_options->dict_size / 2; + if (reserve > (UINT32_C(1) << 30)) + reserve /= 2; + + reserve += (lz_options->before_size + lz_options->match_len_max + + lz_options->after_size) / 2 + (UINT32_C(1) << 19); + + const uint32_t old_size = mf->size; + mf->size = mf->keep_size_before + reserve + mf->keep_size_after; + + // Deallocate the old history buffer if it exists but has different + // size than what is needed now. + if (mf->buffer != NULL && old_size != mf->size) { + lzma_free(mf->buffer, allocator); + mf->buffer = NULL; + } + + // Match finder options + mf->match_len_max = lz_options->match_len_max; + mf->nice_len = lz_options->nice_len; + + // cyclic_size has to stay smaller than 2 Gi. Note that this doesn't + // mean limiting dictionary size to less than 2 GiB. With a match + // finder that uses multibyte resolution (hashes start at e.g. every + // fourth byte), cyclic_size would stay below 2 Gi even when + // dictionary size is greater than 2 GiB. + // + // It would be possible to allow cyclic_size >= 2 Gi, but then we + // would need to be careful to use 64-bit types in various places + // (size_t could do since we would need bigger than 32-bit address + // space anyway). It would also require either zeroing a multigigabyte + // buffer at initialization (waste of time and RAM) or allow + // normalization in lz_encoder_mf.c to access uninitialized + // memory to keep the code simpler. The current way is simple and + // still allows pretty big dictionaries, so I don't expect these + // limits to change. + mf->cyclic_size = lz_options->dict_size + 1; + + // Validate the match finder ID and setup the function pointers. + switch (lz_options->match_finder) { +#ifdef HAVE_MF_HC3 + case LZMA_MF_HC3: + mf->find = &lzma_mf_hc3_find; + mf->skip = &lzma_mf_hc3_skip; + break; +#endif +#ifdef HAVE_MF_HC4 + case LZMA_MF_HC4: + mf->find = &lzma_mf_hc4_find; + mf->skip = &lzma_mf_hc4_skip; + break; +#endif +#ifdef HAVE_MF_BT2 + case LZMA_MF_BT2: + mf->find = &lzma_mf_bt2_find; + mf->skip = &lzma_mf_bt2_skip; + break; +#endif +#ifdef HAVE_MF_BT3 + case LZMA_MF_BT3: + mf->find = &lzma_mf_bt3_find; + mf->skip = &lzma_mf_bt3_skip; + break; +#endif +#ifdef HAVE_MF_BT4 + case LZMA_MF_BT4: + mf->find = &lzma_mf_bt4_find; + mf->skip = &lzma_mf_bt4_skip; + break; +#endif + + default: + return true; + } + + // Calculate the sizes of mf->hash and mf->son. + // + // NOTE: Since 5.3.5beta the LZMA encoder ensures that nice_len + // is big enough for the selected match finder. This makes it + // easier for applications as nice_len = 2 will always be accepted + // even though the effective value can be slightly bigger. + const uint32_t hash_bytes + = mf_get_hash_bytes(lz_options->match_finder); + assert(hash_bytes <= mf->nice_len); + + const bool is_bt = (lz_options->match_finder & 0x10) != 0; + uint32_t hs; + + if (hash_bytes == 2) { + hs = 0xFFFF; + } else { + // Round dictionary size up to the next 2^n - 1 so it can + // be used as a hash mask. + hs = lz_options->dict_size - 1; + hs |= hs >> 1; + hs |= hs >> 2; + hs |= hs >> 4; + hs |= hs >> 8; + hs >>= 1; + hs |= 0xFFFF; + + if (hs > (UINT32_C(1) << 24)) { + if (hash_bytes == 3) + hs = (UINT32_C(1) << 24) - 1; + else + hs >>= 1; + } + } + + mf->hash_mask = hs; + + ++hs; + if (hash_bytes > 2) + hs += HASH_2_SIZE; + if (hash_bytes > 3) + hs += HASH_3_SIZE; +/* + No match finder uses this at the moment. + if (mf->hash_bytes > 4) + hs += HASH_4_SIZE; +*/ + + const uint32_t old_hash_count = mf->hash_count; + const uint32_t old_sons_count = mf->sons_count; + mf->hash_count = hs; + mf->sons_count = mf->cyclic_size; + if (is_bt) + mf->sons_count *= 2; + + // Deallocate the old hash array if it exists and has different size + // than what is needed now. + if (old_hash_count != mf->hash_count + || old_sons_count != mf->sons_count) { + lzma_free(mf->hash, allocator); + mf->hash = NULL; + + lzma_free(mf->son, allocator); + mf->son = NULL; + } + + // Maximum number of match finder cycles + mf->depth = lz_options->depth; + if (mf->depth == 0) { + if (is_bt) + mf->depth = 16 + mf->nice_len / 2; + else + mf->depth = 4 + mf->nice_len / 4; + } + + return false; +} + + +static bool +lz_encoder_init(lzma_mf *mf, const lzma_allocator *allocator, + const lzma_lz_options *lz_options) +{ + // Allocate the history buffer. + if (mf->buffer == NULL) { + // lzma_memcmplen() is used for the dictionary buffer + // so we need to allocate a few extra bytes to prevent + // it from reading past the end of the buffer. + mf->buffer = lzma_alloc(mf->size + LZMA_MEMCMPLEN_EXTRA, + allocator); + if (mf->buffer == NULL) + return true; + + // Keep Valgrind happy with lzma_memcmplen() and initialize + // the extra bytes whose value may get read but which will + // effectively get ignored. + memzero(mf->buffer + mf->size, LZMA_MEMCMPLEN_EXTRA); + } + + // Use cyclic_size as initial mf->offset. This allows + // avoiding a few branches in the match finders. The downside is + // that match finder needs to be normalized more often, which may + // hurt performance with huge dictionaries. + mf->offset = mf->cyclic_size; + mf->read_pos = 0; + mf->read_ahead = 0; + mf->read_limit = 0; + mf->write_pos = 0; + mf->pending = 0; + +#if UINT32_MAX >= SIZE_MAX / 4 + // Check for integer overflow. (Huge dictionaries are not + // possible on 32-bit CPU.) + if (mf->hash_count > SIZE_MAX / sizeof(uint32_t) + || mf->sons_count > SIZE_MAX / sizeof(uint32_t)) + return true; +#endif + + // Allocate and initialize the hash table. Since EMPTY_HASH_VALUE + // is zero, we can use lzma_alloc_zero() or memzero() for mf->hash. + // + // We don't need to initialize mf->son, but not doing that may + // make Valgrind complain in normalization (see normalize() in + // lz_encoder_mf.c). Skipping the initialization is *very* good + // when big dictionary is used but only small amount of data gets + // actually compressed: most of the mf->son won't get actually + // allocated by the kernel, so we avoid wasting RAM and improve + // initialization speed a lot. + if (mf->hash == NULL) { + mf->hash = lzma_alloc_zero(mf->hash_count * sizeof(uint32_t), + allocator); + mf->son = lzma_alloc(mf->sons_count * sizeof(uint32_t), + allocator); + + if (mf->hash == NULL || mf->son == NULL) { + lzma_free(mf->hash, allocator); + mf->hash = NULL; + + lzma_free(mf->son, allocator); + mf->son = NULL; + + return true; + } + } else { +/* + for (uint32_t i = 0; i < mf->hash_count; ++i) + mf->hash[i] = EMPTY_HASH_VALUE; +*/ + memzero(mf->hash, mf->hash_count * sizeof(uint32_t)); + } + + mf->cyclic_pos = 0; + + // Handle preset dictionary. + if (lz_options->preset_dict != NULL + && lz_options->preset_dict_size > 0) { + // If the preset dictionary is bigger than the actual + // dictionary, use only the tail. + mf->write_pos = my_min(lz_options->preset_dict_size, mf->size); + memcpy(mf->buffer, lz_options->preset_dict + + lz_options->preset_dict_size - mf->write_pos, + mf->write_pos); + mf->action = LZMA_SYNC_FLUSH; + mf->skip(mf, mf->write_pos); + } + + mf->action = LZMA_RUN; + + return false; +} + + +extern uint64_t +lzma_lz_encoder_memusage(const lzma_lz_options *lz_options) +{ + // Old buffers must not exist when calling lz_encoder_prepare(). + lzma_mf mf = { + .buffer = NULL, + .hash = NULL, + .son = NULL, + .hash_count = 0, + .sons_count = 0, + }; + + // Setup the size information into mf. + if (lz_encoder_prepare(&mf, NULL, lz_options)) + return UINT64_MAX; + + // Calculate the memory usage. + return ((uint64_t)(mf.hash_count) + mf.sons_count) * sizeof(uint32_t) + + mf.size + sizeof(lzma_coder); +} + + +static void +lz_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_coder *coder = coder_ptr; + + lzma_next_end(&coder->next, allocator); + + lzma_free(coder->mf.son, allocator); + lzma_free(coder->mf.hash, allocator); + lzma_free(coder->mf.buffer, allocator); + + if (coder->lz.end != NULL) + coder->lz.end(coder->lz.coder, allocator); + else + lzma_free(coder->lz.coder, allocator); + + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +lz_encoder_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters_null lzma_attribute((__unused__)), + const lzma_filter *reversed_filters) +{ + lzma_coder *coder = coder_ptr; + + if (coder->lz.options_update == NULL) + return LZMA_PROG_ERROR; + + return_if_error(coder->lz.options_update( + coder->lz.coder, reversed_filters)); + + return lzma_next_filter_update( + &coder->next, allocator, reversed_filters + 1); +} + + +static lzma_ret +lz_encoder_set_out_limit(void *coder_ptr, uint64_t *uncomp_size, + uint64_t out_limit) +{ + lzma_coder *coder = coder_ptr; + + // This is supported only if there are no other filters chained. + if (coder->next.code == NULL && coder->lz.set_out_limit != NULL) + return coder->lz.set_out_limit( + coder->lz.coder, uncomp_size, out_limit); + + return LZMA_OPTIONS_ERROR; +} + + +extern lzma_ret +lzma_lz_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, + lzma_ret (*lz_init)(lzma_lz_encoder *lz, + const lzma_allocator *allocator, + lzma_vli id, const void *options, + lzma_lz_options *lz_options)) +{ +#if defined(HAVE_SMALL) && !defined(HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR) + // The CRC32 table must be initialized. + lzma_crc32_init(); +#endif + + // Allocate and initialize the base data structure. + lzma_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &lz_encode; + next->end = &lz_encoder_end; + next->update = &lz_encoder_update; + next->set_out_limit = &lz_encoder_set_out_limit; + + coder->lz.coder = NULL; + coder->lz.code = NULL; + coder->lz.end = NULL; + coder->lz.options_update = NULL; + coder->lz.set_out_limit = NULL; + + // mf.size is initialized to silence Valgrind + // when used on optimized binaries (GCC may reorder + // code in a way that Valgrind gets unhappy). + coder->mf.buffer = NULL; + coder->mf.size = 0; + coder->mf.hash = NULL; + coder->mf.son = NULL; + coder->mf.hash_count = 0; + coder->mf.sons_count = 0; + + coder->next = LZMA_NEXT_CODER_INIT; + } + + // Initialize the LZ-based encoder. + lzma_lz_options lz_options; + return_if_error(lz_init(&coder->lz, allocator, + filters[0].id, filters[0].options, &lz_options)); + + // Setup the size information into coder->mf and deallocate + // old buffers if they have wrong size. + if (lz_encoder_prepare(&coder->mf, allocator, &lz_options)) + return LZMA_OPTIONS_ERROR; + + // Allocate new buffers if needed, and do the rest of + // the initialization. + if (lz_encoder_init(&coder->mf, allocator, &lz_options)) + return LZMA_MEM_ERROR; + + // Initialize the next filter in the chain, if any. + return lzma_next_filter_init(&coder->next, allocator, filters + 1); +} + + +extern LZMA_API(lzma_bool) +lzma_mf_is_supported(lzma_match_finder mf) +{ + switch (mf) { +#ifdef HAVE_MF_HC3 + case LZMA_MF_HC3: + return true; +#endif +#ifdef HAVE_MF_HC4 + case LZMA_MF_HC4: + return true; +#endif +#ifdef HAVE_MF_BT2 + case LZMA_MF_BT2: + return true; +#endif +#ifdef HAVE_MF_BT3 + case LZMA_MF_BT3: + return true; +#endif +#ifdef HAVE_MF_BT4 + case LZMA_MF_BT4: + return true; +#endif + default: + return false; + } +} diff --git a/src/liblzma/lz/lz_encoder.h b/src/liblzma/lz/lz_encoder.h new file mode 100644 index 0000000000..eb197c6b6c --- /dev/null +++ b/src/liblzma/lz/lz_encoder.h @@ -0,0 +1,352 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lz_encoder.h +/// \brief LZ in window and match finder API +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZ_ENCODER_H +#define LZMA_LZ_ENCODER_H + +#include "common.h" + + +// For now, the dictionary size is limited to 1.5 GiB. This may grow +// in the future if needed, but it needs a little more work than just +// changing this check. +#define IS_ENC_DICT_SIZE_VALID(size) \ + ((size) >= LZMA_DICT_SIZE_MIN \ + && (size) <= (UINT32_C(1) << 30) + (UINT32_C(1) << 29)) + + +/// A table of these is used by the LZ-based encoder to hold +/// the length-distance pairs found by the match finder. +typedef struct { + uint32_t len; + uint32_t dist; +} lzma_match; + + +typedef struct lzma_mf_s lzma_mf; +struct lzma_mf_s { + /////////////// + // In Window // + /////////////// + + /// Pointer to buffer with data to be compressed + uint8_t *buffer; + + /// Total size of the allocated buffer (that is, including all + /// the extra space) + uint32_t size; + + /// Number of bytes that must be kept available in our input history. + /// That is, once keep_size_before bytes have been processed, + /// buffer[read_pos - keep_size_before] is the oldest byte that + /// must be available for reading. + uint32_t keep_size_before; + + /// Number of bytes that must be kept in buffer after read_pos. + /// That is, read_pos <= write_pos - keep_size_after as long as + /// action is LZMA_RUN; when action != LZMA_RUN, read_pos is allowed + /// to reach write_pos so that the last bytes get encoded too. + uint32_t keep_size_after; + + /// Match finders store locations of matches using 32-bit integers. + /// To avoid adjusting several megabytes of integers every time the + /// input window is moved with move_window, we only adjust the + /// offset of the buffer. Thus, buffer[value_in_hash_table - offset] + /// is the byte pointed by value_in_hash_table. + uint32_t offset; + + /// buffer[read_pos] is the next byte to run through the match + /// finder. This is incremented in the match finder once the byte + /// has been processed. + uint32_t read_pos; + + /// Number of bytes that have been ran through the match finder, but + /// which haven't been encoded by the LZ-based encoder yet. + uint32_t read_ahead; + + /// As long as read_pos is less than read_limit, there is enough + /// input available in buffer for at least one encoding loop. + /// + /// Because of the stateful API, read_limit may and will get greater + /// than read_pos quite often. This is taken into account when + /// calculating the value for keep_size_after. + uint32_t read_limit; + + /// buffer[write_pos] is the first byte that doesn't contain valid + /// uncompressed data; that is, the next input byte will be copied + /// to buffer[write_pos]. + uint32_t write_pos; + + /// Number of bytes not hashed before read_pos. This is needed to + /// restart the match finder after LZMA_SYNC_FLUSH. + uint32_t pending; + + ////////////////// + // Match Finder // + ////////////////// + + /// Find matches. Returns the number of distance-length pairs written + /// to the matches array. This is called only via lzma_mf_find(). + uint32_t (*find)(lzma_mf *mf, lzma_match *matches); + + /// Skips num bytes. This is like find() but doesn't make the + /// distance-length pairs available, thus being a little faster. + /// This is called only via mf_skip(). + void (*skip)(lzma_mf *mf, uint32_t num); + + uint32_t *hash; + uint32_t *son; + uint32_t cyclic_pos; + uint32_t cyclic_size; // Must be dictionary size + 1. + uint32_t hash_mask; + + /// Maximum number of loops in the match finder + uint32_t depth; + + /// Maximum length of a match that the match finder will try to find. + uint32_t nice_len; + + /// Maximum length of a match supported by the LZ-based encoder. + /// If the longest match found by the match finder is nice_len, + /// mf_find() tries to expand it up to match_len_max bytes. + uint32_t match_len_max; + + /// When running out of input, binary tree match finders need to know + /// if it is due to flushing or finishing. The action is used also + /// by the LZ-based encoders themselves. + lzma_action action; + + /// Number of elements in hash[] + uint32_t hash_count; + + /// Number of elements in son[] + uint32_t sons_count; +}; + + +typedef struct { + /// Extra amount of data to keep available before the "actual" + /// dictionary. + size_t before_size; + + /// Size of the history buffer + size_t dict_size; + + /// Extra amount of data to keep available after the "actual" + /// dictionary. + size_t after_size; + + /// Maximum length of a match that the LZ-based encoder can accept. + /// This is used to extend matches of length nice_len to the + /// maximum possible length. + size_t match_len_max; + + /// Match finder will search matches up to this length. + /// This must be less than or equal to match_len_max. + size_t nice_len; + + /// Type of the match finder to use + lzma_match_finder match_finder; + + /// Maximum search depth + uint32_t depth; + + /// Initial dictionary for the match finder to search. + const uint8_t *preset_dict; + + /// If the preset dictionary is NULL, this value is ignored. + /// Otherwise this member must indicate the preset dictionary's + /// buffer size. If this size is larger than dict_size, then only + /// the dict_size sized tail of the preset_dict will be used. + uint32_t preset_dict_size; + +} lzma_lz_options; + + +// The total usable buffer space at any moment outside the match finder: +// before_size + dict_size + after_size + match_len_max +// +// In reality, there's some extra space allocated to prevent the number of +// memmove() calls reasonable. The bigger the dict_size is, the bigger +// this extra buffer will be since with bigger dictionaries memmove() would +// also take longer. +// +// A single encoder loop in the LZ-based encoder may call the match finder +// (mf_find() or mf_skip()) at most after_size times. In other words, +// a single encoder loop may increment lzma_mf.read_pos at most after_size +// times. Since matches are looked up to +// lzma_mf.buffer[lzma_mf.read_pos + match_len_max - 1], the total +// amount of extra buffer needed after dict_size becomes +// after_size + match_len_max. +// +// before_size has two uses. The first one is to keep literals available +// in cases when the LZ-based encoder has made some read ahead. +// TODO: Maybe this could be changed by making the LZ-based encoders to +// store the actual literals as they do with length-distance pairs. +// +// Algorithms such as LZMA2 first try to compress a chunk, and then check +// if the encoded result is smaller than the uncompressed one. If the chunk +// was incompressible, it is better to store it in uncompressed form in +// the output stream. To do this, the whole uncompressed chunk has to be +// still available in the history buffer. before_size achieves that. + + +typedef struct { + /// Data specific to the LZ-based encoder + void *coder; + + /// Function to encode from *dict to out[] + lzma_ret (*code)(void *coder, + lzma_mf *restrict mf, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size); + + /// Free allocated resources + void (*end)(void *coder, const lzma_allocator *allocator); + + /// Update the options in the middle of the encoding. + lzma_ret (*options_update)(void *coder, const lzma_filter *filter); + + /// Set maximum allowed output size + lzma_ret (*set_out_limit)(void *coder, uint64_t *uncomp_size, + uint64_t out_limit); + +} lzma_lz_encoder; + + +// Basic steps: +// 1. Input gets copied into the dictionary. +// 2. Data in dictionary gets run through the match finder byte by byte. +// 3. The literals and matches are encoded using e.g. LZMA. +// +// The bytes that have been ran through the match finder, but not encoded yet, +// are called 'read ahead'. + + +/// Get how many bytes the match finder hashes in its initial step. +/// This is also the minimum nice_len value with the match finder. +static inline uint32_t +mf_get_hash_bytes(lzma_match_finder match_finder) +{ + return (uint32_t)match_finder & 0x0F; +} + + +/// Get pointer to the first byte not ran through the match finder +static inline const uint8_t * +mf_ptr(const lzma_mf *mf) +{ + return mf->buffer + mf->read_pos; +} + + +/// Get the number of bytes that haven't been ran through the match finder yet. +static inline uint32_t +mf_avail(const lzma_mf *mf) +{ + return mf->write_pos - mf->read_pos; +} + + +/// Get the number of bytes that haven't been encoded yet (some of these +/// bytes may have been ran through the match finder though). +static inline uint32_t +mf_unencoded(const lzma_mf *mf) +{ + return mf->write_pos - mf->read_pos + mf->read_ahead; +} + + +/// Calculate the absolute offset from the beginning of the most recent +/// dictionary reset. Only the lowest four bits are important, so there's no +/// problem that we don't know the 64-bit size of the data encoded so far. +/// +/// NOTE: When moving the input window, we need to do it so that the lowest +/// bits of dict->read_pos are not modified to keep this macro working +/// as intended. +static inline uint32_t +mf_position(const lzma_mf *mf) +{ + return mf->read_pos - mf->read_ahead; +} + + +/// Since everything else begins with mf_, use it also for lzma_mf_find(). +#define mf_find lzma_mf_find + + +/// Skip the given number of bytes. This is used when a good match was found. +/// For example, if mf_find() finds a match of 200 bytes long, the first byte +/// of that match was already consumed by mf_find(), and the rest 199 bytes +/// have to be skipped with mf_skip(mf, 199). +static inline void +mf_skip(lzma_mf *mf, uint32_t amount) +{ + if (amount != 0) { + mf->skip(mf, amount); + mf->read_ahead += amount; + } +} + + +/// Copies at most *left number of bytes from the history buffer +/// to out[]. This is needed by LZMA2 to encode uncompressed chunks. +static inline void +mf_read(lzma_mf *mf, uint8_t *out, size_t *out_pos, size_t out_size, + size_t *left) +{ + const size_t out_avail = out_size - *out_pos; + const size_t copy_size = my_min(out_avail, *left); + + assert(mf->read_ahead == 0); + assert(mf->read_pos >= *left); + + memcpy(out + *out_pos, mf->buffer + mf->read_pos - *left, + copy_size); + + *out_pos += copy_size; + *left -= copy_size; + return; +} + + +extern lzma_ret lzma_lz_encoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, + lzma_ret (*lz_init)(lzma_lz_encoder *lz, + const lzma_allocator *allocator, + lzma_vli id, const void *options, + lzma_lz_options *lz_options)); + + +extern uint64_t lzma_lz_encoder_memusage(const lzma_lz_options *lz_options); + + +// These are only for LZ encoder's internal use. +extern uint32_t lzma_mf_find( + lzma_mf *mf, uint32_t *count, lzma_match *matches); + +extern uint32_t lzma_mf_hc3_find(lzma_mf *dict, lzma_match *matches); +extern void lzma_mf_hc3_skip(lzma_mf *dict, uint32_t amount); + +extern uint32_t lzma_mf_hc4_find(lzma_mf *dict, lzma_match *matches); +extern void lzma_mf_hc4_skip(lzma_mf *dict, uint32_t amount); + +extern uint32_t lzma_mf_bt2_find(lzma_mf *dict, lzma_match *matches); +extern void lzma_mf_bt2_skip(lzma_mf *dict, uint32_t amount); + +extern uint32_t lzma_mf_bt3_find(lzma_mf *dict, lzma_match *matches); +extern void lzma_mf_bt3_skip(lzma_mf *dict, uint32_t amount); + +extern uint32_t lzma_mf_bt4_find(lzma_mf *dict, lzma_match *matches); +extern void lzma_mf_bt4_skip(lzma_mf *dict, uint32_t amount); + +#endif diff --git a/src/liblzma/lz/lz_encoder_hash.h b/src/liblzma/lz/lz_encoder_hash.h new file mode 100644 index 0000000000..6d4bf837fd --- /dev/null +++ b/src/liblzma/lz/lz_encoder_hash.h @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lz_encoder_hash.h +/// \brief Hash macros for match finders +// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZ_ENCODER_HASH_H +#define LZMA_LZ_ENCODER_HASH_H + +// We need to know if CRC32_GENERIC is defined and we may need the declaration +// of lzma_crc32_table[][]. +#include "crc_common.h" + +// If HAVE_SMALL is defined, then lzma_crc32_table[][] exists and +// it's little endian even on big endian systems. +// +// If HAVE_SMALL isn't defined, lzma_crc32_table[][] is in native endian +// but we want a little endian one so that the compressed output won't +// depend on the processor endianness. Big endian systems are less common +// so those get the burden of an extra 1 KiB table. +// +// If HAVE_SMALL isn't defined and CRC32_GENERIC isn't defined either, +// then lzma_crc32_table[][] doesn't exist. +#if defined(HAVE_SMALL) \ + || (defined(CRC32_GENERIC) && !defined(WORDS_BIGENDIAN)) +# define hash_table lzma_crc32_table[0] +#else + // lz_encoder.c takes care of including the actual table. + lzma_attr_visibility_hidden + extern const uint32_t lzma_lz_hash_table[256]; +# define hash_table lzma_lz_hash_table +# define LZMA_LZ_HASH_TABLE_IS_NEEDED 1 +#endif + +#define HASH_2_SIZE (UINT32_C(1) << 10) +#define HASH_3_SIZE (UINT32_C(1) << 16) +#define HASH_4_SIZE (UINT32_C(1) << 20) + +#define HASH_2_MASK (HASH_2_SIZE - 1) +#define HASH_3_MASK (HASH_3_SIZE - 1) +#define HASH_4_MASK (HASH_4_SIZE - 1) + +#define FIX_3_HASH_SIZE (HASH_2_SIZE) +#define FIX_4_HASH_SIZE (HASH_2_SIZE + HASH_3_SIZE) +#define FIX_5_HASH_SIZE (HASH_2_SIZE + HASH_3_SIZE + HASH_4_SIZE) + +// Endianness doesn't matter in hash_2_calc() (no effect on the output). +#ifdef TUKLIB_FAST_UNALIGNED_ACCESS +# define hash_2_calc() \ + const uint32_t hash_value = read16ne(cur) +#else +# define hash_2_calc() \ + const uint32_t hash_value \ + = (uint32_t)(cur[0]) | ((uint32_t)(cur[1]) << 8) +#endif + +#define hash_3_calc() \ + const uint32_t temp = hash_table[cur[0]] ^ cur[1]; \ + const uint32_t hash_2_value = temp & HASH_2_MASK; \ + const uint32_t hash_value \ + = (temp ^ ((uint32_t)(cur[2]) << 8)) & mf->hash_mask + +#define hash_4_calc() \ + const uint32_t temp = hash_table[cur[0]] ^ cur[1]; \ + const uint32_t hash_2_value = temp & HASH_2_MASK; \ + const uint32_t hash_3_value \ + = (temp ^ ((uint32_t)(cur[2]) << 8)) & HASH_3_MASK; \ + const uint32_t hash_value = (temp ^ ((uint32_t)(cur[2]) << 8) \ + ^ (hash_table[cur[3]] << 5)) & mf->hash_mask + + +// The following are not currently used. + +#define hash_5_calc() \ + const uint32_t temp = hash_table[cur[0]] ^ cur[1]; \ + const uint32_t hash_2_value = temp & HASH_2_MASK; \ + const uint32_t hash_3_value \ + = (temp ^ ((uint32_t)(cur[2]) << 8)) & HASH_3_MASK; \ + uint32_t hash_4_value = (temp ^ ((uint32_t)(cur[2]) << 8) ^ \ + ^ hash_table[cur[3]] << 5); \ + const uint32_t hash_value \ + = (hash_4_value ^ (hash_table[cur[4]] << 3)) \ + & mf->hash_mask; \ + hash_4_value &= HASH_4_MASK + +/* +#define hash_zip_calc() \ + const uint32_t hash_value \ + = (((uint32_t)(cur[0]) | ((uint32_t)(cur[1]) << 8)) \ + ^ hash_table[cur[2]]) & 0xFFFF +*/ + +#define hash_zip_calc() \ + const uint32_t hash_value \ + = (((uint32_t)(cur[2]) | ((uint32_t)(cur[0]) << 8)) \ + ^ hash_table[cur[1]]) & 0xFFFF + +#define mt_hash_2_calc() \ + const uint32_t hash_2_value \ + = (hash_table[cur[0]] ^ cur[1]) & HASH_2_MASK + +#define mt_hash_3_calc() \ + const uint32_t temp = hash_table[cur[0]] ^ cur[1]; \ + const uint32_t hash_2_value = temp & HASH_2_MASK; \ + const uint32_t hash_3_value \ + = (temp ^ ((uint32_t)(cur[2]) << 8)) & HASH_3_MASK + +#define mt_hash_4_calc() \ + const uint32_t temp = hash_table[cur[0]] ^ cur[1]; \ + const uint32_t hash_2_value = temp & HASH_2_MASK; \ + const uint32_t hash_3_value \ + = (temp ^ ((uint32_t)(cur[2]) << 8)) & HASH_3_MASK; \ + const uint32_t hash_4_value = (temp ^ ((uint32_t)(cur[2]) << 8) ^ \ + (hash_table[cur[3]] << 5)) & HASH_4_MASK + +#endif diff --git a/src/liblzma/lz/lz_encoder_hash_table.h b/src/liblzma/lz/lz_encoder_hash_table.h new file mode 100644 index 0000000000..2b3a60e43e --- /dev/null +++ b/src/liblzma/lz/lz_encoder_hash_table.h @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by crc32_tablegen.c. + +const uint32_t lzma_lz_hash_table[256] = { + 0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, + 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3, + 0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, + 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91, + 0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, + 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7, + 0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, + 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5, + 0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, + 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B, + 0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, + 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59, + 0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, + 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F, + 0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, + 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D, + 0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, + 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433, + 0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, + 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01, + 0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, + 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457, + 0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, + 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65, + 0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, + 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB, + 0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, + 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9, + 0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, + 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F, + 0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, + 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD, + 0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, + 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683, + 0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, + 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1, + 0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, + 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7, + 0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, + 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5, + 0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, + 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B, + 0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, + 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79, + 0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, + 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F, + 0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, + 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D, + 0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, + 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713, + 0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, + 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21, + 0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, + 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777, + 0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, + 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45, + 0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, + 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB, + 0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, + 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9, + 0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, + 0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF, + 0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, + 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D +}; diff --git a/src/liblzma/lz/lz_encoder_mf.c b/src/liblzma/lz/lz_encoder_mf.c new file mode 100644 index 0000000000..557c2612f2 --- /dev/null +++ b/src/liblzma/lz/lz_encoder_mf.c @@ -0,0 +1,744 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lz_encoder_mf.c +/// \brief Match finders +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lz_encoder.h" +#include "lz_encoder_hash.h" +#include "memcmplen.h" + + +/// \brief Find matches starting from the current byte +/// +/// \return The length of the longest match found +extern uint32_t +lzma_mf_find(lzma_mf *mf, uint32_t *count_ptr, lzma_match *matches) +{ + // Call the match finder. It returns the number of length-distance + // pairs found. + // FIXME: Minimum count is zero, what _exactly_ is the maximum? + const uint32_t count = mf->find(mf, matches); + + // Length of the longest match; assume that no matches were found + // and thus the maximum length is zero. + uint32_t len_best = 0; + + if (count > 0) { +#ifndef NDEBUG + // Validate the matches. + for (uint32_t i = 0; i < count; ++i) { + assert(matches[i].len <= mf->nice_len); + assert(matches[i].dist < mf->read_pos); + assert(memcmp(mf_ptr(mf) - 1, + mf_ptr(mf) - matches[i].dist - 2, + matches[i].len) == 0); + } +#endif + + // The last used element in the array contains + // the longest match. + len_best = matches[count - 1].len; + + // If a match of maximum search length was found, try to + // extend the match to maximum possible length. + if (len_best == mf->nice_len) { + // The limit for the match length is either the + // maximum match length supported by the LZ-based + // encoder or the number of bytes left in the + // dictionary, whichever is smaller. + uint32_t limit = mf_avail(mf) + 1; + if (limit > mf->match_len_max) + limit = mf->match_len_max; + + // Pointer to the byte we just ran through + // the match finder. + const uint8_t *p1 = mf_ptr(mf) - 1; + + // Pointer to the beginning of the match. We need -1 + // here because the match distances are zero based. + const uint8_t *p2 = p1 - matches[count - 1].dist - 1; + + len_best = lzma_memcmplen(p1, p2, len_best, limit); + } + } + + *count_ptr = count; + + // Finally update the read position to indicate that match finder was + // run for this dictionary offset. + ++mf->read_ahead; + + return len_best; +} + + +/// Hash value to indicate unused element in the hash. Since we start the +/// positions from dict_size + 1, zero is always too far to qualify +/// as usable match position. +#define EMPTY_HASH_VALUE 0 + + +/// Normalization must be done when lzma_mf.offset + lzma_mf.read_pos +/// reaches MUST_NORMALIZE_POS. +#define MUST_NORMALIZE_POS UINT32_MAX + + +/// \brief Normalizes hash values +/// +/// The hash arrays store positions of match candidates. The positions are +/// relative to an arbitrary offset that is not the same as the absolute +/// offset in the input stream. The relative position of the current byte +/// is lzma_mf.offset + lzma_mf.read_pos. The distances of the matches are +/// the differences of the current read position and the position found from +/// the hash. +/// +/// To prevent integer overflows of the offsets stored in the hash arrays, +/// we need to "normalize" the stored values now and then. During the +/// normalization, we drop values that indicate distance greater than the +/// dictionary size, thus making space for new values. +static void +normalize(lzma_mf *mf) +{ + assert(mf->read_pos + mf->offset == MUST_NORMALIZE_POS); + + // In future we may not want to touch the lowest bits, because there + // may be match finders that use larger resolution than one byte. + const uint32_t subvalue + = (MUST_NORMALIZE_POS - mf->cyclic_size); + // & ~((UINT32_C(1) << 10) - 1); + + for (uint32_t i = 0; i < mf->hash_count; ++i) { + // If the distance is greater than the dictionary size, + // we can simply mark the hash element as empty. + if (mf->hash[i] <= subvalue) + mf->hash[i] = EMPTY_HASH_VALUE; + else + mf->hash[i] -= subvalue; + } + + for (uint32_t i = 0; i < mf->sons_count; ++i) { + // Do the same for mf->son. + // + // NOTE: There may be uninitialized elements in mf->son. + // Valgrind may complain that the "if" below depends on + // an uninitialized value. In this case it is safe to ignore + // the warning. See also the comments in lz_encoder_init() + // in lz_encoder.c. + if (mf->son[i] <= subvalue) + mf->son[i] = EMPTY_HASH_VALUE; + else + mf->son[i] -= subvalue; + } + + // Update offset to match the new locations. + mf->offset -= subvalue; + + return; +} + + +/// Mark the current byte as processed from point of view of the match finder. +static void +move_pos(lzma_mf *mf) +{ + if (++mf->cyclic_pos == mf->cyclic_size) + mf->cyclic_pos = 0; + + ++mf->read_pos; + assert(mf->read_pos <= mf->write_pos); + + if (unlikely(mf->read_pos + mf->offset == UINT32_MAX)) + normalize(mf); +} + + +/// When flushing, we cannot run the match finder unless there is nice_len +/// bytes available in the dictionary. Instead, we skip running the match +/// finder (indicating that no match was found), and count how many bytes we +/// have ignored this way. +/// +/// When new data is given after the flushing was completed, the match finder +/// is restarted by rewinding mf->read_pos backwards by mf->pending. Then +/// the missed bytes are added to the hash using the match finder's skip +/// function (with small amount of input, it may start using mf->pending +/// again if flushing). +/// +/// Due to this rewinding, we don't touch cyclic_pos or test for +/// normalization. It will be done when the match finder's skip function +/// catches up after a flush. +static void +move_pending(lzma_mf *mf) +{ + ++mf->read_pos; + assert(mf->read_pos <= mf->write_pos); + ++mf->pending; +} + + +/// Calculate len_limit and determine if there is enough input to run +/// the actual match finder code. Sets up "cur" and "pos". This macro +/// is used by all find functions and binary tree skip functions. Hash +/// chain skip function doesn't need len_limit so a simpler code is used +/// in them. +#define header(is_bt, len_min, ret_op) \ + uint32_t len_limit = mf_avail(mf); \ + if (mf->nice_len <= len_limit) { \ + len_limit = mf->nice_len; \ + } else if (len_limit < (len_min) \ + || (is_bt && mf->action == LZMA_SYNC_FLUSH)) { \ + assert(mf->action != LZMA_RUN); \ + move_pending(mf); \ + ret_op; \ + } \ + const uint8_t *cur = mf_ptr(mf); \ + const uint32_t pos = mf->read_pos + mf->offset + + +/// Header for find functions. "return 0" indicates that zero matches +/// were found. +#define header_find(is_bt, len_min) \ + header(is_bt, len_min, return 0); \ + uint32_t matches_count = 0 + + +/// Header for a loop in a skip function. "continue" tells to skip the rest +/// of the code in the loop. +#define header_skip(is_bt, len_min) \ + header(is_bt, len_min, continue) + + +/// Calls hc_find_func() or bt_find_func() and calculates the total number +/// of matches found. Updates the dictionary position and returns the number +/// of matches found. +#define call_find(func, len_best) \ +do { \ + matches_count = (uint32_t)(func(len_limit, pos, cur, cur_match, \ + mf->depth, mf->son, \ + mf->cyclic_pos, mf->cyclic_size, \ + matches + matches_count, len_best) \ + - matches); \ + move_pos(mf); \ + return matches_count; \ +} while (0) + + +//////////////// +// Hash Chain // +//////////////// + +#if defined(HAVE_MF_HC3) || defined(HAVE_MF_HC4) +/// +/// +/// \param len_limit Don't look for matches longer than len_limit. +/// \param pos lzma_mf.read_pos + lzma_mf.offset +/// \param cur Pointer to current byte (mf_ptr(mf)) +/// \param cur_match Start position of the current match candidate +/// \param depth Maximum length of the hash chain +/// \param son lzma_mf.son (contains the hash chain) +/// \param cyclic_pos lzma_mf.cyclic_pos +/// \param cyclic_size lzma_mf_cyclic_size +/// \param matches Array to hold the matches. +/// \param len_best The length of the longest match found so far. +static lzma_match * +hc_find_func( + const uint32_t len_limit, + const uint32_t pos, + const uint8_t *const cur, + uint32_t cur_match, + uint32_t depth, + uint32_t *const son, + const uint32_t cyclic_pos, + const uint32_t cyclic_size, + lzma_match *matches, + uint32_t len_best) +{ + son[cyclic_pos] = cur_match; + + while (true) { + const uint32_t delta = pos - cur_match; + if (depth-- == 0 || delta >= cyclic_size) + return matches; + + const uint8_t *const pb = cur - delta; + cur_match = son[cyclic_pos - delta + + (delta > cyclic_pos ? cyclic_size : 0)]; + + if (pb[len_best] == cur[len_best] && pb[0] == cur[0]) { + uint32_t len = lzma_memcmplen(pb, cur, 1, len_limit); + + if (len_best < len) { + len_best = len; + matches->len = len; + matches->dist = delta - 1; + ++matches; + + if (len == len_limit) + return matches; + } + } + } +} + + +#define hc_find(len_best) \ + call_find(hc_find_func, len_best) + + +#define hc_skip() \ +do { \ + mf->son[mf->cyclic_pos] = cur_match; \ + move_pos(mf); \ +} while (0) + +#endif + + +#ifdef HAVE_MF_HC3 +extern uint32_t +lzma_mf_hc3_find(lzma_mf *mf, lzma_match *matches) +{ + header_find(false, 3); + + hash_3_calc(); + + const uint32_t delta2 = pos - mf->hash[hash_2_value]; + const uint32_t cur_match = mf->hash[FIX_3_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_value] = pos; + + uint32_t len_best = 2; + + if (delta2 < mf->cyclic_size && *(cur - delta2) == *cur) { + len_best = lzma_memcmplen(cur - delta2, cur, + len_best, len_limit); + + matches[0].len = len_best; + matches[0].dist = delta2 - 1; + matches_count = 1; + + if (len_best == len_limit) { + hc_skip(); + return 1; // matches_count + } + } + + hc_find(len_best); +} + + +extern void +lzma_mf_hc3_skip(lzma_mf *mf, uint32_t amount) +{ + do { + if (mf_avail(mf) < 3) { + move_pending(mf); + continue; + } + + const uint8_t *cur = mf_ptr(mf); + const uint32_t pos = mf->read_pos + mf->offset; + + hash_3_calc(); + + const uint32_t cur_match + = mf->hash[FIX_3_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_value] = pos; + + hc_skip(); + + } while (--amount != 0); +} +#endif + + +#ifdef HAVE_MF_HC4 +extern uint32_t +lzma_mf_hc4_find(lzma_mf *mf, lzma_match *matches) +{ + header_find(false, 4); + + hash_4_calc(); + + uint32_t delta2 = pos - mf->hash[hash_2_value]; + const uint32_t delta3 + = pos - mf->hash[FIX_3_HASH_SIZE + hash_3_value]; + const uint32_t cur_match = mf->hash[FIX_4_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value ] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_3_value] = pos; + mf->hash[FIX_4_HASH_SIZE + hash_value] = pos; + + uint32_t len_best = 1; + + if (delta2 < mf->cyclic_size && *(cur - delta2) == *cur) { + len_best = 2; + matches[0].len = 2; + matches[0].dist = delta2 - 1; + matches_count = 1; + } + + if (delta2 != delta3 && delta3 < mf->cyclic_size + && *(cur - delta3) == *cur) { + len_best = 3; + matches[matches_count++].dist = delta3 - 1; + delta2 = delta3; + } + + if (matches_count != 0) { + len_best = lzma_memcmplen(cur - delta2, cur, + len_best, len_limit); + + matches[matches_count - 1].len = len_best; + + if (len_best == len_limit) { + hc_skip(); + return matches_count; + } + } + + if (len_best < 3) + len_best = 3; + + hc_find(len_best); +} + + +extern void +lzma_mf_hc4_skip(lzma_mf *mf, uint32_t amount) +{ + do { + if (mf_avail(mf) < 4) { + move_pending(mf); + continue; + } + + const uint8_t *cur = mf_ptr(mf); + const uint32_t pos = mf->read_pos + mf->offset; + + hash_4_calc(); + + const uint32_t cur_match + = mf->hash[FIX_4_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_3_value] = pos; + mf->hash[FIX_4_HASH_SIZE + hash_value] = pos; + + hc_skip(); + + } while (--amount != 0); +} +#endif + + +///////////////// +// Binary Tree // +///////////////// + +#if defined(HAVE_MF_BT2) || defined(HAVE_MF_BT3) || defined(HAVE_MF_BT4) +static lzma_match * +bt_find_func( + const uint32_t len_limit, + const uint32_t pos, + const uint8_t *const cur, + uint32_t cur_match, + uint32_t depth, + uint32_t *const son, + const uint32_t cyclic_pos, + const uint32_t cyclic_size, + lzma_match *matches, + uint32_t len_best) +{ + uint32_t *ptr0 = son + (cyclic_pos << 1) + 1; + uint32_t *ptr1 = son + (cyclic_pos << 1); + + uint32_t len0 = 0; + uint32_t len1 = 0; + + while (true) { + const uint32_t delta = pos - cur_match; + if (depth-- == 0 || delta >= cyclic_size) { + *ptr0 = EMPTY_HASH_VALUE; + *ptr1 = EMPTY_HASH_VALUE; + return matches; + } + + uint32_t *const pair = son + ((cyclic_pos - delta + + (delta > cyclic_pos ? cyclic_size : 0)) + << 1); + + const uint8_t *const pb = cur - delta; + uint32_t len = my_min(len0, len1); + + if (pb[len] == cur[len]) { + len = lzma_memcmplen(pb, cur, len + 1, len_limit); + + if (len_best < len) { + len_best = len; + matches->len = len; + matches->dist = delta - 1; + ++matches; + + if (len == len_limit) { + *ptr1 = pair[0]; + *ptr0 = pair[1]; + return matches; + } + } + } + + if (pb[len] < cur[len]) { + *ptr1 = cur_match; + ptr1 = pair + 1; + cur_match = *ptr1; + len1 = len; + } else { + *ptr0 = cur_match; + ptr0 = pair; + cur_match = *ptr0; + len0 = len; + } + } +} + + +static void +bt_skip_func( + const uint32_t len_limit, + const uint32_t pos, + const uint8_t *const cur, + uint32_t cur_match, + uint32_t depth, + uint32_t *const son, + const uint32_t cyclic_pos, + const uint32_t cyclic_size) +{ + uint32_t *ptr0 = son + (cyclic_pos << 1) + 1; + uint32_t *ptr1 = son + (cyclic_pos << 1); + + uint32_t len0 = 0; + uint32_t len1 = 0; + + while (true) { + const uint32_t delta = pos - cur_match; + if (depth-- == 0 || delta >= cyclic_size) { + *ptr0 = EMPTY_HASH_VALUE; + *ptr1 = EMPTY_HASH_VALUE; + return; + } + + uint32_t *pair = son + ((cyclic_pos - delta + + (delta > cyclic_pos ? cyclic_size : 0)) + << 1); + const uint8_t *pb = cur - delta; + uint32_t len = my_min(len0, len1); + + if (pb[len] == cur[len]) { + len = lzma_memcmplen(pb, cur, len + 1, len_limit); + + if (len == len_limit) { + *ptr1 = pair[0]; + *ptr0 = pair[1]; + return; + } + } + + if (pb[len] < cur[len]) { + *ptr1 = cur_match; + ptr1 = pair + 1; + cur_match = *ptr1; + len1 = len; + } else { + *ptr0 = cur_match; + ptr0 = pair; + cur_match = *ptr0; + len0 = len; + } + } +} + + +#define bt_find(len_best) \ + call_find(bt_find_func, len_best) + +#define bt_skip() \ +do { \ + bt_skip_func(len_limit, pos, cur, cur_match, mf->depth, \ + mf->son, mf->cyclic_pos, \ + mf->cyclic_size); \ + move_pos(mf); \ +} while (0) + +#endif + + +#ifdef HAVE_MF_BT2 +extern uint32_t +lzma_mf_bt2_find(lzma_mf *mf, lzma_match *matches) +{ + header_find(true, 2); + + hash_2_calc(); + + const uint32_t cur_match = mf->hash[hash_value]; + mf->hash[hash_value] = pos; + + bt_find(1); +} + + +extern void +lzma_mf_bt2_skip(lzma_mf *mf, uint32_t amount) +{ + do { + header_skip(true, 2); + + hash_2_calc(); + + const uint32_t cur_match = mf->hash[hash_value]; + mf->hash[hash_value] = pos; + + bt_skip(); + + } while (--amount != 0); +} +#endif + + +#ifdef HAVE_MF_BT3 +extern uint32_t +lzma_mf_bt3_find(lzma_mf *mf, lzma_match *matches) +{ + header_find(true, 3); + + hash_3_calc(); + + const uint32_t delta2 = pos - mf->hash[hash_2_value]; + const uint32_t cur_match = mf->hash[FIX_3_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_value] = pos; + + uint32_t len_best = 2; + + if (delta2 < mf->cyclic_size && *(cur - delta2) == *cur) { + len_best = lzma_memcmplen( + cur, cur - delta2, len_best, len_limit); + + matches[0].len = len_best; + matches[0].dist = delta2 - 1; + matches_count = 1; + + if (len_best == len_limit) { + bt_skip(); + return 1; // matches_count + } + } + + bt_find(len_best); +} + + +extern void +lzma_mf_bt3_skip(lzma_mf *mf, uint32_t amount) +{ + do { + header_skip(true, 3); + + hash_3_calc(); + + const uint32_t cur_match + = mf->hash[FIX_3_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_value] = pos; + + bt_skip(); + + } while (--amount != 0); +} +#endif + + +#ifdef HAVE_MF_BT4 +extern uint32_t +lzma_mf_bt4_find(lzma_mf *mf, lzma_match *matches) +{ + header_find(true, 4); + + hash_4_calc(); + + uint32_t delta2 = pos - mf->hash[hash_2_value]; + const uint32_t delta3 + = pos - mf->hash[FIX_3_HASH_SIZE + hash_3_value]; + const uint32_t cur_match = mf->hash[FIX_4_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_3_value] = pos; + mf->hash[FIX_4_HASH_SIZE + hash_value] = pos; + + uint32_t len_best = 1; + + if (delta2 < mf->cyclic_size && *(cur - delta2) == *cur) { + len_best = 2; + matches[0].len = 2; + matches[0].dist = delta2 - 1; + matches_count = 1; + } + + if (delta2 != delta3 && delta3 < mf->cyclic_size + && *(cur - delta3) == *cur) { + len_best = 3; + matches[matches_count++].dist = delta3 - 1; + delta2 = delta3; + } + + if (matches_count != 0) { + len_best = lzma_memcmplen( + cur, cur - delta2, len_best, len_limit); + + matches[matches_count - 1].len = len_best; + + if (len_best == len_limit) { + bt_skip(); + return matches_count; + } + } + + if (len_best < 3) + len_best = 3; + + bt_find(len_best); +} + + +extern void +lzma_mf_bt4_skip(lzma_mf *mf, uint32_t amount) +{ + do { + header_skip(true, 4); + + hash_4_calc(); + + const uint32_t cur_match + = mf->hash[FIX_4_HASH_SIZE + hash_value]; + + mf->hash[hash_2_value] = pos; + mf->hash[FIX_3_HASH_SIZE + hash_3_value] = pos; + mf->hash[FIX_4_HASH_SIZE + hash_value] = pos; + + bt_skip(); + + } while (--amount != 0); +} +#endif diff --git a/src/liblzma/lzma/Makefile.inc b/src/liblzma/lzma/Makefile.inc new file mode 100644 index 0000000000..dca6b764e1 --- /dev/null +++ b/src/liblzma/lzma/Makefile.inc @@ -0,0 +1,40 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +EXTRA_DIST += lzma/fastpos_tablegen.c + +liblzma_la_SOURCES += \ + lzma/lzma_common.h \ + lzma/lzma_encoder_presets.c + +if COND_ENCODER_LZMA1 +liblzma_la_SOURCES += \ + lzma/fastpos.h \ + lzma/lzma_encoder.h \ + lzma/lzma_encoder.c \ + lzma/lzma_encoder_private.h \ + lzma/lzma_encoder_optimum_fast.c \ + lzma/lzma_encoder_optimum_normal.c + +if !COND_SMALL +liblzma_la_SOURCES += lzma/fastpos_table.c +endif +endif + +if COND_DECODER_LZMA1 +liblzma_la_SOURCES += \ + lzma/lzma_decoder.c \ + lzma/lzma_decoder.h +endif + +if COND_ENCODER_LZMA2 +liblzma_la_SOURCES += \ + lzma/lzma2_encoder.c \ + lzma/lzma2_encoder.h +endif + +if COND_DECODER_LZMA2 +liblzma_la_SOURCES += \ + lzma/lzma2_decoder.c \ + lzma/lzma2_decoder.h +endif diff --git a/src/liblzma/lzma/fastpos.h b/src/liblzma/lzma/fastpos.h new file mode 100644 index 0000000000..d3969a753f --- /dev/null +++ b/src/liblzma/lzma/fastpos.h @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file fastpos.h +/// \brief Kind of two-bit version of bit scan reverse +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_FASTPOS_H +#define LZMA_FASTPOS_H + +// LZMA encodes match distances by storing the highest two bits using +// a six-bit value [0, 63], and then the missing lower bits. +// Dictionary size is also stored using this encoding in the .xz +// file format header. +// +// fastpos.h provides a way to quickly find out the correct six-bit +// values. The following table gives some examples of this encoding: +// +// dist return +// 0 0 +// 1 1 +// 2 2 +// 3 3 +// 4 4 +// 5 4 +// 6 5 +// 7 5 +// 8 6 +// 11 6 +// 12 7 +// ... ... +// 15 7 +// 16 8 +// 17 8 +// ... ... +// 23 8 +// 24 9 +// 25 9 +// ... ... +// +// +// Provided functions or macros +// ---------------------------- +// +// get_dist_slot(dist) is the basic version. get_dist_slot_2(dist) +// assumes that dist >= FULL_DISTANCES, thus the result is at least +// FULL_DISTANCES_BITS * 2. Using get_dist_slot(dist) instead of +// get_dist_slot_2(dist) would give the same result, but get_dist_slot_2(dist) +// should be tiny bit faster due to the assumption being made. +// +// +// Size vs. speed +// -------------- +// +// With some CPUs that have fast BSR (bit scan reverse) instruction, the +// size optimized version is slightly faster than the bigger table based +// approach. Such CPUs include Intel Pentium Pro, Pentium II, Pentium III +// and Core 2 (possibly others). AMD K7 seems to have slower BSR, but that +// would still have speed roughly comparable to the table version. Older +// x86 CPUs like the original Pentium have very slow BSR; on those systems +// the table version is a lot faster. +// +// On some CPUs, the table version is a lot faster when using position +// dependent code, but with position independent code the size optimized +// version is slightly faster. This occurs at least on 32-bit SPARC (no +// ASM optimizations). +// +// I'm making the table version the default, because that has good speed +// on all systems I have tried. The size optimized version is sometimes +// slightly faster, but sometimes it is a lot slower. + +#ifdef HAVE_SMALL +# define get_dist_slot(dist) \ + ((dist) <= 4 ? (dist) : get_dist_slot_2(dist)) + +static inline uint32_t +get_dist_slot_2(uint32_t dist) +{ + const uint32_t i = bsr32(dist); + return (i + i) + ((dist >> (i - 1)) & 1); +} + + +#else + +#define FASTPOS_BITS 13 + +lzma_attr_visibility_hidden +extern const uint8_t lzma_fastpos[1 << FASTPOS_BITS]; + + +#define fastpos_shift(extra, n) \ + ((extra) + (n) * (FASTPOS_BITS - 1)) + +#define fastpos_limit(extra, n) \ + (UINT32_C(1) << (FASTPOS_BITS + fastpos_shift(extra, n))) + +#define fastpos_result(dist, extra, n) \ + (uint32_t)(lzma_fastpos[(dist) >> fastpos_shift(extra, n)]) \ + + 2 * fastpos_shift(extra, n) + + +static inline uint32_t +get_dist_slot(uint32_t dist) +{ + // If it is small enough, we can pick the result directly from + // the precalculated table. + if (dist < fastpos_limit(0, 0)) + return lzma_fastpos[dist]; + + if (dist < fastpos_limit(0, 1)) + return fastpos_result(dist, 0, 1); + + return fastpos_result(dist, 0, 2); +} + + +#ifdef FULL_DISTANCES_BITS +static inline uint32_t +get_dist_slot_2(uint32_t dist) +{ + assert(dist >= FULL_DISTANCES); + + if (dist < fastpos_limit(FULL_DISTANCES_BITS - 1, 0)) + return fastpos_result(dist, FULL_DISTANCES_BITS - 1, 0); + + if (dist < fastpos_limit(FULL_DISTANCES_BITS - 1, 1)) + return fastpos_result(dist, FULL_DISTANCES_BITS - 1, 1); + + return fastpos_result(dist, FULL_DISTANCES_BITS - 1, 2); +} +#endif + +#endif + +#endif diff --git a/src/liblzma/lzma/fastpos_table.c b/src/liblzma/lzma/fastpos_table.c new file mode 100644 index 0000000000..4e10e3795e --- /dev/null +++ b/src/liblzma/lzma/fastpos_table.c @@ -0,0 +1,521 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by fastpos_tablegen.c. + +#include "common.h" +#include "fastpos.h" + +const uint8_t lzma_fastpos[1 << FASTPOS_BITS] = { + 0, 1, 2, 3, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, + 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, + 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, + 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, + 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, + 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, + 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, + 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, + 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, + 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 +}; diff --git a/src/liblzma/lzma/fastpos_tablegen.c b/src/liblzma/lzma/fastpos_tablegen.c new file mode 100644 index 0000000000..957ccb7a64 --- /dev/null +++ b/src/liblzma/lzma/fastpos_tablegen.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file fastpos_tablegen.c +/// \brief Generates the lzma_fastpos[] lookup table +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include +#include + +#define lzma_attr_visibility_hidden +#include "fastpos.h" + + +int +main(void) +{ + uint8_t fastpos[1 << FASTPOS_BITS]; + + const uint8_t fast_slots = 2 * FASTPOS_BITS; + uint32_t c = 2; + + fastpos[0] = 0; + fastpos[1] = 1; + + for (uint8_t slot_fast = 2; slot_fast < fast_slots; ++slot_fast) { + const uint32_t k = 1 << ((slot_fast >> 1) - 1); + for (uint32_t j = 0; j < k; ++j, ++c) + fastpos[c] = slot_fast; + } + + // Split the SPDX string so that it won't accidentally match + // when tools search for the string. + printf("// SPDX" "-License-Identifier" ": 0BSD\n\n" + "// This file has been generated by fastpos_tablegen.c.\n\n" + "#include \"common.h\"\n" + "#include \"fastpos.h\"\n\n" + "const uint8_t lzma_fastpos[1 << FASTPOS_BITS] = {"); + + for (size_t i = 0; i < (1 << FASTPOS_BITS); ++i) { + if (i % 16 == 0) + printf("\n\t"); + + printf("%3u", (unsigned int)(fastpos[i])); + + if (i != (1 << FASTPOS_BITS) - 1) + printf(","); + } + + printf("\n};\n"); + + return 0; +} diff --git a/src/liblzma/lzma/lzma2_decoder.c b/src/liblzma/lzma/lzma2_decoder.c new file mode 100644 index 0000000000..37ab253f5b --- /dev/null +++ b/src/liblzma/lzma/lzma2_decoder.c @@ -0,0 +1,310 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma2_decoder.c +/// \brief LZMA2 decoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma2_decoder.h" +#include "lz_decoder.h" +#include "lzma_decoder.h" + + +typedef struct { + enum sequence { + SEQ_CONTROL, + SEQ_UNCOMPRESSED_1, + SEQ_UNCOMPRESSED_2, + SEQ_COMPRESSED_0, + SEQ_COMPRESSED_1, + SEQ_PROPERTIES, + SEQ_LZMA, + SEQ_COPY, + } sequence; + + /// Sequence after the size fields have been decoded. + enum sequence next_sequence; + + /// LZMA decoder + lzma_lz_decoder lzma; + + /// Uncompressed size of LZMA chunk + size_t uncompressed_size; + + /// Compressed size of the chunk (naturally equals to uncompressed + /// size of uncompressed chunk) + size_t compressed_size; + + /// True if properties are needed. This is false before the + /// first LZMA chunk. + bool need_properties; + + /// True if dictionary reset is needed. This is false before the + /// first chunk (LZMA or uncompressed). + bool need_dictionary_reset; + + lzma_options_lzma options; +} lzma_lzma2_coder; + + +static lzma_ret +lzma2_decode(void *coder_ptr, lzma_dict *restrict dict, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size) +{ + lzma_lzma2_coder *restrict coder = coder_ptr; + + // With SEQ_LZMA it is possible that no new input is needed to do + // some progress. The rest of the sequences assume that there is + // at least one byte of input. + while (*in_pos < in_size || coder->sequence == SEQ_LZMA) + switch (coder->sequence) { + case SEQ_CONTROL: { + const uint32_t control = in[*in_pos]; + ++*in_pos; + + // End marker + if (control == 0x00) + return LZMA_STREAM_END; + + if (control >= 0xE0 || control == 1) { + // Dictionary reset implies that next LZMA chunk has + // to set new properties. + coder->need_properties = true; + coder->need_dictionary_reset = true; + } else if (coder->need_dictionary_reset) { + return LZMA_DATA_ERROR; + } + + if (control >= 0x80) { + // LZMA chunk. The highest five bits of the + // uncompressed size are taken from the control byte. + coder->uncompressed_size = (control & 0x1F) << 16; + coder->sequence = SEQ_UNCOMPRESSED_1; + + // See if there are new properties or if we need to + // reset the state. + if (control >= 0xC0) { + // When there are new properties, state reset + // is done at SEQ_PROPERTIES. + coder->need_properties = false; + coder->next_sequence = SEQ_PROPERTIES; + + } else if (coder->need_properties) { + return LZMA_DATA_ERROR; + + } else { + coder->next_sequence = SEQ_LZMA; + + // If only state reset is wanted with old + // properties, do the resetting here for + // simplicity. + if (control >= 0xA0) + coder->lzma.reset(coder->lzma.coder, + &coder->options); + } + } else { + // Invalid control values + if (control > 2) + return LZMA_DATA_ERROR; + + // It's uncompressed chunk + coder->sequence = SEQ_COMPRESSED_0; + coder->next_sequence = SEQ_COPY; + } + + if (coder->need_dictionary_reset) { + // Finish the dictionary reset and let the caller + // flush the dictionary to the actual output buffer. + coder->need_dictionary_reset = false; + dict_reset(dict); + return LZMA_OK; + } + + break; + } + + case SEQ_UNCOMPRESSED_1: + coder->uncompressed_size += (uint32_t)(in[(*in_pos)++]) << 8; + coder->sequence = SEQ_UNCOMPRESSED_2; + break; + + case SEQ_UNCOMPRESSED_2: + coder->uncompressed_size += in[(*in_pos)++] + 1U; + coder->sequence = SEQ_COMPRESSED_0; + coder->lzma.set_uncompressed(coder->lzma.coder, + coder->uncompressed_size, false); + break; + + case SEQ_COMPRESSED_0: + coder->compressed_size = (uint32_t)(in[(*in_pos)++]) << 8; + coder->sequence = SEQ_COMPRESSED_1; + break; + + case SEQ_COMPRESSED_1: + coder->compressed_size += in[(*in_pos)++] + 1U; + coder->sequence = coder->next_sequence; + break; + + case SEQ_PROPERTIES: + if (lzma_lzma_lclppb_decode(&coder->options, in[(*in_pos)++])) + return LZMA_DATA_ERROR; + + coder->lzma.reset(coder->lzma.coder, &coder->options); + + coder->sequence = SEQ_LZMA; + break; + + case SEQ_LZMA: { + // Store the start offset so that we can update + // coder->compressed_size later. + const size_t in_start = *in_pos; + + // Decode from in[] to *dict. + const lzma_ret ret = coder->lzma.code(coder->lzma.coder, + dict, in, in_pos, in_size); + + // Validate and update coder->compressed_size. + const size_t in_used = *in_pos - in_start; + if (in_used > coder->compressed_size) + return LZMA_DATA_ERROR; + + coder->compressed_size -= in_used; + + // Return if we didn't finish the chunk, or an error occurred. + if (ret != LZMA_STREAM_END) + return ret; + + // The LZMA decoder must have consumed the whole chunk now. + // We don't need to worry about uncompressed size since it + // is checked by the LZMA decoder. + if (coder->compressed_size != 0) + return LZMA_DATA_ERROR; + + coder->sequence = SEQ_CONTROL; + break; + } + + case SEQ_COPY: { + // Copy from input to the dictionary as is. + dict_write(dict, in, in_pos, in_size, &coder->compressed_size); + if (coder->compressed_size != 0) + return LZMA_OK; + + coder->sequence = SEQ_CONTROL; + break; + } + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + return LZMA_OK; +} + + +static void +lzma2_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_lzma2_coder *coder = coder_ptr; + + assert(coder->lzma.end == NULL); + lzma_free(coder->lzma.coder, allocator); + + lzma_free(coder, allocator); + + return; +} + + +static lzma_ret +lzma2_decoder_init(lzma_lz_decoder *lz, const lzma_allocator *allocator, + lzma_vli id lzma_attribute((__unused__)), const void *opt, + lzma_lz_options *lz_options) +{ + lzma_lzma2_coder *coder = lz->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_lzma2_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + lz->coder = coder; + lz->code = &lzma2_decode; + lz->end = &lzma2_decoder_end; + + coder->lzma = LZMA_LZ_DECODER_INIT; + } + + const lzma_options_lzma *options = opt; + + coder->sequence = SEQ_CONTROL; + coder->need_properties = true; + coder->need_dictionary_reset = options->preset_dict == NULL + || options->preset_dict_size == 0; + + return lzma_lzma_decoder_create(&coder->lzma, + allocator, options, lz_options); +} + + +extern lzma_ret +lzma_lzma2_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + // LZMA2 can only be the last filter in the chain. This is enforced + // by the raw_decoder initialization. + assert(filters[1].init == NULL); + + return lzma_lz_decoder_init(next, allocator, filters, + &lzma2_decoder_init); +} + + +extern uint64_t +lzma_lzma2_decoder_memusage(const void *options) +{ + return sizeof(lzma_lzma2_coder) + + lzma_lzma_decoder_memusage_nocheck(options); +} + + +extern lzma_ret +lzma_lzma2_props_decode(void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size) +{ + if (props_size != 1) + return LZMA_OPTIONS_ERROR; + + // Check that reserved bits are unset. + if (props[0] & 0xC0) + return LZMA_OPTIONS_ERROR; + + // Decode the dictionary size. + if (props[0] > 40) + return LZMA_OPTIONS_ERROR; + + lzma_options_lzma *opt = lzma_alloc( + sizeof(lzma_options_lzma), allocator); + if (opt == NULL) + return LZMA_MEM_ERROR; + + if (props[0] == 40) { + opt->dict_size = UINT32_MAX; + } else { + opt->dict_size = 2 | (props[0] & 1U); + opt->dict_size <<= props[0] / 2U + 11; + } + + opt->preset_dict = NULL; + opt->preset_dict_size = 0; + + *options = opt; + + return LZMA_OK; +} diff --git a/src/liblzma/lzma/lzma2_decoder.h b/src/liblzma/lzma/lzma2_decoder.h new file mode 100644 index 0000000000..cdd8b463ab --- /dev/null +++ b/src/liblzma/lzma/lzma2_decoder.h @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma2_decoder.h +/// \brief LZMA2 decoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZMA2_DECODER_H +#define LZMA_LZMA2_DECODER_H + +#include "common.h" + +extern lzma_ret lzma_lzma2_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern uint64_t lzma_lzma2_decoder_memusage(const void *options); + +extern lzma_ret lzma_lzma2_props_decode( + void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size); + +#endif diff --git a/src/liblzma/lzma/lzma2_encoder.c b/src/liblzma/lzma/lzma2_encoder.c new file mode 100644 index 0000000000..71cfd9b411 --- /dev/null +++ b/src/liblzma/lzma/lzma2_encoder.c @@ -0,0 +1,413 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma2_encoder.c +/// \brief LZMA2 encoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lz_encoder.h" +#include "lzma_encoder.h" +#include "fastpos.h" +#include "lzma2_encoder.h" + + +typedef struct { + enum { + SEQ_INIT, + SEQ_LZMA_ENCODE, + SEQ_LZMA_COPY, + SEQ_UNCOMPRESSED_HEADER, + SEQ_UNCOMPRESSED_COPY, + } sequence; + + /// LZMA encoder + void *lzma; + + /// LZMA options currently in use. + lzma_options_lzma opt_cur; + + bool need_properties; + bool need_state_reset; + bool need_dictionary_reset; + + /// Uncompressed size of a chunk + size_t uncompressed_size; + + /// Compressed size of a chunk (excluding headers); this is also used + /// to indicate the end of buf[] in SEQ_LZMA_COPY. + size_t compressed_size; + + /// Read position in buf[] + size_t buf_pos; + + /// Buffer to hold the chunk header and LZMA compressed data + uint8_t buf[LZMA2_HEADER_MAX + LZMA2_CHUNK_MAX]; +} lzma_lzma2_coder; + + +static void +lzma2_header_lzma(lzma_lzma2_coder *coder) +{ + assert(coder->uncompressed_size > 0); + assert(coder->uncompressed_size <= LZMA2_UNCOMPRESSED_MAX); + assert(coder->compressed_size > 0); + assert(coder->compressed_size <= LZMA2_CHUNK_MAX); + + size_t pos; + + if (coder->need_properties) { + pos = 0; + + if (coder->need_dictionary_reset) + coder->buf[pos] = 0x80 + (3 << 5); + else + coder->buf[pos] = 0x80 + (2 << 5); + } else { + pos = 1; + + if (coder->need_state_reset) + coder->buf[pos] = 0x80 + (1 << 5); + else + coder->buf[pos] = 0x80; + } + + // Set the start position for copying. + coder->buf_pos = pos; + + // Uncompressed size + size_t size = coder->uncompressed_size - 1; + coder->buf[pos++] += size >> 16; + coder->buf[pos++] = (size >> 8) & 0xFF; + coder->buf[pos++] = size & 0xFF; + + // Compressed size + size = coder->compressed_size - 1; + coder->buf[pos++] = size >> 8; + coder->buf[pos++] = size & 0xFF; + + // Properties, if needed + if (coder->need_properties) + lzma_lzma_lclppb_encode(&coder->opt_cur, coder->buf + pos); + + coder->need_properties = false; + coder->need_state_reset = false; + coder->need_dictionary_reset = false; + + // The copying code uses coder->compressed_size to indicate the end + // of coder->buf[], so we need add the maximum size of the header here. + coder->compressed_size += LZMA2_HEADER_MAX; + + return; +} + + +static void +lzma2_header_uncompressed(lzma_lzma2_coder *coder) +{ + assert(coder->uncompressed_size > 0); + assert(coder->uncompressed_size <= LZMA2_CHUNK_MAX); + + // If this is the first chunk, we need to include dictionary + // reset indicator. + if (coder->need_dictionary_reset) + coder->buf[0] = 1; + else + coder->buf[0] = 2; + + coder->need_dictionary_reset = false; + + // "Compressed" size + coder->buf[1] = (coder->uncompressed_size - 1) >> 8; + coder->buf[2] = (coder->uncompressed_size - 1) & 0xFF; + + // Set the start position for copying. + coder->buf_pos = 0; + return; +} + + +static lzma_ret +lzma2_encode(void *coder_ptr, lzma_mf *restrict mf, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size) +{ + lzma_lzma2_coder *restrict coder = coder_ptr; + + while (*out_pos < out_size) + switch (coder->sequence) { + case SEQ_INIT: + // If there's no input left and we are flushing or finishing, + // don't start a new chunk. + if (mf_unencoded(mf) == 0) { + // Write end of payload marker if finishing. + if (mf->action == LZMA_FINISH) + out[(*out_pos)++] = 0; + + return mf->action == LZMA_RUN + ? LZMA_OK : LZMA_STREAM_END; + } + + if (coder->need_state_reset) + return_if_error(lzma_lzma_encoder_reset( + coder->lzma, &coder->opt_cur)); + + coder->uncompressed_size = 0; + coder->compressed_size = 0; + coder->sequence = SEQ_LZMA_ENCODE; + FALLTHROUGH; + + case SEQ_LZMA_ENCODE: { + // Calculate how much more uncompressed data this chunk + // could accept. + const uint32_t left = LZMA2_UNCOMPRESSED_MAX + - coder->uncompressed_size; + uint32_t limit; + + if (left < mf->match_len_max) { + // Must flush immediately since the next LZMA symbol + // could make the uncompressed size of the chunk too + // big. + limit = 0; + } else { + // Calculate maximum read_limit that is OK from point + // of view of LZMA2 chunk size. + limit = mf->read_pos - mf->read_ahead + + left - mf->match_len_max; + } + + // Save the start position so that we can update + // coder->uncompressed_size. + const uint32_t read_start = mf->read_pos - mf->read_ahead; + + // Call the LZMA encoder until the chunk is finished. + const lzma_ret ret = lzma_lzma_encode(coder->lzma, mf, + coder->buf + LZMA2_HEADER_MAX, + &coder->compressed_size, + LZMA2_CHUNK_MAX, limit); + + coder->uncompressed_size += mf->read_pos - mf->read_ahead + - read_start; + + assert(coder->compressed_size <= LZMA2_CHUNK_MAX); + assert(coder->uncompressed_size <= LZMA2_UNCOMPRESSED_MAX); + + if (ret != LZMA_STREAM_END) + return LZMA_OK; + + // See if the chunk compressed. If it didn't, we encode it + // as uncompressed chunk. This saves a few bytes of space + // and makes decoding faster. + if (coder->compressed_size >= coder->uncompressed_size) { + coder->uncompressed_size += mf->read_ahead; + assert(coder->uncompressed_size + <= LZMA2_UNCOMPRESSED_MAX); + mf->read_ahead = 0; + lzma2_header_uncompressed(coder); + coder->need_state_reset = true; + coder->sequence = SEQ_UNCOMPRESSED_HEADER; + break; + } + + // The chunk did compress at least by one byte, so we store + // the chunk as LZMA. + lzma2_header_lzma(coder); + + coder->sequence = SEQ_LZMA_COPY; + FALLTHROUGH; + } + + case SEQ_LZMA_COPY: + // Copy the compressed chunk along its headers to the + // output buffer. + lzma_bufcpy(coder->buf, &coder->buf_pos, + coder->compressed_size, + out, out_pos, out_size); + if (coder->buf_pos != coder->compressed_size) + return LZMA_OK; + + coder->sequence = SEQ_INIT; + break; + + case SEQ_UNCOMPRESSED_HEADER: + // Copy the three-byte header to indicate uncompressed chunk. + lzma_bufcpy(coder->buf, &coder->buf_pos, + LZMA2_HEADER_UNCOMPRESSED, + out, out_pos, out_size); + if (coder->buf_pos != LZMA2_HEADER_UNCOMPRESSED) + return LZMA_OK; + + coder->sequence = SEQ_UNCOMPRESSED_COPY; + FALLTHROUGH; + + case SEQ_UNCOMPRESSED_COPY: + // Copy the uncompressed data as is from the dictionary + // to the output buffer. + mf_read(mf, out, out_pos, out_size, &coder->uncompressed_size); + if (coder->uncompressed_size != 0) + return LZMA_OK; + + coder->sequence = SEQ_INIT; + break; + } + + return LZMA_OK; +} + + +static void +lzma2_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_lzma2_coder *coder = coder_ptr; + lzma_free(coder->lzma, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +lzma2_encoder_options_update(void *coder_ptr, const lzma_filter *filter) +{ + lzma_lzma2_coder *coder = coder_ptr; + + // New options can be set only when there is no incomplete chunk. + // This is the case at the beginning of the raw stream and right + // after LZMA_SYNC_FLUSH. + if (filter->options == NULL || coder->sequence != SEQ_INIT) + return LZMA_PROG_ERROR; + + // Look if there are new options. At least for now, + // only lc/lp/pb can be changed. + const lzma_options_lzma *opt = filter->options; + if (coder->opt_cur.lc != opt->lc || coder->opt_cur.lp != opt->lp + || coder->opt_cur.pb != opt->pb) { + // Validate the options. + if (opt->lc > LZMA_LCLP_MAX || opt->lp > LZMA_LCLP_MAX + || opt->lc + opt->lp > LZMA_LCLP_MAX + || opt->pb > LZMA_PB_MAX) + return LZMA_OPTIONS_ERROR; + + // The new options will be used when the encoder starts + // a new LZMA2 chunk. + coder->opt_cur.lc = opt->lc; + coder->opt_cur.lp = opt->lp; + coder->opt_cur.pb = opt->pb; + coder->need_properties = true; + coder->need_state_reset = true; + } + + return LZMA_OK; +} + + +static lzma_ret +lzma2_encoder_init(lzma_lz_encoder *lz, const lzma_allocator *allocator, + lzma_vli id lzma_attribute((__unused__)), const void *options, + lzma_lz_options *lz_options) +{ + if (options == NULL) + return LZMA_PROG_ERROR; + + lzma_lzma2_coder *coder = lz->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_lzma2_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + lz->coder = coder; + lz->code = &lzma2_encode; + lz->end = &lzma2_encoder_end; + lz->options_update = &lzma2_encoder_options_update; + + coder->lzma = NULL; + } + + coder->opt_cur = *(const lzma_options_lzma *)(options); + + coder->sequence = SEQ_INIT; + coder->need_properties = true; + coder->need_state_reset = false; + coder->need_dictionary_reset + = coder->opt_cur.preset_dict == NULL + || coder->opt_cur.preset_dict_size == 0; + + // Initialize LZMA encoder + return_if_error(lzma_lzma_encoder_create(&coder->lzma, allocator, + LZMA_FILTER_LZMA2, &coder->opt_cur, lz_options)); + + // Make sure that we will always have enough history available in + // case we need to use uncompressed chunks. They are used when the + // compressed size of a chunk is not smaller than the uncompressed + // size, so we need to have at least LZMA2_COMPRESSED_MAX bytes + // history available. + if (lz_options->before_size + lz_options->dict_size < LZMA2_CHUNK_MAX) + lz_options->before_size + = LZMA2_CHUNK_MAX - lz_options->dict_size; + + return LZMA_OK; +} + + +extern lzma_ret +lzma_lzma2_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return lzma_lz_encoder_init( + next, allocator, filters, &lzma2_encoder_init); +} + + +extern uint64_t +lzma_lzma2_encoder_memusage(const void *options) +{ + const uint64_t lzma_mem = lzma_lzma_encoder_memusage(options); + if (lzma_mem == UINT64_MAX) + return UINT64_MAX; + + return sizeof(lzma_lzma2_coder) + lzma_mem; +} + + +extern lzma_ret +lzma_lzma2_props_encode(const void *options, uint8_t *out) +{ + if (options == NULL) + return LZMA_PROG_ERROR; + + const lzma_options_lzma *const opt = options; + uint32_t d = my_max(opt->dict_size, LZMA_DICT_SIZE_MIN); + + // Round up to the next 2^n - 1 or 2^n + 2^(n - 1) - 1 depending + // on which one is the next: + --d; + d |= d >> 2; + d |= d >> 3; + d |= d >> 4; + d |= d >> 8; + d |= d >> 16; + + // Get the highest two bits using the proper encoding: + if (d == UINT32_MAX) + out[0] = 40; + else + out[0] = get_dist_slot(d + 1) - 24; + + return LZMA_OK; +} + + +extern uint64_t +lzma_lzma2_block_size(const void *options) +{ + const lzma_options_lzma *const opt = options; + + if (!IS_ENC_DICT_SIZE_VALID(opt->dict_size)) + return UINT64_MAX; + + // Use at least 1 MiB to keep compression ratio better. + return my_max((uint64_t)(opt->dict_size) * 3, UINT64_C(1) << 20); +} diff --git a/src/liblzma/lzma/lzma2_encoder.h b/src/liblzma/lzma/lzma2_encoder.h new file mode 100644 index 0000000000..29966a66d2 --- /dev/null +++ b/src/liblzma/lzma/lzma2_encoder.h @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma2_encoder.h +/// \brief LZMA2 encoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZMA2_ENCODER_H +#define LZMA_LZMA2_ENCODER_H + +#include "common.h" + + +/// Maximum number of bytes of actual data per chunk (no headers) +#define LZMA2_CHUNK_MAX (UINT32_C(1) << 16) + +/// Maximum uncompressed size of LZMA chunk (no headers) +#define LZMA2_UNCOMPRESSED_MAX (UINT32_C(1) << 21) + +/// Maximum size of LZMA2 headers +#define LZMA2_HEADER_MAX 6 + +/// Size of a header for uncompressed chunk +#define LZMA2_HEADER_UNCOMPRESSED 3 + + +extern lzma_ret lzma_lzma2_encoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern uint64_t lzma_lzma2_encoder_memusage(const void *options); + +extern lzma_ret lzma_lzma2_props_encode(const void *options, uint8_t *out); + +extern uint64_t lzma_lzma2_block_size(const void *options); + +#endif diff --git a/src/liblzma/lzma/lzma_common.h b/src/liblzma/lzma/lzma_common.h new file mode 100644 index 0000000000..c3c587f090 --- /dev/null +++ b/src/liblzma/lzma/lzma_common.h @@ -0,0 +1,240 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_common.h +/// \brief Private definitions common to LZMA encoder and decoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZMA_COMMON_H +#define LZMA_LZMA_COMMON_H + +#include "common.h" +#include "range_common.h" + + +/////////////////// +// Miscellaneous // +/////////////////// + +/// Maximum number of position states. A position state is the lowest pos bits +/// number of bits of the current uncompressed offset. In some places there +/// are different sets of probabilities for different pos states. +#define POS_STATES_MAX (1 << LZMA_PB_MAX) + + +/// Validates lc, lp, and pb. +static inline bool +is_lclppb_valid(const lzma_options_lzma *options) +{ + return options->lc <= LZMA_LCLP_MAX && options->lp <= LZMA_LCLP_MAX + && options->lc + options->lp <= LZMA_LCLP_MAX + && options->pb <= LZMA_PB_MAX; +} + + +/////////// +// State // +/////////// + +/// This enum is used to track which events have occurred most recently and +/// in which order. This information is used to predict the next event. +/// +/// Events: +/// - Literal: One 8-bit byte +/// - Match: Repeat a chunk of data at some distance +/// - Long repeat: Multi-byte match at a recently seen distance +/// - Short repeat: One-byte repeat at a recently seen distance +/// +/// The event names are in from STATE_oldest_older_previous. REP means +/// either short or long repeated match, and NONLIT means any non-literal. +typedef enum { + STATE_LIT_LIT, + STATE_MATCH_LIT_LIT, + STATE_REP_LIT_LIT, + STATE_SHORTREP_LIT_LIT, + STATE_MATCH_LIT, + STATE_REP_LIT, + STATE_SHORTREP_LIT, + STATE_LIT_MATCH, + STATE_LIT_LONGREP, + STATE_LIT_SHORTREP, + STATE_NONLIT_MATCH, + STATE_NONLIT_REP, +} lzma_lzma_state; + + +/// Total number of states +#define STATES 12 + +/// The lowest 7 states indicate that the previous state was a literal. +#define LIT_STATES 7 + + +/// Indicate that the latest state was a literal. +#define update_literal(state) \ + state = ((state) <= STATE_SHORTREP_LIT_LIT \ + ? STATE_LIT_LIT \ + : ((state) <= STATE_LIT_SHORTREP \ + ? (state) - 3 \ + : (state) - 6)) + +/// Like update_literal(state) but when it is already known that +/// is_literal_state(state) is true. +#define update_literal_normal(state) \ + state = ((state) <= STATE_SHORTREP_LIT_LIT \ + ? STATE_LIT_LIT \ + : (state) - 3); + +/// Like update_literal(state) but when it is already known that +/// is_literal_state(state) is false. +#define update_literal_matched(state) \ + state = ((state) <= STATE_LIT_SHORTREP \ + ? (state) - 3 \ + : (state) - 6); + +/// Indicate that the latest state was a match. +#define update_match(state) \ + state = ((state) < LIT_STATES ? STATE_LIT_MATCH : STATE_NONLIT_MATCH) + +/// Indicate that the latest state was a long repeated match. +#define update_long_rep(state) \ + state = ((state) < LIT_STATES ? STATE_LIT_LONGREP : STATE_NONLIT_REP) + +/// Indicate that the latest state was a short match. +#define update_short_rep(state) \ + state = ((state) < LIT_STATES ? STATE_LIT_SHORTREP : STATE_NONLIT_REP) + +/// Test if the previous state was a literal. +#define is_literal_state(state) \ + ((state) < LIT_STATES) + + +///////////// +// Literal // +///////////// + +/// Each literal coder is divided in three sections: +/// - 0x001-0x0FF: Without match byte +/// - 0x101-0x1FF: With match byte; match bit is 0 +/// - 0x201-0x2FF: With match byte; match bit is 1 +/// +/// Match byte is used when the previous LZMA symbol was something else than +/// a literal (that is, it was some kind of match). +#define LITERAL_CODER_SIZE UINT32_C(0x300) + +/// Maximum number of literal coders +#define LITERAL_CODERS_MAX (1 << LZMA_LCLP_MAX) + +/// Calculates the literal_mask that literal_subcoder() needs. +#define literal_mask_calc(lc, lp) \ + ((UINT32_C(0x100) << (lp)) - (UINT32_C(0x100) >> (lc))) + +/// Locate the literal coder for the next literal byte. The choice depends on +/// - the lowest literal_pos_bits bits of the position of the current +/// byte; and +/// - the highest literal_context_bits bits of the previous byte. +#define literal_subcoder(probs, lc, literal_mask, pos, prev_byte) \ + ((probs) + UINT32_C(3) * \ + (((((pos) << 8) + (prev_byte)) & (literal_mask)) << (lc))) + + +static inline void +literal_init(probability *probs, uint32_t lc, uint32_t lp) +{ + assert(lc + lp <= LZMA_LCLP_MAX); + + const size_t coders = LITERAL_CODER_SIZE << (lc + lp); + + for (size_t i = 0; i < coders; ++i) + bit_reset(probs[i]); + + return; +} + + +////////////////// +// Match length // +////////////////// + +// Minimum length of a match is two bytes. +#define MATCH_LEN_MIN 2 + +// Match length is encoded with 4, 5, or 10 bits. +// +// Length Bits +// 2-9 4 = Choice=0 + 3 bits +// 10-17 5 = Choice=1 + Choice2=0 + 3 bits +// 18-273 10 = Choice=1 + Choice2=1 + 8 bits +#define LEN_LOW_BITS 3 +#define LEN_LOW_SYMBOLS (1 << LEN_LOW_BITS) +#define LEN_MID_BITS 3 +#define LEN_MID_SYMBOLS (1 << LEN_MID_BITS) +#define LEN_HIGH_BITS 8 +#define LEN_HIGH_SYMBOLS (1 << LEN_HIGH_BITS) +#define LEN_SYMBOLS (LEN_LOW_SYMBOLS + LEN_MID_SYMBOLS + LEN_HIGH_SYMBOLS) + +// Maximum length of a match is 273 which is a result of the encoding +// described above. +#define MATCH_LEN_MAX (MATCH_LEN_MIN + LEN_SYMBOLS - 1) + + +//////////////////// +// Match distance // +//////////////////// + +// Different sets of probabilities are used for match distances that have very +// short match length: Lengths of 2, 3, and 4 bytes have a separate set of +// probabilities for each length. The matches with longer length use a shared +// set of probabilities. +#define DIST_STATES 4 + +// Macro to get the index of the appropriate probability array. +#define get_dist_state(len) \ + ((len) < DIST_STATES + MATCH_LEN_MIN \ + ? (len) - MATCH_LEN_MIN \ + : DIST_STATES - 1) + +// The highest two bits of a match distance (distance slot) are encoded +// using six bits. See fastpos.h for more explanation. +#define DIST_SLOT_BITS 6 +#define DIST_SLOTS (1 << DIST_SLOT_BITS) + +// Match distances up to 127 are fully encoded using probabilities. Since +// the highest two bits (distance slot) are always encoded using six bits, +// the distances 0-3 don't need any additional bits to encode, since the +// distance slot itself is the same as the actual distance. DIST_MODEL_START +// indicates the first distance slot where at least one additional bit is +// needed. +#define DIST_MODEL_START 4 + +// Match distances greater than 127 are encoded in three pieces: +// - distance slot: the highest two bits +// - direct bits: 2-26 bits below the highest two bits +// - alignment bits: four lowest bits +// +// Direct bits don't use any probabilities. +// +// The distance slot value of 14 is for distances 128-191 (see the table in +// fastpos.h to understand why). +#define DIST_MODEL_END 14 + +// Distance slots that indicate a distance <= 127. +#define FULL_DISTANCES_BITS (DIST_MODEL_END / 2) +#define FULL_DISTANCES (1 << FULL_DISTANCES_BITS) + +// For match distances greater than 127, only the highest two bits and the +// lowest four bits (alignment) is encoded using probabilities. +#define ALIGN_BITS 4 +#define ALIGN_SIZE (1 << ALIGN_BITS) +#define ALIGN_MASK (ALIGN_SIZE - 1) + +// LZMA remembers the four most recent match distances. Reusing these distances +// tends to take less space than re-encoding the actual distance value. +#define REPS 4 + +#endif diff --git a/src/liblzma/lzma/lzma_decoder.c b/src/liblzma/lzma/lzma_decoder.c new file mode 100644 index 0000000000..2088a2faa5 --- /dev/null +++ b/src/liblzma/lzma/lzma_decoder.c @@ -0,0 +1,1263 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_decoder.c +/// \brief LZMA decoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// Jia Tan +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lz_decoder.h" +#include "lzma_common.h" +#include "lzma_decoder.h" +#include "range_decoder.h" + +// The macros unroll loops with switch statements. +// Silence warnings about missing fall-through comments. +#if TUKLIB_GNUC_REQ(7, 0) || defined(__clang__) +# pragma GCC diagnostic ignored "-Wimplicit-fallthrough" +#endif + +// Minimum number of input bytes to safely decode one LZMA symbol. +// The worst case is that we decode 22 bits using probabilities and 26 +// direct bits. This may decode at maximum 20 bytes of input. +#define LZMA_IN_REQUIRED 20 + + +// Macros for (somewhat) size-optimized code. +// This is used to decode the match length (how many bytes must be repeated +// from the dictionary). This version is used in the Resumable mode and +// does not unroll any loops. +#define len_decode(target, ld, pos_state, seq) \ +do { \ +case seq ## _CHOICE: \ + rc_if_0_safe(ld.choice, seq ## _CHOICE) { \ + rc_update_0(ld.choice); \ + probs = ld.low[pos_state];\ + limit = LEN_LOW_SYMBOLS; \ + target = MATCH_LEN_MIN; \ + } else { \ + rc_update_1(ld.choice); \ +case seq ## _CHOICE2: \ + rc_if_0_safe(ld.choice2, seq ## _CHOICE2) { \ + rc_update_0(ld.choice2); \ + probs = ld.mid[pos_state]; \ + limit = LEN_MID_SYMBOLS; \ + target = MATCH_LEN_MIN + LEN_LOW_SYMBOLS; \ + } else { \ + rc_update_1(ld.choice2); \ + probs = ld.high; \ + limit = LEN_HIGH_SYMBOLS; \ + target = MATCH_LEN_MIN + LEN_LOW_SYMBOLS \ + + LEN_MID_SYMBOLS; \ + } \ + } \ + symbol = 1; \ +case seq ## _BITTREE: \ + do { \ + rc_bit_safe(probs[symbol], , , seq ## _BITTREE); \ + } while (symbol < limit); \ + target += symbol - limit; \ +} while (0) + + +// This is the faster version of the match length decoder that does not +// worry about being resumable. It unrolls the bittree decoding loop. +#define len_decode_fast(target, ld, pos_state) \ +do { \ + symbol = 1; \ + rc_if_0(ld.choice) { \ + rc_update_0(ld.choice); \ + rc_bittree3(ld.low[pos_state], \ + -LEN_LOW_SYMBOLS + MATCH_LEN_MIN); \ + target = symbol; \ + } else { \ + rc_update_1(ld.choice); \ + rc_if_0(ld.choice2) { \ + rc_update_0(ld.choice2); \ + rc_bittree3(ld.mid[pos_state], -LEN_MID_SYMBOLS \ + + MATCH_LEN_MIN + LEN_LOW_SYMBOLS); \ + target = symbol; \ + } else { \ + rc_update_1(ld.choice2); \ + rc_bittree8(ld.high, -LEN_HIGH_SYMBOLS \ + + MATCH_LEN_MIN \ + + LEN_LOW_SYMBOLS + LEN_MID_SYMBOLS); \ + target = symbol; \ + } \ + } \ +} while (0) + + +/// Length decoder probabilities; see comments in lzma_common.h. +typedef struct { + probability choice; + probability choice2; + probability low[POS_STATES_MAX][LEN_LOW_SYMBOLS]; + probability mid[POS_STATES_MAX][LEN_MID_SYMBOLS]; + probability high[LEN_HIGH_SYMBOLS]; +} lzma_length_decoder; + + +typedef struct { + /////////////////// + // Probabilities // + /////////////////// + + /// Literals; see comments in lzma_common.h. + probability literal[LITERAL_CODERS_MAX * LITERAL_CODER_SIZE]; + + /// If 1, it's a match. Otherwise it's a single 8-bit literal. + probability is_match[STATES][POS_STATES_MAX]; + + /// If 1, it's a repeated match. The distance is one of rep0 .. rep3. + probability is_rep[STATES]; + + /// If 0, distance of a repeated match is rep0. + /// Otherwise check is_rep1. + probability is_rep0[STATES]; + + /// If 0, distance of a repeated match is rep1. + /// Otherwise check is_rep2. + probability is_rep1[STATES]; + + /// If 0, distance of a repeated match is rep2. Otherwise it is rep3. + probability is_rep2[STATES]; + + /// If 1, the repeated match has length of one byte. Otherwise + /// the length is decoded from rep_len_decoder. + probability is_rep0_long[STATES][POS_STATES_MAX]; + + /// Probability tree for the highest two bits of the match distance. + /// There is a separate probability tree for match lengths of + /// 2 (i.e. MATCH_LEN_MIN), 3, 4, and [5, 273]. + probability dist_slot[DIST_STATES][DIST_SLOTS]; + + /// Probability trees for additional bits for match distance when the + /// distance is in the range [4, 127]. + probability pos_special[FULL_DISTANCES - DIST_MODEL_END]; + + /// Probability tree for the lowest four bits of a match distance + /// that is equal to or greater than 128. + probability pos_align[ALIGN_SIZE]; + + /// Length of a normal match + lzma_length_decoder match_len_decoder; + + /// Length of a repeated match + lzma_length_decoder rep_len_decoder; + + /////////////////// + // Decoder state // + /////////////////// + + // Range coder + lzma_range_decoder rc; + + // Types of the most recently seen LZMA symbols + lzma_lzma_state state; + + uint32_t rep0; ///< Distance of the latest match + uint32_t rep1; ///< Distance of second latest match + uint32_t rep2; ///< Distance of third latest match + uint32_t rep3; ///< Distance of fourth latest match + + uint32_t pos_mask; // (1U << pb) - 1 + uint32_t literal_context_bits; + uint32_t literal_mask; + + /// Uncompressed size as bytes, or LZMA_VLI_UNKNOWN if end of + /// payload marker is expected. + lzma_vli uncompressed_size; + + /// True if end of payload marker (EOPM) is allowed even when + /// uncompressed_size is known; false if EOPM must not be present. + /// This is ignored if uncompressed_size == LZMA_VLI_UNKNOWN. + bool allow_eopm; + + //////////////////////////////// + // State of incomplete symbol // + //////////////////////////////// + + /// Position where to continue the decoder loop + enum { + SEQ_NORMALIZE, + SEQ_IS_MATCH, + SEQ_LITERAL, + SEQ_LITERAL_MATCHED, + SEQ_LITERAL_WRITE, + SEQ_IS_REP, + SEQ_MATCH_LEN_CHOICE, + SEQ_MATCH_LEN_CHOICE2, + SEQ_MATCH_LEN_BITTREE, + SEQ_DIST_SLOT, + SEQ_DIST_MODEL, + SEQ_DIRECT, + SEQ_ALIGN, + SEQ_EOPM, + SEQ_IS_REP0, + SEQ_SHORTREP, + SEQ_IS_REP0_LONG, + SEQ_IS_REP1, + SEQ_IS_REP2, + SEQ_REP_LEN_CHOICE, + SEQ_REP_LEN_CHOICE2, + SEQ_REP_LEN_BITTREE, + SEQ_COPY, + } sequence; + + /// Base of the current probability tree + probability *probs; + + /// Symbol being decoded. This is also used as an index variable in + /// bittree decoders: probs[symbol] + uint32_t symbol; + + /// Used as a loop termination condition on bittree decoders and + /// direct bits decoder. + uint32_t limit; + + /// Matched literal decoder: 0x100 or 0 to help avoiding branches. + /// Bittree reverse decoders: Offset of the next bit: 1 << offset + uint32_t offset; + + /// If decoding a literal: match byte. + /// If decoding a match: length of the match. + uint32_t len; +} lzma_lzma1_decoder; + + +static lzma_ret +lzma_decode(void *coder_ptr, lzma_dict *restrict dictptr, + const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size) +{ + lzma_lzma1_decoder *restrict coder = coder_ptr; + + //////////////////// + // Initialization // + //////////////////// + + { + const lzma_ret ret = rc_read_init( + &coder->rc, in, in_pos, in_size); + if (ret != LZMA_STREAM_END) + return ret; + } + + /////////////// + // Variables // + /////////////// + + // Making local copies of often-used variables improves both + // speed and readability. + + lzma_dict dict = *dictptr; + + const size_t dict_start = dict.pos; + + // Range decoder + rc_to_local(coder->rc, *in_pos, LZMA_IN_REQUIRED); + + // State + uint32_t state = coder->state; + uint32_t rep0 = coder->rep0; + uint32_t rep1 = coder->rep1; + uint32_t rep2 = coder->rep2; + uint32_t rep3 = coder->rep3; + + const uint32_t pos_mask = coder->pos_mask; + + // These variables are actually needed only if we last time ran + // out of input in the middle of the decoder loop. + probability *probs = coder->probs; + uint32_t symbol = coder->symbol; + uint32_t limit = coder->limit; + uint32_t offset = coder->offset; + uint32_t len = coder->len; + + const uint32_t literal_mask = coder->literal_mask; + const uint32_t literal_context_bits = coder->literal_context_bits; + + // Temporary variables + uint32_t pos_state = dict.pos & pos_mask; + + lzma_ret ret = LZMA_OK; + + // This is true when the next LZMA symbol is allowed to be EOPM. + // That is, if this is false, then EOPM is considered + // an invalid symbol and we will return LZMA_DATA_ERROR. + // + // EOPM is always required (not just allowed) when + // the uncompressed size isn't known. When uncompressed size + // is known, eopm_is_valid may be set to true later. + bool eopm_is_valid = coder->uncompressed_size == LZMA_VLI_UNKNOWN; + + // If uncompressed size is known and there is enough output space + // to decode all the data, limit the available buffer space so that + // the main loop won't try to decode past the end of the stream. + bool might_finish_without_eopm = false; + if (coder->uncompressed_size != LZMA_VLI_UNKNOWN + && coder->uncompressed_size <= dict.limit - dict.pos) { + dict.limit = dict.pos + (size_t)(coder->uncompressed_size); + might_finish_without_eopm = true; + } + + // The main decoder loop. The "switch" is used to resume the decoder at + // correct location. Once resumed, the "switch" is no longer used. + // The decoder loops is split into two modes: + // + // 1 - Non-resumable mode (fast). This is used when it is guaranteed + // there is enough input to decode the next symbol. If the output + // limit is reached, then the decoder loop will save the place + // for the resumable mode to continue. This mode is not used if + // HAVE_SMALL is defined. This is faster than Resumable mode + // because it reduces the number of branches needed and allows + // for more compiler optimizations. + // + // 2 - Resumable mode (slow). This is used when a previous decoder + // loop did not have enough space in the input or output buffers + // to complete. It uses sequence enum values to set remind + // coder->sequence where to resume in the decoder loop. This + // is the only mode used when HAVE_SMALL is defined. + + switch (coder->sequence) + while (true) { + // Calculate new pos_state. This is skipped on the first loop + // since we already calculated it when setting up the local + // variables. + pos_state = dict.pos & pos_mask; + +#ifndef HAVE_SMALL + + /////////////////////////////// + // Non-resumable Mode (fast) // + /////////////////////////////// + + // Go to Resumable mode (1) if there is not enough input to + // safely decode any possible LZMA symbol or (2) if the + // dictionary is full, which may need special checks that + // are only done in the Resumable mode. + if (unlikely(!rc_is_fast_allowed() + || dict.pos == dict.limit)) + goto slow; + + // Decode the first bit from the next LZMA symbol. + // If the bit is a 0, then we handle it as a literal. + // If the bit is a 1, then it is a match of previously + // decoded data. + rc_if_0(coder->is_match[state][pos_state]) { + ///////////////////// + // Decode literal. // + ///////////////////// + + // Update the RC that we have decoded a 0. + rc_update_0(coder->is_match[state][pos_state]); + + // Get the correct probability array from lp and + // lc params. + probs = literal_subcoder(coder->literal, + literal_context_bits, literal_mask, + dict.pos, dict_get0(&dict)); + + if (is_literal_state(state)) { + update_literal_normal(state); + + // Decode literal without match byte. + rc_bittree8(probs, 0); + } else { + update_literal_matched(state); + + // Decode literal with match byte. + rc_matched_literal(probs, + dict_get(&dict, rep0)); + } + + // Write decoded literal to dictionary + dict_put(&dict, symbol); + continue; + } + + /////////////////// + // Decode match. // + /////////////////// + + // Instead of a new byte we are going to decode a + // distance-length pair. The distance represents how far + // back in the dictionary to begin copying. The length + // represents how many bytes to copy. + + rc_update_1(coder->is_match[state][pos_state]); + + rc_if_0(coder->is_rep[state]) { + /////////////////// + // Simple match. // + /////////////////// + + // Not a repeated match. In this case, + // the length (how many bytes to copy) must be + // decoded first. Then, the distance (where to + // start copying) is decoded. + // + // This is also how we know when we are done + // decoding. If the distance decodes to UINT32_MAX, + // then we know to stop decoding (end of payload + // marker). + + rc_update_0(coder->is_rep[state]); + update_match(state); + + // The latest three match distances are kept in + // memory in case there are repeated matches. + rep3 = rep2; + rep2 = rep1; + rep1 = rep0; + + // Decode the length of the match. + len_decode_fast(len, coder->match_len_decoder, + pos_state); + + // Next, decode the distance into rep0. + + // The next 6 bits determine how to decode the + // rest of the distance. + probs = coder->dist_slot[get_dist_state(len)]; + + rc_bittree6(probs, -DIST_SLOTS); + assert(symbol <= 63); + + if (symbol < DIST_MODEL_START) { + // If the decoded symbol is < DIST_MODEL_START + // then we use its value directly as the + // match distance. No other bits are needed. + // The only possible distance values + // are [0, 3]. + rep0 = symbol; + } else { + // Use the first two bits of symbol as the + // highest bits of the match distance. + + // "limit" represents the number of low bits + // to decode. + limit = (symbol >> 1) - 1; + assert(limit >= 1 && limit <= 30); + rep0 = 2 + (symbol & 1); + + if (symbol < DIST_MODEL_END) { + // When symbol is > DIST_MODEL_START, + // but symbol < DIST_MODEL_END, then + // it can decode distances between + // [4, 127]. + assert(limit <= 5); + rep0 <<= limit; + assert(rep0 <= 96); + + // -1 is fine, because we start + // decoding at probs[1], not probs[0]. + // NOTE: This violates the C standard, + // since we are doing pointer + // arithmetic past the beginning of + // the array. + assert((int32_t)(rep0 - symbol - 1) + >= -1); + assert((int32_t)(rep0 - symbol - 1) + <= 82); + probs = coder->pos_special + rep0 + - symbol - 1; + symbol = 1; + offset = 1; + + // Variable number (1-5) of bits + // from a reverse bittree. This + // isn't worth manual unrolling. + // + // NOTE: Making one or many of the + // variables (probs, symbol, offset, + // or limit) local here (instead of + // using those declared outside the + // main loop) can affect code size + // and performance which isn't a + // surprise but it's not so clear + // what is the best. + do { + rc_bit_add_if_1(probs, + rep0, offset); + offset <<= 1; + } while (--limit > 0); + } else { + // The distance is >= 128. Decode the + // lower bits without probabilities + // except the lowest four bits. + assert(symbol >= 14); + assert(limit >= 6); + + limit -= ALIGN_BITS; + assert(limit >= 2); + + rc_direct(rep0, limit); + + // Decode the lowest four bits using + // probabilities. + rep0 <<= ALIGN_BITS; + rc_bittree_rev4(coder->pos_align); + rep0 += symbol; + + // If the end of payload marker (EOPM) + // is detected, jump to the safe code. + // The EOPM handling isn't speed + // critical at all. + // + // A final normalization is needed + // after the EOPM (there can be a + // dummy byte to read in some cases). + // If the normalization was done here + // in the fast code, it would need to + // be taken into account in the value + // of LZMA_IN_REQUIRED. Using the + // safe code allows keeping + // LZMA_IN_REQUIRED as 20 instead of + // 21. + if (rep0 == UINT32_MAX) + goto eopm; + } + } + + // Validate the distance we just decoded. + if (unlikely(!dict_is_distance_valid(&dict, rep0))) { + ret = LZMA_DATA_ERROR; + goto out; + } + + } else { + rc_update_1(coder->is_rep[state]); + + ///////////////////// + // Repeated match. // + ///////////////////// + + // The match distance is a value that we have decoded + // recently. The latest four match distances are + // available as rep0, rep1, rep2 and rep3. We will + // now decode which of them is the new distance. + // + // There cannot be a match if we haven't produced + // any output, so check that first. + if (unlikely(!dict_is_distance_valid(&dict, 0))) { + ret = LZMA_DATA_ERROR; + goto out; + } + + rc_if_0(coder->is_rep0[state]) { + rc_update_0(coder->is_rep0[state]); + // The distance is rep0. + + // Decode the next bit to determine if 1 byte + // should be copied from rep0 distance or + // if the number of bytes needs to be decoded. + + // If the next bit is 0, then it is a + // "Short Rep Match" and only 1 bit is copied. + // Otherwise, the length of the match is + // decoded after the "else" statement. + rc_if_0(coder->is_rep0_long[state][pos_state]) { + rc_update_0(coder->is_rep0_long[ + state][pos_state]); + + update_short_rep(state); + dict_put(&dict, dict_get(&dict, rep0)); + continue; + } + + // Repeating more than one byte at + // distance of rep0. + rc_update_1(coder->is_rep0_long[ + state][pos_state]); + + } else { + rc_update_1(coder->is_rep0[state]); + + // The distance is rep1, rep2 or rep3. Once + // we find out which one of these three, it + // is stored to rep0 and rep1, rep2 and rep3 + // are updated accordingly. There is no + // "Short Rep Match" option, so the length + // of the match must always be decoded next. + rc_if_0(coder->is_rep1[state]) { + // The distance is rep1. + rc_update_0(coder->is_rep1[state]); + + const uint32_t distance = rep1; + rep1 = rep0; + rep0 = distance; + + } else { + rc_update_1(coder->is_rep1[state]); + + rc_if_0(coder->is_rep2[state]) { + // The distance is rep2. + rc_update_0(coder->is_rep2[ + state]); + + const uint32_t distance = rep2; + rep2 = rep1; + rep1 = rep0; + rep0 = distance; + + } else { + // The distance is rep3. + rc_update_1(coder->is_rep2[ + state]); + + const uint32_t distance = rep3; + rep3 = rep2; + rep2 = rep1; + rep1 = rep0; + rep0 = distance; + } + } + } + + update_long_rep(state); + + // Decode the length of the repeated match. + len_decode_fast(len, coder->rep_len_decoder, + pos_state); + } + + ///////////////////////////////// + // Repeat from history buffer. // + ///////////////////////////////// + + // The length is always between these limits. There is no way + // to trigger the algorithm to set len outside this range. + assert(len >= MATCH_LEN_MIN); + assert(len <= MATCH_LEN_MAX); + + // Repeat len bytes from distance of rep0. + if (unlikely(dict_repeat(&dict, rep0, &len))) { + coder->sequence = SEQ_COPY; + goto out; + } + + continue; + +slow: +#endif + /////////////////////////// + // Resumable Mode (slow) // + /////////////////////////// + + // This is very similar to Non-resumable Mode, so most of the + // comments are not repeated. The main differences are: + // - case labels are used to resume at the correct location. + // - Loops are not unrolled. + // - Range coder macros take an extra sequence argument + // so they can save to coder->sequence the location to + // resume in case there is not enough input. + case SEQ_NORMALIZE: + case SEQ_IS_MATCH: + if (unlikely(might_finish_without_eopm + && dict.pos == dict.limit)) { + // In rare cases there is a useless byte that needs + // to be read anyway. + rc_normalize_safe(SEQ_NORMALIZE); + + // If the range decoder state is such that we can + // be at the end of the LZMA stream, then the + // decoding is finished. + if (rc_is_finished(rc)) { + ret = LZMA_STREAM_END; + goto out; + } + + // If the caller hasn't allowed EOPM to be present + // together with known uncompressed size, then the + // LZMA stream is corrupt. + if (!coder->allow_eopm) { + ret = LZMA_DATA_ERROR; + goto out; + } + + // Otherwise continue decoding with the expectation + // that the next LZMA symbol is EOPM. + eopm_is_valid = true; + } + + rc_if_0_safe(coder->is_match[state][pos_state], SEQ_IS_MATCH) { + ///////////////////// + // Decode literal. // + ///////////////////// + + rc_update_0(coder->is_match[state][pos_state]); + + probs = literal_subcoder(coder->literal, + literal_context_bits, literal_mask, + dict.pos, dict_get0(&dict)); + symbol = 1; + + if (is_literal_state(state)) { + update_literal_normal(state); + + // Decode literal without match byte. + // The "slow" version does not unroll + // the loop. + case SEQ_LITERAL: + do { + rc_bit_safe(probs[symbol], , , + SEQ_LITERAL); + } while (symbol < (1 << 8)); + } else { + update_literal_matched(state); + + // Decode literal with match byte. + len = (uint32_t)(dict_get(&dict, rep0)) << 1; + + offset = 0x100; + + case SEQ_LITERAL_MATCHED: + do { + const uint32_t match_bit + = len & offset; + const uint32_t subcoder_index + = offset + match_bit + + symbol; + + rc_bit_safe(probs[subcoder_index], + offset &= ~match_bit, + offset &= match_bit, + SEQ_LITERAL_MATCHED); + + // It seems to be faster to do this + // here instead of putting it to the + // beginning of the loop and then + // putting the "case" in the middle + // of the loop. + len <<= 1; + + } while (symbol < (1 << 8)); + } + + case SEQ_LITERAL_WRITE: + if (dict_put_safe(&dict, symbol)) { + coder->sequence = SEQ_LITERAL_WRITE; + goto out; + } + + continue; + } + + /////////////////// + // Decode match. // + /////////////////// + + rc_update_1(coder->is_match[state][pos_state]); + + case SEQ_IS_REP: + rc_if_0_safe(coder->is_rep[state], SEQ_IS_REP) { + /////////////////// + // Simple match. // + /////////////////// + + rc_update_0(coder->is_rep[state]); + update_match(state); + + rep3 = rep2; + rep2 = rep1; + rep1 = rep0; + + len_decode(len, coder->match_len_decoder, + pos_state, SEQ_MATCH_LEN); + + probs = coder->dist_slot[get_dist_state(len)]; + symbol = 1; + + case SEQ_DIST_SLOT: + do { + rc_bit_safe(probs[symbol], , , SEQ_DIST_SLOT); + } while (symbol < DIST_SLOTS); + + symbol -= DIST_SLOTS; + assert(symbol <= 63); + + if (symbol < DIST_MODEL_START) { + rep0 = symbol; + } else { + limit = (symbol >> 1) - 1; + assert(limit >= 1 && limit <= 30); + rep0 = 2 + (symbol & 1); + + if (symbol < DIST_MODEL_END) { + assert(limit <= 5); + rep0 <<= limit; + assert(rep0 <= 96); + // -1 is fine, because we start + // decoding at probs[1], not probs[0]. + // NOTE: This violates the C standard, + // since we are doing pointer + // arithmetic past the beginning of + // the array. + assert((int32_t)(rep0 - symbol - 1) + >= -1); + assert((int32_t)(rep0 - symbol - 1) + <= 82); + probs = coder->pos_special + rep0 + - symbol - 1; + symbol = 1; + offset = 0; + case SEQ_DIST_MODEL: + do { + rc_bit_safe(probs[symbol], , + rep0 += 1U << offset, + SEQ_DIST_MODEL); + } while (++offset < limit); + } else { + assert(symbol >= 14); + assert(limit >= 6); + limit -= ALIGN_BITS; + assert(limit >= 2); + case SEQ_DIRECT: + rc_direct_safe(rep0, limit, + SEQ_DIRECT); + + rep0 <<= ALIGN_BITS; + symbol = 0; + offset = 1; + case SEQ_ALIGN: + do { + rc_bit_last_safe( + coder->pos_align[ + offset + + symbol], + , + symbol += offset, + SEQ_ALIGN); + offset <<= 1; + } while (offset < ALIGN_SIZE); + + rep0 += symbol; + + if (rep0 == UINT32_MAX) { + // End of payload marker was + // found. It may only be + // present if + // - uncompressed size is + // unknown or + // - after known uncompressed + // size amount of bytes has + // been decompressed and + // caller has indicated + // that EOPM might be used + // (it's not allowed in + // LZMA2). +#ifndef HAVE_SMALL +eopm: +#endif + if (!eopm_is_valid) { + ret = LZMA_DATA_ERROR; + goto out; + } + + case SEQ_EOPM: + // LZMA1 stream with + // end-of-payload marker. + rc_normalize_safe(SEQ_EOPM); + ret = rc_is_finished(rc) + ? LZMA_STREAM_END + : LZMA_DATA_ERROR; + goto out; + } + } + } + + if (unlikely(!dict_is_distance_valid(&dict, rep0))) { + ret = LZMA_DATA_ERROR; + goto out; + } + + } else { + ///////////////////// + // Repeated match. // + ///////////////////// + + rc_update_1(coder->is_rep[state]); + + if (unlikely(!dict_is_distance_valid(&dict, 0))) { + ret = LZMA_DATA_ERROR; + goto out; + } + + case SEQ_IS_REP0: + rc_if_0_safe(coder->is_rep0[state], SEQ_IS_REP0) { + rc_update_0(coder->is_rep0[state]); + + case SEQ_IS_REP0_LONG: + rc_if_0_safe(coder->is_rep0_long + [state][pos_state], + SEQ_IS_REP0_LONG) { + rc_update_0(coder->is_rep0_long[ + state][pos_state]); + + update_short_rep(state); + + case SEQ_SHORTREP: + if (dict_put_safe(&dict, + dict_get(&dict, + rep0))) { + coder->sequence = SEQ_SHORTREP; + goto out; + } + + continue; + } + + rc_update_1(coder->is_rep0_long[ + state][pos_state]); + + } else { + rc_update_1(coder->is_rep0[state]); + + case SEQ_IS_REP1: + rc_if_0_safe(coder->is_rep1[state], SEQ_IS_REP1) { + rc_update_0(coder->is_rep1[state]); + + const uint32_t distance = rep1; + rep1 = rep0; + rep0 = distance; + + } else { + rc_update_1(coder->is_rep1[state]); + case SEQ_IS_REP2: + rc_if_0_safe(coder->is_rep2[state], + SEQ_IS_REP2) { + rc_update_0(coder->is_rep2[ + state]); + + const uint32_t distance = rep2; + rep2 = rep1; + rep1 = rep0; + rep0 = distance; + + } else { + rc_update_1(coder->is_rep2[ + state]); + + const uint32_t distance = rep3; + rep3 = rep2; + rep2 = rep1; + rep1 = rep0; + rep0 = distance; + } + } + } + + update_long_rep(state); + + len_decode(len, coder->rep_len_decoder, + pos_state, SEQ_REP_LEN); + } + + ///////////////////////////////// + // Repeat from history buffer. // + ///////////////////////////////// + + assert(len >= MATCH_LEN_MIN); + assert(len <= MATCH_LEN_MAX); + + case SEQ_COPY: + if (unlikely(dict_repeat(&dict, rep0, &len))) { + coder->sequence = SEQ_COPY; + goto out; + } + } + +out: + // Save state + + // NOTE: Must not copy dict.limit. + dictptr->pos = dict.pos; + dictptr->full = dict.full; + + rc_from_local(coder->rc, *in_pos); + + coder->state = state; + coder->rep0 = rep0; + coder->rep1 = rep1; + coder->rep2 = rep2; + coder->rep3 = rep3; + + coder->probs = probs; + coder->symbol = symbol; + coder->limit = limit; + coder->offset = offset; + coder->len = len; + + // Update the remaining amount of uncompressed data if uncompressed + // size was known. + if (coder->uncompressed_size != LZMA_VLI_UNKNOWN) { + coder->uncompressed_size -= dict.pos - dict_start; + + // If we have gotten all the output but the decoder wants + // to write more output, the file is corrupt. There are + // three SEQ values where output is produced. + if (coder->uncompressed_size == 0 && ret == LZMA_OK + && (coder->sequence == SEQ_LITERAL_WRITE + || coder->sequence == SEQ_SHORTREP + || coder->sequence == SEQ_COPY)) + ret = LZMA_DATA_ERROR; + } + + if (ret == LZMA_STREAM_END) { + // Reset the range decoder so that it is ready to reinitialize + // for a new LZMA2 chunk. + rc_reset(coder->rc); + coder->sequence = SEQ_IS_MATCH; + } + + return ret; +} + + +static void +lzma_decoder_uncompressed(void *coder_ptr, lzma_vli uncompressed_size, + bool allow_eopm) +{ + lzma_lzma1_decoder *coder = coder_ptr; + coder->uncompressed_size = uncompressed_size; + coder->allow_eopm = allow_eopm; +} + + +static void +lzma_decoder_reset(void *coder_ptr, const void *opt) +{ + lzma_lzma1_decoder *coder = coder_ptr; + const lzma_options_lzma *options = opt; + + // NOTE: We assume that lc/lp/pb are valid since they were + // successfully decoded with lzma_lzma_decode_properties(). + + // Calculate pos_mask. We don't need pos_bits as is for anything. + coder->pos_mask = (1U << options->pb) - 1; + + // Initialize the literal decoder. + literal_init(coder->literal, options->lc, options->lp); + + coder->literal_context_bits = options->lc; + coder->literal_mask = literal_mask_calc(options->lc, options->lp); + + // State + coder->state = STATE_LIT_LIT; + coder->rep0 = 0; + coder->rep1 = 0; + coder->rep2 = 0; + coder->rep3 = 0; + coder->pos_mask = (1U << options->pb) - 1; + + // Range decoder + rc_reset(coder->rc); + + // Bit and bittree decoders + for (uint32_t i = 0; i < STATES; ++i) { + for (uint32_t j = 0; j <= coder->pos_mask; ++j) { + bit_reset(coder->is_match[i][j]); + bit_reset(coder->is_rep0_long[i][j]); + } + + bit_reset(coder->is_rep[i]); + bit_reset(coder->is_rep0[i]); + bit_reset(coder->is_rep1[i]); + bit_reset(coder->is_rep2[i]); + } + + for (uint32_t i = 0; i < DIST_STATES; ++i) + bittree_reset(coder->dist_slot[i], DIST_SLOT_BITS); + + for (uint32_t i = 0; i < FULL_DISTANCES - DIST_MODEL_END; ++i) + bit_reset(coder->pos_special[i]); + + bittree_reset(coder->pos_align, ALIGN_BITS); + + // Len decoders (also bit/bittree) + const uint32_t num_pos_states = 1U << options->pb; + bit_reset(coder->match_len_decoder.choice); + bit_reset(coder->match_len_decoder.choice2); + bit_reset(coder->rep_len_decoder.choice); + bit_reset(coder->rep_len_decoder.choice2); + + for (uint32_t pos_state = 0; pos_state < num_pos_states; ++pos_state) { + bittree_reset(coder->match_len_decoder.low[pos_state], + LEN_LOW_BITS); + bittree_reset(coder->match_len_decoder.mid[pos_state], + LEN_MID_BITS); + + bittree_reset(coder->rep_len_decoder.low[pos_state], + LEN_LOW_BITS); + bittree_reset(coder->rep_len_decoder.mid[pos_state], + LEN_MID_BITS); + } + + bittree_reset(coder->match_len_decoder.high, LEN_HIGH_BITS); + bittree_reset(coder->rep_len_decoder.high, LEN_HIGH_BITS); + + coder->sequence = SEQ_IS_MATCH; + coder->probs = NULL; + coder->symbol = 0; + coder->limit = 0; + coder->offset = 0; + coder->len = 0; + + return; +} + + +extern lzma_ret +lzma_lzma_decoder_create(lzma_lz_decoder *lz, const lzma_allocator *allocator, + const lzma_options_lzma *options, lzma_lz_options *lz_options) +{ + if (lz->coder == NULL) { + lz->coder = lzma_alloc(sizeof(lzma_lzma1_decoder), allocator); + if (lz->coder == NULL) + return LZMA_MEM_ERROR; + + lz->code = &lzma_decode; + lz->reset = &lzma_decoder_reset; + lz->set_uncompressed = &lzma_decoder_uncompressed; + } + + // All dictionary sizes are OK here. LZ decoder will take care of + // the special cases. + lz_options->dict_size = options->dict_size; + lz_options->preset_dict = options->preset_dict; + lz_options->preset_dict_size = options->preset_dict_size; + + return LZMA_OK; +} + + +/// Allocate and initialize LZMA decoder. This is used only via LZ +/// initialization (lzma_lzma_decoder_init() passes function pointer to +/// the LZ initialization). +static lzma_ret +lzma_decoder_init(lzma_lz_decoder *lz, const lzma_allocator *allocator, + lzma_vli id, const void *options, lzma_lz_options *lz_options) +{ + if (!is_lclppb_valid(options)) + return LZMA_PROG_ERROR; + + lzma_vli uncomp_size = LZMA_VLI_UNKNOWN; + bool allow_eopm = true; + + if (id == LZMA_FILTER_LZMA1EXT) { + const lzma_options_lzma *opt = options; + + // Only one flag is supported. + if (opt->ext_flags & ~LZMA_LZMA1EXT_ALLOW_EOPM) + return LZMA_OPTIONS_ERROR; + + // FIXME? Using lzma_vli instead of uint64_t is weird because + // this has nothing to do with .xz headers and variable-length + // integer encoding. On the other hand, using LZMA_VLI_UNKNOWN + // instead of UINT64_MAX is clearer when unknown size is + // meant. A problem with using lzma_vli is that now we + // allow > LZMA_VLI_MAX which is fine in this file but + // it's still confusing. Note that alone_decoder.c also + // allows > LZMA_VLI_MAX when setting uncompressed size. + uncomp_size = opt->ext_size_low + + ((uint64_t)(opt->ext_size_high) << 32); + allow_eopm = (opt->ext_flags & LZMA_LZMA1EXT_ALLOW_EOPM) != 0 + || uncomp_size == LZMA_VLI_UNKNOWN; + } + + return_if_error(lzma_lzma_decoder_create( + lz, allocator, options, lz_options)); + + lzma_decoder_reset(lz->coder, options); + lzma_decoder_uncompressed(lz->coder, uncomp_size, allow_eopm); + + return LZMA_OK; +} + + +extern lzma_ret +lzma_lzma_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + // LZMA can only be the last filter in the chain. This is enforced + // by the raw_decoder initialization. + assert(filters[1].init == NULL); + + return lzma_lz_decoder_init(next, allocator, filters, + &lzma_decoder_init); +} + + +extern bool +lzma_lzma_lclppb_decode(lzma_options_lzma *options, uint8_t byte) +{ + if (byte > (4 * 5 + 4) * 9 + 8) + return true; + + // See the file format specification to understand this. + options->pb = byte / (9 * 5); + byte -= options->pb * 9 * 5; + options->lp = byte / 9; + options->lc = byte - options->lp * 9; + + return options->lc + options->lp > LZMA_LCLP_MAX; +} + + +extern uint64_t +lzma_lzma_decoder_memusage_nocheck(const void *options) +{ + const lzma_options_lzma *const opt = options; + return sizeof(lzma_lzma1_decoder) + + lzma_lz_decoder_memusage(opt->dict_size); +} + + +extern uint64_t +lzma_lzma_decoder_memusage(const void *options) +{ + if (!is_lclppb_valid(options)) + return UINT64_MAX; + + return lzma_lzma_decoder_memusage_nocheck(options); +} + + +extern lzma_ret +lzma_lzma_props_decode(void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size) +{ + if (props_size != 5) + return LZMA_OPTIONS_ERROR; + + lzma_options_lzma *opt + = lzma_alloc(sizeof(lzma_options_lzma), allocator); + if (opt == NULL) + return LZMA_MEM_ERROR; + + if (lzma_lzma_lclppb_decode(opt, props[0])) + goto error; + + // All dictionary sizes are accepted, including zero. LZ decoder + // will automatically use a dictionary at least a few KiB even if + // a smaller dictionary is requested. + opt->dict_size = read32le(props + 1); + + opt->preset_dict = NULL; + opt->preset_dict_size = 0; + + *options = opt; + + return LZMA_OK; + +error: + lzma_free(opt, allocator); + return LZMA_OPTIONS_ERROR; +} diff --git a/src/liblzma/lzma/lzma_decoder.h b/src/liblzma/lzma/lzma_decoder.h new file mode 100644 index 0000000000..9730f56fc2 --- /dev/null +++ b/src/liblzma/lzma/lzma_decoder.h @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_decoder.h +/// \brief LZMA decoder API +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZMA_DECODER_H +#define LZMA_LZMA_DECODER_H + +#include "common.h" + + +/// Allocates and initializes LZMA decoder +extern lzma_ret lzma_lzma_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern uint64_t lzma_lzma_decoder_memusage(const void *options); + +extern lzma_ret lzma_lzma_props_decode( + void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size); + + +/// \brief Decodes the LZMA Properties byte (lc/lp/pb) +/// +/// \return true if error occurred, false on success +/// +extern bool lzma_lzma_lclppb_decode( + lzma_options_lzma *options, uint8_t byte); + + +#ifdef LZMA_LZ_DECODER_H +/// Allocate and setup function pointers only. This is used by LZMA1 and +/// LZMA2 decoders. +extern lzma_ret lzma_lzma_decoder_create( + lzma_lz_decoder *lz, const lzma_allocator *allocator, + const lzma_options_lzma *opt, lzma_lz_options *lz_options); + +/// Gets memory usage without validating lc/lp/pb. This is used by LZMA2 +/// decoder, because raw LZMA2 decoding doesn't need lc/lp/pb. +extern uint64_t lzma_lzma_decoder_memusage_nocheck(const void *options); + +#endif + +#endif diff --git a/src/liblzma/lzma/lzma_encoder.c b/src/liblzma/lzma/lzma_encoder.c new file mode 100644 index 0000000000..543ca321c3 --- /dev/null +++ b/src/liblzma/lzma/lzma_encoder.c @@ -0,0 +1,786 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_encoder.c +/// \brief LZMA encoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma2_encoder.h" +#include "lzma_encoder_private.h" +#include "fastpos.h" + + +///////////// +// Literal // +///////////// + +static inline void +literal_matched(lzma_range_encoder *rc, probability *subcoder, + uint32_t match_byte, uint32_t symbol) +{ + uint32_t offset = 0x100; + symbol += UINT32_C(1) << 8; + + do { + match_byte <<= 1; + const uint32_t match_bit = match_byte & offset; + const uint32_t subcoder_index + = offset + match_bit + (symbol >> 8); + const uint32_t bit = (symbol >> 7) & 1; + rc_bit(rc, &subcoder[subcoder_index], bit); + + symbol <<= 1; + offset &= ~(match_byte ^ symbol); + + } while (symbol < (UINT32_C(1) << 16)); +} + + +static inline void +literal(lzma_lzma1_encoder *coder, lzma_mf *mf, uint32_t position) +{ + // Locate the literal byte to be encoded and the subcoder. + const uint8_t cur_byte = mf->buffer[ + mf->read_pos - mf->read_ahead]; + probability *subcoder = literal_subcoder(coder->literal, + coder->literal_context_bits, coder->literal_mask, + position, mf->buffer[mf->read_pos - mf->read_ahead - 1]); + + if (is_literal_state(coder->state)) { + // Previous LZMA-symbol was a literal. Encode a normal + // literal without a match byte. + update_literal_normal(coder->state); + rc_bittree(&coder->rc, subcoder, 8, cur_byte); + } else { + // Previous LZMA-symbol was a match. Use the last byte of + // the match as a "match byte". That is, compare the bits + // of the current literal and the match byte. + update_literal_matched(coder->state); + const uint8_t match_byte = mf->buffer[ + mf->read_pos - coder->reps[0] - 1 + - mf->read_ahead]; + literal_matched(&coder->rc, subcoder, match_byte, cur_byte); + } +} + + +////////////////// +// Match length // +////////////////// + +static void +length_update_prices(lzma_length_encoder *lc, const uint32_t pos_state) +{ + const uint32_t table_size = lc->table_size; + lc->counters[pos_state] = table_size; + + const uint32_t a0 = rc_bit_0_price(lc->choice); + const uint32_t a1 = rc_bit_1_price(lc->choice); + const uint32_t b0 = a1 + rc_bit_0_price(lc->choice2); + const uint32_t b1 = a1 + rc_bit_1_price(lc->choice2); + uint32_t *const prices = lc->prices[pos_state]; + + uint32_t i; + for (i = 0; i < table_size && i < LEN_LOW_SYMBOLS; ++i) + prices[i] = a0 + rc_bittree_price(lc->low[pos_state], + LEN_LOW_BITS, i); + + for (; i < table_size && i < LEN_LOW_SYMBOLS + LEN_MID_SYMBOLS; ++i) + prices[i] = b0 + rc_bittree_price(lc->mid[pos_state], + LEN_MID_BITS, i - LEN_LOW_SYMBOLS); + + for (; i < table_size; ++i) + prices[i] = b1 + rc_bittree_price(lc->high, LEN_HIGH_BITS, + i - LEN_LOW_SYMBOLS - LEN_MID_SYMBOLS); + + return; +} + + +static inline void +length(lzma_range_encoder *rc, lzma_length_encoder *lc, + const uint32_t pos_state, uint32_t len, const bool fast_mode) +{ + assert(len <= MATCH_LEN_MAX); + len -= MATCH_LEN_MIN; + + if (len < LEN_LOW_SYMBOLS) { + rc_bit(rc, &lc->choice, 0); + rc_bittree(rc, lc->low[pos_state], LEN_LOW_BITS, len); + } else { + rc_bit(rc, &lc->choice, 1); + len -= LEN_LOW_SYMBOLS; + + if (len < LEN_MID_SYMBOLS) { + rc_bit(rc, &lc->choice2, 0); + rc_bittree(rc, lc->mid[pos_state], LEN_MID_BITS, len); + } else { + rc_bit(rc, &lc->choice2, 1); + len -= LEN_MID_SYMBOLS; + rc_bittree(rc, lc->high, LEN_HIGH_BITS, len); + } + } + + // Only getoptimum uses the prices so don't update the table when + // in fast mode. + if (!fast_mode) + if (--lc->counters[pos_state] == 0) + length_update_prices(lc, pos_state); +} + + +/////////// +// Match // +/////////// + +static inline void +match(lzma_lzma1_encoder *coder, const uint32_t pos_state, + const uint32_t distance, const uint32_t len) +{ + update_match(coder->state); + + length(&coder->rc, &coder->match_len_encoder, pos_state, len, + coder->fast_mode); + + const uint32_t dist_slot = get_dist_slot(distance); + const uint32_t dist_state = get_dist_state(len); + rc_bittree(&coder->rc, coder->dist_slot[dist_state], + DIST_SLOT_BITS, dist_slot); + + if (dist_slot >= DIST_MODEL_START) { + const uint32_t footer_bits = (dist_slot >> 1) - 1; + const uint32_t base = (2 | (dist_slot & 1)) << footer_bits; + const uint32_t dist_reduced = distance - base; + + if (dist_slot < DIST_MODEL_END) { + // Careful here: base - dist_slot - 1 can be -1, but + // rc_bittree_reverse starts at probs[1], not probs[0]. + rc_bittree_reverse(&coder->rc, + coder->dist_special + base - dist_slot - 1, + footer_bits, dist_reduced); + } else { + rc_direct(&coder->rc, dist_reduced >> ALIGN_BITS, + footer_bits - ALIGN_BITS); + rc_bittree_reverse( + &coder->rc, coder->dist_align, + ALIGN_BITS, dist_reduced & ALIGN_MASK); + ++coder->align_price_count; + } + } + + coder->reps[3] = coder->reps[2]; + coder->reps[2] = coder->reps[1]; + coder->reps[1] = coder->reps[0]; + coder->reps[0] = distance; + ++coder->match_price_count; +} + + +//////////////////// +// Repeated match // +//////////////////// + +static inline void +rep_match(lzma_lzma1_encoder *coder, const uint32_t pos_state, + const uint32_t rep, const uint32_t len) +{ + if (rep == 0) { + rc_bit(&coder->rc, &coder->is_rep0[coder->state], 0); + rc_bit(&coder->rc, + &coder->is_rep0_long[coder->state][pos_state], + len != 1); + } else { + const uint32_t distance = coder->reps[rep]; + rc_bit(&coder->rc, &coder->is_rep0[coder->state], 1); + + if (rep == 1) { + rc_bit(&coder->rc, &coder->is_rep1[coder->state], 0); + } else { + rc_bit(&coder->rc, &coder->is_rep1[coder->state], 1); + rc_bit(&coder->rc, &coder->is_rep2[coder->state], + rep - 2); + + if (rep == 3) + coder->reps[3] = coder->reps[2]; + + coder->reps[2] = coder->reps[1]; + } + + coder->reps[1] = coder->reps[0]; + coder->reps[0] = distance; + } + + if (len == 1) { + update_short_rep(coder->state); + } else { + length(&coder->rc, &coder->rep_len_encoder, pos_state, len, + coder->fast_mode); + update_long_rep(coder->state); + } +} + + +////////// +// Main // +////////// + +static void +encode_symbol(lzma_lzma1_encoder *coder, lzma_mf *mf, + uint32_t back, uint32_t len, uint32_t position) +{ + const uint32_t pos_state = position & coder->pos_mask; + + if (back == UINT32_MAX) { + // Literal i.e. eight-bit byte + assert(len == 1); + rc_bit(&coder->rc, + &coder->is_match[coder->state][pos_state], 0); + literal(coder, mf, position); + } else { + // Some type of match + rc_bit(&coder->rc, + &coder->is_match[coder->state][pos_state], 1); + + if (back < REPS) { + // It's a repeated match i.e. the same distance + // has been used earlier. + rc_bit(&coder->rc, &coder->is_rep[coder->state], 1); + rep_match(coder, pos_state, back, len); + } else { + // Normal match + rc_bit(&coder->rc, &coder->is_rep[coder->state], 0); + match(coder, pos_state, back - REPS, len); + } + } + + assert(mf->read_ahead >= len); + mf->read_ahead -= len; +} + + +static bool +encode_init(lzma_lzma1_encoder *coder, lzma_mf *mf) +{ + assert(mf_position(mf) == 0); + assert(coder->uncomp_size == 0); + + if (mf->read_pos == mf->read_limit) { + if (mf->action == LZMA_RUN) + return false; // We cannot do anything. + + // We are finishing (we cannot get here when flushing). + assert(mf->write_pos == mf->read_pos); + assert(mf->action == LZMA_FINISH); + } else { + // Do the actual initialization. The first LZMA symbol must + // always be a literal. + mf_skip(mf, 1); + mf->read_ahead = 0; + rc_bit(&coder->rc, &coder->is_match[0][0], 0); + rc_bittree(&coder->rc, coder->literal + 0, 8, mf->buffer[0]); + ++coder->uncomp_size; + } + + // Initialization is done (except if empty file). + coder->is_initialized = true; + + return true; +} + + +static void +encode_eopm(lzma_lzma1_encoder *coder, uint32_t position) +{ + const uint32_t pos_state = position & coder->pos_mask; + rc_bit(&coder->rc, &coder->is_match[coder->state][pos_state], 1); + rc_bit(&coder->rc, &coder->is_rep[coder->state], 0); + match(coder, pos_state, UINT32_MAX, MATCH_LEN_MIN); +} + + +/// Number of bytes that a single encoding loop in lzma_lzma_encode() can +/// consume from the dictionary. This limit comes from lzma_lzma_optimum() +/// and may need to be updated if that function is significantly modified. +#define LOOP_INPUT_MAX (OPTS + 1) + + +extern lzma_ret +lzma_lzma_encode(lzma_lzma1_encoder *restrict coder, lzma_mf *restrict mf, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, uint32_t limit) +{ + // Initialize the stream if no data has been encoded yet. + if (!coder->is_initialized && !encode_init(coder, mf)) + return LZMA_OK; + + // Encode pending output bytes from the range encoder. + // At the start of the stream, encode_init() encodes one literal. + // Later there can be pending output only with LZMA1 because LZMA2 + // ensures that there is always enough output space. Thus when using + // LZMA2, rc_encode() calls in this function will always return false. + if (rc_encode(&coder->rc, out, out_pos, out_size)) { + // We don't get here with LZMA2. + assert(limit == UINT32_MAX); + return LZMA_OK; + } + + // If the range encoder was flushed in an earlier call to this + // function but there wasn't enough output buffer space, those + // bytes would have now been encoded by the above rc_encode() call + // and the stream has now been finished. This can only happen with + // LZMA1 as LZMA2 always provides enough output buffer space. + if (coder->is_flushed) { + assert(limit == UINT32_MAX); + return LZMA_STREAM_END; + } + + while (true) { + // With LZMA2 we need to take care that compressed size of + // a chunk doesn't get too big. + // FIXME? Check if this could be improved. + if (limit != UINT32_MAX + && (mf->read_pos - mf->read_ahead >= limit + || *out_pos + rc_pending(&coder->rc) + >= LZMA2_CHUNK_MAX + - LOOP_INPUT_MAX)) + break; + + // Check that there is some input to process. + if (mf->read_pos >= mf->read_limit) { + if (mf->action == LZMA_RUN) + return LZMA_OK; + + if (mf->read_ahead == 0) + break; + } + + // Get optimal match (repeat position and length). + // Value ranges for pos: + // - [0, REPS): repeated match + // - [REPS, UINT32_MAX): + // match at (pos - REPS) + // - UINT32_MAX: not a match but a literal + // Value ranges for len: + // - [MATCH_LEN_MIN, MATCH_LEN_MAX] + uint32_t len; + uint32_t back; + + if (coder->fast_mode) + lzma_lzma_optimum_fast(coder, mf, &back, &len); + else + lzma_lzma_optimum_normal(coder, mf, &back, &len, + (uint32_t)(coder->uncomp_size)); + + encode_symbol(coder, mf, back, len, + (uint32_t)(coder->uncomp_size)); + + // If output size limiting is active (out_limit != 0), check + // if encoding this LZMA symbol would make the output size + // exceed the specified limit. + if (coder->out_limit != 0 && rc_encode_dummy( + &coder->rc, coder->out_limit)) { + // The most recent LZMA symbol would make the output + // too big. Throw it away. + rc_forget(&coder->rc); + + // FIXME: Tell the LZ layer to not read more input as + // it would be waste of time. This doesn't matter if + // output-size-limited encoding is done with a single + // call though. + + break; + } + + // This symbol will be encoded so update the uncompressed size. + coder->uncomp_size += len; + + // Encode the LZMA symbol. + if (rc_encode(&coder->rc, out, out_pos, out_size)) { + // Once again, this can only happen with LZMA1. + assert(limit == UINT32_MAX); + return LZMA_OK; + } + } + + // Make the uncompressed size available to the application. + if (coder->uncomp_size_ptr != NULL) + *coder->uncomp_size_ptr = coder->uncomp_size; + + // LZMA2 doesn't use EOPM at LZMA level. + // + // Plain LZMA streams without EOPM aren't supported except when + // output size limiting is enabled. + if (coder->use_eopm) + encode_eopm(coder, (uint32_t)(coder->uncomp_size)); + + // Flush the remaining bytes from the range encoder. + rc_flush(&coder->rc); + + // Copy the remaining bytes to the output buffer. If there + // isn't enough output space, we will copy out the remaining + // bytes on the next call to this function. + if (rc_encode(&coder->rc, out, out_pos, out_size)) { + // This cannot happen with LZMA2. + assert(limit == UINT32_MAX); + + coder->is_flushed = true; + return LZMA_OK; + } + + return LZMA_STREAM_END; +} + + +static lzma_ret +lzma_encode(void *coder, lzma_mf *restrict mf, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size) +{ + // Plain LZMA has no support for sync-flushing. + if (unlikely(mf->action == LZMA_SYNC_FLUSH)) + return LZMA_OPTIONS_ERROR; + + return lzma_lzma_encode(coder, mf, out, out_pos, out_size, UINT32_MAX); +} + + +static lzma_ret +lzma_lzma_set_out_limit( + void *coder_ptr, uint64_t *uncomp_size, uint64_t out_limit) +{ + // Minimum output size is 5 bytes but that cannot hold any output + // so we use 6 bytes. + if (out_limit < 6) + return LZMA_BUF_ERROR; + + lzma_lzma1_encoder *coder = coder_ptr; + coder->out_limit = out_limit; + coder->uncomp_size_ptr = uncomp_size; + coder->use_eopm = false; + return LZMA_OK; +} + + +//////////////////// +// Initialization // +//////////////////// + +static bool +is_options_valid(const lzma_options_lzma *options) +{ + // Validate some of the options. LZ encoder validates nice_len too + // but we need a valid value here earlier. + return is_lclppb_valid(options) + && options->nice_len >= MATCH_LEN_MIN + && options->nice_len <= MATCH_LEN_MAX + && (options->mode == LZMA_MODE_FAST + || options->mode == LZMA_MODE_NORMAL); +} + + +static void +set_lz_options(lzma_lz_options *lz_options, const lzma_options_lzma *options) +{ + // LZ encoder initialization does the validation for these so we + // don't need to validate here. + lz_options->before_size = OPTS; + lz_options->dict_size = options->dict_size; + lz_options->after_size = LOOP_INPUT_MAX; + lz_options->match_len_max = MATCH_LEN_MAX; + lz_options->nice_len = my_max(mf_get_hash_bytes(options->mf), + options->nice_len); + lz_options->match_finder = options->mf; + lz_options->depth = options->depth; + lz_options->preset_dict = options->preset_dict; + lz_options->preset_dict_size = options->preset_dict_size; + return; +} + + +static void +length_encoder_reset(lzma_length_encoder *lencoder, + const uint32_t num_pos_states, const bool fast_mode) +{ + bit_reset(lencoder->choice); + bit_reset(lencoder->choice2); + + for (size_t pos_state = 0; pos_state < num_pos_states; ++pos_state) { + bittree_reset(lencoder->low[pos_state], LEN_LOW_BITS); + bittree_reset(lencoder->mid[pos_state], LEN_MID_BITS); + } + + bittree_reset(lencoder->high, LEN_HIGH_BITS); + + if (!fast_mode) + for (uint32_t pos_state = 0; pos_state < num_pos_states; + ++pos_state) + length_update_prices(lencoder, pos_state); + + return; +} + + +extern lzma_ret +lzma_lzma_encoder_reset(lzma_lzma1_encoder *coder, + const lzma_options_lzma *options) +{ + if (!is_options_valid(options)) + return LZMA_OPTIONS_ERROR; + + coder->pos_mask = (1U << options->pb) - 1; + coder->literal_context_bits = options->lc; + coder->literal_mask = literal_mask_calc(options->lc, options->lp); + + // Range coder + rc_reset(&coder->rc); + + // State + coder->state = STATE_LIT_LIT; + for (size_t i = 0; i < REPS; ++i) + coder->reps[i] = 0; + + literal_init(coder->literal, options->lc, options->lp); + + // Bit encoders + for (size_t i = 0; i < STATES; ++i) { + for (size_t j = 0; j <= coder->pos_mask; ++j) { + bit_reset(coder->is_match[i][j]); + bit_reset(coder->is_rep0_long[i][j]); + } + + bit_reset(coder->is_rep[i]); + bit_reset(coder->is_rep0[i]); + bit_reset(coder->is_rep1[i]); + bit_reset(coder->is_rep2[i]); + } + + for (size_t i = 0; i < FULL_DISTANCES - DIST_MODEL_END; ++i) + bit_reset(coder->dist_special[i]); + + // Bit tree encoders + for (size_t i = 0; i < DIST_STATES; ++i) + bittree_reset(coder->dist_slot[i], DIST_SLOT_BITS); + + bittree_reset(coder->dist_align, ALIGN_BITS); + + // Length encoders + length_encoder_reset(&coder->match_len_encoder, + 1U << options->pb, coder->fast_mode); + + length_encoder_reset(&coder->rep_len_encoder, + 1U << options->pb, coder->fast_mode); + + // Price counts are incremented every time appropriate probabilities + // are changed. price counts are set to zero when the price tables + // are updated, which is done when the appropriate price counts have + // big enough value, and lzma_mf.read_ahead == 0 which happens at + // least every OPTS (a few thousand) possible price count increments. + // + // By resetting price counts to UINT32_MAX / 2, we make sure that the + // price tables will be initialized before they will be used (since + // the value is definitely big enough), and that it is OK to increment + // price counts without risk of integer overflow (since UINT32_MAX / 2 + // is small enough). The current code doesn't increment price counts + // before initializing price tables, but it maybe done in future if + // we add support for saving the state between LZMA2 chunks. + coder->match_price_count = UINT32_MAX / 2; + coder->align_price_count = UINT32_MAX / 2; + + coder->opts_end_index = 0; + coder->opts_current_index = 0; + + return LZMA_OK; +} + + +extern lzma_ret +lzma_lzma_encoder_create(void **coder_ptr, const lzma_allocator *allocator, + lzma_vli id, const lzma_options_lzma *options, + lzma_lz_options *lz_options) +{ + assert(id == LZMA_FILTER_LZMA1 || id == LZMA_FILTER_LZMA1EXT + || id == LZMA_FILTER_LZMA2); + + // Allocate lzma_lzma1_encoder if it wasn't already allocated. + if (*coder_ptr == NULL) { + *coder_ptr = lzma_alloc(sizeof(lzma_lzma1_encoder), allocator); + if (*coder_ptr == NULL) + return LZMA_MEM_ERROR; + } + + lzma_lzma1_encoder *coder = *coder_ptr; + + // Set compression mode. Note that we haven't validated the options + // yet. Invalid options will get rejected by lzma_lzma_encoder_reset() + // call at the end of this function. + switch (options->mode) { + case LZMA_MODE_FAST: + coder->fast_mode = true; + break; + + case LZMA_MODE_NORMAL: { + coder->fast_mode = false; + + // Set dist_table_size. + // Round the dictionary size up to next 2^n. + // + // Currently the maximum encoder dictionary size + // is 1.5 GiB due to lz_encoder.c and here we need + // to be below 2 GiB to make the rounded up value + // fit in an uint32_t and avoid an infinite while-loop + // (and undefined behavior due to a too large shift). + // So do the same check as in LZ encoder, + // limiting to 1.5 GiB. + if (options->dict_size > (UINT32_C(1) << 30) + + (UINT32_C(1) << 29)) + return LZMA_OPTIONS_ERROR; + + uint32_t log_size = 0; + while ((UINT32_C(1) << log_size) < options->dict_size) + ++log_size; + + coder->dist_table_size = log_size * 2; + + // Length encoders' price table size + const uint32_t nice_len = my_max( + mf_get_hash_bytes(options->mf), + options->nice_len); + + coder->match_len_encoder.table_size + = nice_len + 1 - MATCH_LEN_MIN; + coder->rep_len_encoder.table_size + = nice_len + 1 - MATCH_LEN_MIN; + break; + } + + default: + return LZMA_OPTIONS_ERROR; + } + + // We don't need to write the first byte as literal if there is + // a non-empty preset dictionary. encode_init() wouldn't even work + // if there is a non-empty preset dictionary, because encode_init() + // assumes that position is zero and previous byte is also zero. + coder->is_initialized = options->preset_dict != NULL + && options->preset_dict_size > 0; + coder->is_flushed = false; + coder->uncomp_size = 0; + coder->uncomp_size_ptr = NULL; + + // Output size limiting is disabled by default. + coder->out_limit = 0; + + // Determine if end marker is wanted: + // - It is never used with LZMA2. + // - It is always used with LZMA_FILTER_LZMA1 (unless + // lzma_lzma_set_out_limit() is called later). + // - LZMA_FILTER_LZMA1EXT has a flag for it in the options. + coder->use_eopm = (id == LZMA_FILTER_LZMA1); + if (id == LZMA_FILTER_LZMA1EXT) { + // Check if unsupported flags are present. + if (options->ext_flags & ~LZMA_LZMA1EXT_ALLOW_EOPM) + return LZMA_OPTIONS_ERROR; + + coder->use_eopm = (options->ext_flags + & LZMA_LZMA1EXT_ALLOW_EOPM) != 0; + + // TODO? As long as there are no filters that change the size + // of the data, it is enough to look at lzma_stream.total_in + // after encoding has been finished to know the uncompressed + // size of the LZMA1 stream. But in the future there could be + // filters that change the size of the data and then total_in + // doesn't work as the LZMA1 stream size might be different + // due to another filter in the chain. The problem is simple + // to solve: Add another flag to ext_flags and then set + // coder->uncomp_size_ptr to the address stored in + // lzma_options_lzma.reserved_ptr2 (or _ptr1). + } + + set_lz_options(lz_options, options); + + return lzma_lzma_encoder_reset(coder, options); +} + + +static lzma_ret +lzma_encoder_init(lzma_lz_encoder *lz, const lzma_allocator *allocator, + lzma_vli id, const void *options, lzma_lz_options *lz_options) +{ + if (options == NULL) + return LZMA_PROG_ERROR; + + lz->code = &lzma_encode; + lz->set_out_limit = &lzma_lzma_set_out_limit; + return lzma_lzma_encoder_create( + &lz->coder, allocator, id, options, lz_options); +} + + +extern lzma_ret +lzma_lzma_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return lzma_lz_encoder_init( + next, allocator, filters, &lzma_encoder_init); +} + + +extern uint64_t +lzma_lzma_encoder_memusage(const void *options) +{ + if (!is_options_valid(options)) + return UINT64_MAX; + + lzma_lz_options lz_options; + set_lz_options(&lz_options, options); + + const uint64_t lz_memusage = lzma_lz_encoder_memusage(&lz_options); + if (lz_memusage == UINT64_MAX) + return UINT64_MAX; + + return (uint64_t)(sizeof(lzma_lzma1_encoder)) + lz_memusage; +} + + +extern bool +lzma_lzma_lclppb_encode(const lzma_options_lzma *options, uint8_t *byte) +{ + if (!is_lclppb_valid(options)) + return true; + + *byte = (options->pb * 5 + options->lp) * 9 + options->lc; + assert(*byte <= (4 * 5 + 4) * 9 + 8); + + return false; +} + + +#ifdef HAVE_ENCODER_LZMA1 +extern lzma_ret +lzma_lzma_props_encode(const void *options, uint8_t *out) +{ + if (options == NULL) + return LZMA_PROG_ERROR; + + const lzma_options_lzma *const opt = options; + + if (lzma_lzma_lclppb_encode(opt, out)) + return LZMA_PROG_ERROR; + + write32le(out + 1, opt->dict_size); + + return LZMA_OK; +} +#endif + + +extern LZMA_API(lzma_bool) +lzma_mode_is_supported(lzma_mode mode) +{ + return mode == LZMA_MODE_FAST || mode == LZMA_MODE_NORMAL; +} diff --git a/src/liblzma/lzma/lzma_encoder.h b/src/liblzma/lzma/lzma_encoder.h new file mode 100644 index 0000000000..e8ae807930 --- /dev/null +++ b/src/liblzma/lzma/lzma_encoder.h @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_encoder.h +/// \brief LZMA encoder API +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZMA_ENCODER_H +#define LZMA_LZMA_ENCODER_H + +#include "common.h" + + +typedef struct lzma_lzma1_encoder_s lzma_lzma1_encoder; + + +extern lzma_ret lzma_lzma_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern uint64_t lzma_lzma_encoder_memusage(const void *options); + +extern lzma_ret lzma_lzma_props_encode(const void *options, uint8_t *out); + + +/// Encodes lc/lp/pb into one byte. Returns false on success and true on error. +extern bool lzma_lzma_lclppb_encode( + const lzma_options_lzma *options, uint8_t *byte); + + +#ifdef LZMA_LZ_ENCODER_H + +/// Initializes raw LZMA encoder; this is used by LZMA2. +extern lzma_ret lzma_lzma_encoder_create( + void **coder_ptr, const lzma_allocator *allocator, + lzma_vli id, const lzma_options_lzma *options, + lzma_lz_options *lz_options); + + +/// Resets an already initialized LZMA encoder; this is used by LZMA2. +extern lzma_ret lzma_lzma_encoder_reset( + lzma_lzma1_encoder *coder, const lzma_options_lzma *options); + + +extern lzma_ret lzma_lzma_encode(lzma_lzma1_encoder *restrict coder, + lzma_mf *restrict mf, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, + uint32_t read_limit); + +#endif + +#endif diff --git a/src/liblzma/lzma/lzma_encoder_optimum_fast.c b/src/liblzma/lzma/lzma_encoder_optimum_fast.c new file mode 100644 index 0000000000..0f063d5be7 --- /dev/null +++ b/src/liblzma/lzma/lzma_encoder_optimum_fast.c @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_encoder_optimum_fast.c +// +// Author: Igor Pavlov +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma_encoder_private.h" +#include "memcmplen.h" + + +#define change_pair(small_dist, big_dist) \ + (((big_dist) >> 7) > (small_dist)) + + +extern void +lzma_lzma_optimum_fast(lzma_lzma1_encoder *restrict coder, + lzma_mf *restrict mf, + uint32_t *restrict back_res, uint32_t *restrict len_res) +{ + const uint32_t nice_len = mf->nice_len; + + uint32_t len_main; + uint32_t matches_count; + if (mf->read_ahead == 0) { + len_main = mf_find(mf, &matches_count, coder->matches); + } else { + assert(mf->read_ahead == 1); + len_main = coder->longest_match_length; + matches_count = coder->matches_count; + } + + const uint8_t *buf = mf_ptr(mf) - 1; + const uint32_t buf_avail = my_min(mf_avail(mf) + 1, MATCH_LEN_MAX); + + if (buf_avail < 2) { + // There's not enough input left to encode a match. + *back_res = UINT32_MAX; + *len_res = 1; + return; + } + + // Look for repeated matches; scan the previous four match distances + uint32_t rep_len = 0; + uint32_t rep_index = 0; + + for (uint32_t i = 0; i < REPS; ++i) { + // Pointer to the beginning of the match candidate + const uint8_t *const buf_back = buf - coder->reps[i] - 1; + + // If the first two bytes (2 == MATCH_LEN_MIN) do not match, + // this rep is not useful. + if (not_equal_16(buf, buf_back)) + continue; + + // The first two bytes matched. + // Calculate the length of the match. + const uint32_t len = lzma_memcmplen( + buf, buf_back, 2, buf_avail); + + // If we have found a repeated match that is at least + // nice_len long, return it immediately. + if (len >= nice_len) { + *back_res = i; + *len_res = len; + mf_skip(mf, len - 1); + return; + } + + if (len > rep_len) { + rep_index = i; + rep_len = len; + } + } + + // We didn't find a long enough repeated match. Encode it as a normal + // match if the match length is at least nice_len. + if (len_main >= nice_len) { + *back_res = coder->matches[matches_count - 1].dist + REPS; + *len_res = len_main; + mf_skip(mf, len_main - 1); + return; + } + + uint32_t back_main = 0; + if (len_main >= 2) { + back_main = coder->matches[matches_count - 1].dist; + + while (matches_count > 1 && len_main == + coder->matches[matches_count - 2].len + 1) { + if (!change_pair(coder->matches[ + matches_count - 2].dist, + back_main)) + break; + + --matches_count; + len_main = coder->matches[matches_count - 1].len; + back_main = coder->matches[matches_count - 1].dist; + } + + if (len_main == 2 && back_main >= 0x80) + len_main = 1; + } + + if (rep_len >= 2) { + if (rep_len + 1 >= len_main + || (rep_len + 2 >= len_main + && back_main > (UINT32_C(1) << 9)) + || (rep_len + 3 >= len_main + && back_main > (UINT32_C(1) << 15))) { + *back_res = rep_index; + *len_res = rep_len; + mf_skip(mf, rep_len - 1); + return; + } + } + + if (len_main < 2 || buf_avail <= 2) { + *back_res = UINT32_MAX; + *len_res = 1; + return; + } + + // Get the matches for the next byte. If we find a better match, + // the current byte is encoded as a literal. + coder->longest_match_length = mf_find(mf, + &coder->matches_count, coder->matches); + + if (coder->longest_match_length >= 2) { + const uint32_t new_dist = coder->matches[ + coder->matches_count - 1].dist; + + if ((coder->longest_match_length >= len_main + && new_dist < back_main) + || (coder->longest_match_length == len_main + 1 + && !change_pair(back_main, new_dist)) + || (coder->longest_match_length > len_main + 1) + || (coder->longest_match_length + 1 >= len_main + && len_main >= 3 + && change_pair(new_dist, back_main))) { + *back_res = UINT32_MAX; + *len_res = 1; + return; + } + } + + // In contrast to LZMA SDK, dictionary could not have been moved + // between mf_find() calls, thus it is safe to just increment + // the old buf pointer instead of recalculating it with mf_ptr(). + ++buf; + + const uint32_t limit = my_max(2, len_main - 1); + + for (uint32_t i = 0; i < REPS; ++i) { + if (memcmp(buf, buf - coder->reps[i] - 1, limit) == 0) { + *back_res = UINT32_MAX; + *len_res = 1; + return; + } + } + + *back_res = back_main + REPS; + *len_res = len_main; + mf_skip(mf, len_main - 2); + return; +} diff --git a/src/liblzma/lzma/lzma_encoder_optimum_normal.c b/src/liblzma/lzma/lzma_encoder_optimum_normal.c new file mode 100644 index 0000000000..a6c0398f3a --- /dev/null +++ b/src/liblzma/lzma/lzma_encoder_optimum_normal.c @@ -0,0 +1,858 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_encoder_optimum_normal.c +// +// Author: Igor Pavlov +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma_encoder_private.h" +#include "fastpos.h" +#include "memcmplen.h" + + +//////////// +// Prices // +//////////// + +static uint32_t +get_literal_price(const lzma_lzma1_encoder *const coder, const uint32_t pos, + const uint32_t prev_byte, const bool match_mode, + uint32_t match_byte, uint32_t symbol) +{ + const probability *const subcoder = literal_subcoder(coder->literal, + coder->literal_context_bits, coder->literal_mask, + pos, prev_byte); + + uint32_t price = 0; + + if (!match_mode) { + price = rc_bittree_price(subcoder, 8, symbol); + } else { + uint32_t offset = 0x100; + symbol += UINT32_C(1) << 8; + + do { + match_byte <<= 1; + + const uint32_t match_bit = match_byte & offset; + const uint32_t subcoder_index + = offset + match_bit + (symbol >> 8); + const uint32_t bit = (symbol >> 7) & 1; + price += rc_bit_price(subcoder[subcoder_index], bit); + + symbol <<= 1; + offset &= ~(match_byte ^ symbol); + + } while (symbol < (UINT32_C(1) << 16)); + } + + return price; +} + + +static inline uint32_t +get_len_price(const lzma_length_encoder *const lencoder, + const uint32_t len, const uint32_t pos_state) +{ + // NOTE: Unlike the other price tables, length prices are updated + // in lzma_encoder.c + return lencoder->prices[pos_state][len - MATCH_LEN_MIN]; +} + + +static inline uint32_t +get_short_rep_price(const lzma_lzma1_encoder *const coder, + const lzma_lzma_state state, const uint32_t pos_state) +{ + return rc_bit_0_price(coder->is_rep0[state]) + + rc_bit_0_price(coder->is_rep0_long[state][pos_state]); +} + + +static inline uint32_t +get_pure_rep_price(const lzma_lzma1_encoder *const coder, const uint32_t rep_index, + const lzma_lzma_state state, uint32_t pos_state) +{ + uint32_t price; + + if (rep_index == 0) { + price = rc_bit_0_price(coder->is_rep0[state]); + price += rc_bit_1_price(coder->is_rep0_long[state][pos_state]); + } else { + price = rc_bit_1_price(coder->is_rep0[state]); + + if (rep_index == 1) { + price += rc_bit_0_price(coder->is_rep1[state]); + } else { + price += rc_bit_1_price(coder->is_rep1[state]); + price += rc_bit_price(coder->is_rep2[state], + rep_index - 2); + } + } + + return price; +} + + +static inline uint32_t +get_rep_price(const lzma_lzma1_encoder *const coder, const uint32_t rep_index, + const uint32_t len, const lzma_lzma_state state, + const uint32_t pos_state) +{ + return get_len_price(&coder->rep_len_encoder, len, pos_state) + + get_pure_rep_price(coder, rep_index, state, pos_state); +} + + +static inline uint32_t +get_dist_len_price(const lzma_lzma1_encoder *const coder, const uint32_t dist, + const uint32_t len, const uint32_t pos_state) +{ + const uint32_t dist_state = get_dist_state(len); + uint32_t price; + + if (dist < FULL_DISTANCES) { + price = coder->dist_prices[dist_state][dist]; + } else { + const uint32_t dist_slot = get_dist_slot_2(dist); + price = coder->dist_slot_prices[dist_state][dist_slot] + + coder->align_prices[dist & ALIGN_MASK]; + } + + price += get_len_price(&coder->match_len_encoder, len, pos_state); + + return price; +} + + +static void +fill_dist_prices(lzma_lzma1_encoder *coder) +{ + for (uint32_t dist_state = 0; dist_state < DIST_STATES; ++dist_state) { + + uint32_t *const dist_slot_prices + = coder->dist_slot_prices[dist_state]; + + // Price to encode the dist_slot. + for (uint32_t dist_slot = 0; + dist_slot < coder->dist_table_size; ++dist_slot) + dist_slot_prices[dist_slot] = rc_bittree_price( + coder->dist_slot[dist_state], + DIST_SLOT_BITS, dist_slot); + + // For matches with distance >= FULL_DISTANCES, add the price + // of the direct bits part of the match distance. (Align bits + // are handled by fill_align_prices()). + for (uint32_t dist_slot = DIST_MODEL_END; + dist_slot < coder->dist_table_size; + ++dist_slot) + dist_slot_prices[dist_slot] += rc_direct_price( + ((dist_slot >> 1) - 1) - ALIGN_BITS); + + // Distances in the range [0, 3] are fully encoded with + // dist_slot, so they are used for coder->dist_prices + // as is. + for (uint32_t i = 0; i < DIST_MODEL_START; ++i) + coder->dist_prices[dist_state][i] + = dist_slot_prices[i]; + } + + // Distances in the range [4, 127] depend on dist_slot and + // dist_special. We do this in a loop separate from the above + // loop to avoid redundant calls to get_dist_slot(). + for (uint32_t i = DIST_MODEL_START; i < FULL_DISTANCES; ++i) { + const uint32_t dist_slot = get_dist_slot(i); + const uint32_t footer_bits = ((dist_slot >> 1) - 1); + const uint32_t base = (2 | (dist_slot & 1)) << footer_bits; + const uint32_t price = rc_bittree_reverse_price( + coder->dist_special + base - dist_slot - 1, + footer_bits, i - base); + + for (uint32_t dist_state = 0; dist_state < DIST_STATES; + ++dist_state) + coder->dist_prices[dist_state][i] + = price + coder->dist_slot_prices[ + dist_state][dist_slot]; + } + + coder->match_price_count = 0; + return; +} + + +static void +fill_align_prices(lzma_lzma1_encoder *coder) +{ + for (uint32_t i = 0; i < ALIGN_SIZE; ++i) + coder->align_prices[i] = rc_bittree_reverse_price( + coder->dist_align, ALIGN_BITS, i); + + coder->align_price_count = 0; + return; +} + + +///////////// +// Optimal // +///////////// + +static inline void +make_literal(lzma_optimal *optimal) +{ + optimal->back_prev = UINT32_MAX; + optimal->prev_1_is_literal = false; +} + + +static inline void +make_short_rep(lzma_optimal *optimal) +{ + optimal->back_prev = 0; + optimal->prev_1_is_literal = false; +} + + +#define is_short_rep(optimal) \ + ((optimal).back_prev == 0) + + +static void +backward(lzma_lzma1_encoder *restrict coder, uint32_t *restrict len_res, + uint32_t *restrict back_res, uint32_t cur) +{ + coder->opts_end_index = cur; + + uint32_t pos_mem = coder->opts[cur].pos_prev; + uint32_t back_mem = coder->opts[cur].back_prev; + + do { + if (coder->opts[cur].prev_1_is_literal) { + make_literal(&coder->opts[pos_mem]); + coder->opts[pos_mem].pos_prev = pos_mem - 1; + + if (coder->opts[cur].prev_2) { + coder->opts[pos_mem - 1].prev_1_is_literal + = false; + coder->opts[pos_mem - 1].pos_prev + = coder->opts[cur].pos_prev_2; + coder->opts[pos_mem - 1].back_prev + = coder->opts[cur].back_prev_2; + } + } + + const uint32_t pos_prev = pos_mem; + const uint32_t back_cur = back_mem; + + back_mem = coder->opts[pos_prev].back_prev; + pos_mem = coder->opts[pos_prev].pos_prev; + + coder->opts[pos_prev].back_prev = back_cur; + coder->opts[pos_prev].pos_prev = cur; + cur = pos_prev; + + } while (cur != 0); + + coder->opts_current_index = coder->opts[0].pos_prev; + *len_res = coder->opts[0].pos_prev; + *back_res = coder->opts[0].back_prev; + + return; +} + + +////////// +// Main // +////////// + +static inline uint32_t +helper1(lzma_lzma1_encoder *restrict coder, lzma_mf *restrict mf, + uint32_t *restrict back_res, uint32_t *restrict len_res, + uint32_t position) +{ + const uint32_t nice_len = mf->nice_len; + + uint32_t len_main; + uint32_t matches_count; + + if (mf->read_ahead == 0) { + len_main = mf_find(mf, &matches_count, coder->matches); + } else { + assert(mf->read_ahead == 1); + len_main = coder->longest_match_length; + matches_count = coder->matches_count; + } + + const uint32_t buf_avail = my_min(mf_avail(mf) + 1, MATCH_LEN_MAX); + if (buf_avail < 2) { + *back_res = UINT32_MAX; + *len_res = 1; + return UINT32_MAX; + } + + const uint8_t *const buf = mf_ptr(mf) - 1; + + uint32_t rep_lens[REPS]; + uint32_t rep_max_index = 0; + + for (uint32_t i = 0; i < REPS; ++i) { + const uint8_t *const buf_back = buf - coder->reps[i] - 1; + + if (not_equal_16(buf, buf_back)) { + rep_lens[i] = 0; + continue; + } + + rep_lens[i] = lzma_memcmplen(buf, buf_back, 2, buf_avail); + + if (rep_lens[i] > rep_lens[rep_max_index]) + rep_max_index = i; + } + + if (rep_lens[rep_max_index] >= nice_len) { + *back_res = rep_max_index; + *len_res = rep_lens[rep_max_index]; + mf_skip(mf, *len_res - 1); + return UINT32_MAX; + } + + + if (len_main >= nice_len) { + *back_res = coder->matches[matches_count - 1].dist + REPS; + *len_res = len_main; + mf_skip(mf, len_main - 1); + return UINT32_MAX; + } + + const uint8_t current_byte = *buf; + const uint8_t match_byte = *(buf - coder->reps[0] - 1); + + if (len_main < 2 && current_byte != match_byte + && rep_lens[rep_max_index] < 2) { + *back_res = UINT32_MAX; + *len_res = 1; + return UINT32_MAX; + } + + coder->opts[0].state = coder->state; + + const uint32_t pos_state = position & coder->pos_mask; + + coder->opts[1].price = rc_bit_0_price( + coder->is_match[coder->state][pos_state]) + + get_literal_price(coder, position, buf[-1], + !is_literal_state(coder->state), + match_byte, current_byte); + + make_literal(&coder->opts[1]); + + const uint32_t match_price = rc_bit_1_price( + coder->is_match[coder->state][pos_state]); + const uint32_t rep_match_price = match_price + + rc_bit_1_price(coder->is_rep[coder->state]); + + if (match_byte == current_byte) { + const uint32_t short_rep_price = rep_match_price + + get_short_rep_price( + coder, coder->state, pos_state); + + if (short_rep_price < coder->opts[1].price) { + coder->opts[1].price = short_rep_price; + make_short_rep(&coder->opts[1]); + } + } + + const uint32_t len_end = my_max(len_main, rep_lens[rep_max_index]); + + if (len_end < 2) { + *back_res = coder->opts[1].back_prev; + *len_res = 1; + return UINT32_MAX; + } + + coder->opts[1].pos_prev = 0; + + for (uint32_t i = 0; i < REPS; ++i) + coder->opts[0].backs[i] = coder->reps[i]; + + uint32_t len = len_end; + do { + coder->opts[len].price = RC_INFINITY_PRICE; + } while (--len >= 2); + + + for (uint32_t i = 0; i < REPS; ++i) { + uint32_t rep_len = rep_lens[i]; + if (rep_len < 2) + continue; + + const uint32_t price = rep_match_price + get_pure_rep_price( + coder, i, coder->state, pos_state); + + do { + const uint32_t cur_and_len_price = price + + get_len_price( + &coder->rep_len_encoder, + rep_len, pos_state); + + if (cur_and_len_price < coder->opts[rep_len].price) { + coder->opts[rep_len].price = cur_and_len_price; + coder->opts[rep_len].pos_prev = 0; + coder->opts[rep_len].back_prev = i; + coder->opts[rep_len].prev_1_is_literal = false; + } + } while (--rep_len >= 2); + } + + + const uint32_t normal_match_price = match_price + + rc_bit_0_price(coder->is_rep[coder->state]); + + len = rep_lens[0] >= 2 ? rep_lens[0] + 1 : 2; + if (len <= len_main) { + uint32_t i = 0; + while (len > coder->matches[i].len) + ++i; + + for(; ; ++len) { + const uint32_t dist = coder->matches[i].dist; + const uint32_t cur_and_len_price = normal_match_price + + get_dist_len_price(coder, + dist, len, pos_state); + + if (cur_and_len_price < coder->opts[len].price) { + coder->opts[len].price = cur_and_len_price; + coder->opts[len].pos_prev = 0; + coder->opts[len].back_prev = dist + REPS; + coder->opts[len].prev_1_is_literal = false; + } + + if (len == coder->matches[i].len) + if (++i == matches_count) + break; + } + } + + return len_end; +} + + +static inline uint32_t +helper2(lzma_lzma1_encoder *coder, uint32_t *reps, const uint8_t *buf, + uint32_t len_end, uint32_t position, const uint32_t cur, + const uint32_t nice_len, const uint32_t buf_avail_full) +{ + uint32_t matches_count = coder->matches_count; + uint32_t new_len = coder->longest_match_length; + uint32_t pos_prev = coder->opts[cur].pos_prev; + lzma_lzma_state state; + + if (coder->opts[cur].prev_1_is_literal) { + --pos_prev; + + if (coder->opts[cur].prev_2) { + state = coder->opts[coder->opts[cur].pos_prev_2].state; + + if (coder->opts[cur].back_prev_2 < REPS) + update_long_rep(state); + else + update_match(state); + + } else { + state = coder->opts[pos_prev].state; + } + + update_literal(state); + + } else { + state = coder->opts[pos_prev].state; + } + + if (pos_prev == cur - 1) { + if (is_short_rep(coder->opts[cur])) + update_short_rep(state); + else + update_literal(state); + } else { + uint32_t pos; + if (coder->opts[cur].prev_1_is_literal + && coder->opts[cur].prev_2) { + pos_prev = coder->opts[cur].pos_prev_2; + pos = coder->opts[cur].back_prev_2; + update_long_rep(state); + } else { + pos = coder->opts[cur].back_prev; + if (pos < REPS) + update_long_rep(state); + else + update_match(state); + } + + if (pos < REPS) { + reps[0] = coder->opts[pos_prev].backs[pos]; + + uint32_t i; + for (i = 1; i <= pos; ++i) + reps[i] = coder->opts[pos_prev].backs[i - 1]; + + for (; i < REPS; ++i) + reps[i] = coder->opts[pos_prev].backs[i]; + + } else { + reps[0] = pos - REPS; + + for (uint32_t i = 1; i < REPS; ++i) + reps[i] = coder->opts[pos_prev].backs[i - 1]; + } + } + + coder->opts[cur].state = state; + + for (uint32_t i = 0; i < REPS; ++i) + coder->opts[cur].backs[i] = reps[i]; + + const uint32_t cur_price = coder->opts[cur].price; + + const uint8_t current_byte = *buf; + const uint8_t match_byte = *(buf - reps[0] - 1); + + const uint32_t pos_state = position & coder->pos_mask; + + const uint32_t cur_and_1_price = cur_price + + rc_bit_0_price(coder->is_match[state][pos_state]) + + get_literal_price(coder, position, buf[-1], + !is_literal_state(state), match_byte, current_byte); + + bool next_is_literal = false; + + if (cur_and_1_price < coder->opts[cur + 1].price) { + coder->opts[cur + 1].price = cur_and_1_price; + coder->opts[cur + 1].pos_prev = cur; + make_literal(&coder->opts[cur + 1]); + next_is_literal = true; + } + + const uint32_t match_price = cur_price + + rc_bit_1_price(coder->is_match[state][pos_state]); + const uint32_t rep_match_price = match_price + + rc_bit_1_price(coder->is_rep[state]); + + if (match_byte == current_byte + && !(coder->opts[cur + 1].pos_prev < cur + && coder->opts[cur + 1].back_prev == 0)) { + + const uint32_t short_rep_price = rep_match_price + + get_short_rep_price(coder, state, pos_state); + + if (short_rep_price <= coder->opts[cur + 1].price) { + coder->opts[cur + 1].price = short_rep_price; + coder->opts[cur + 1].pos_prev = cur; + make_short_rep(&coder->opts[cur + 1]); + next_is_literal = true; + } + } + + if (buf_avail_full < 2) + return len_end; + + const uint32_t buf_avail = my_min(buf_avail_full, nice_len); + + if (!next_is_literal && match_byte != current_byte) { // speed optimization + // try literal + rep0 + const uint8_t *const buf_back = buf - reps[0] - 1; + const uint32_t limit = my_min(buf_avail_full, nice_len + 1); + + const uint32_t len_test = lzma_memcmplen(buf, buf_back, 1, limit) - 1; + + if (len_test >= 2) { + lzma_lzma_state state_2 = state; + update_literal(state_2); + + const uint32_t pos_state_next = (position + 1) & coder->pos_mask; + const uint32_t next_rep_match_price = cur_and_1_price + + rc_bit_1_price(coder->is_match[state_2][pos_state_next]) + + rc_bit_1_price(coder->is_rep[state_2]); + + //for (; len_test >= 2; --len_test) { + const uint32_t offset = cur + 1 + len_test; + + while (len_end < offset) + coder->opts[++len_end].price = RC_INFINITY_PRICE; + + const uint32_t cur_and_len_price = next_rep_match_price + + get_rep_price(coder, 0, len_test, + state_2, pos_state_next); + + if (cur_and_len_price < coder->opts[offset].price) { + coder->opts[offset].price = cur_and_len_price; + coder->opts[offset].pos_prev = cur + 1; + coder->opts[offset].back_prev = 0; + coder->opts[offset].prev_1_is_literal = true; + coder->opts[offset].prev_2 = false; + } + //} + } + } + + + uint32_t start_len = 2; // speed optimization + + for (uint32_t rep_index = 0; rep_index < REPS; ++rep_index) { + const uint8_t *const buf_back = buf - reps[rep_index] - 1; + if (not_equal_16(buf, buf_back)) + continue; + + uint32_t len_test = lzma_memcmplen(buf, buf_back, 2, buf_avail); + + while (len_end < cur + len_test) + coder->opts[++len_end].price = RC_INFINITY_PRICE; + + const uint32_t len_test_temp = len_test; + const uint32_t price = rep_match_price + get_pure_rep_price( + coder, rep_index, state, pos_state); + + do { + const uint32_t cur_and_len_price = price + + get_len_price(&coder->rep_len_encoder, + len_test, pos_state); + + if (cur_and_len_price < coder->opts[cur + len_test].price) { + coder->opts[cur + len_test].price = cur_and_len_price; + coder->opts[cur + len_test].pos_prev = cur; + coder->opts[cur + len_test].back_prev = rep_index; + coder->opts[cur + len_test].prev_1_is_literal = false; + } + } while (--len_test >= 2); + + len_test = len_test_temp; + + if (rep_index == 0) + start_len = len_test + 1; + + + uint32_t len_test_2 = len_test + 1; + const uint32_t limit = my_min(buf_avail_full, + len_test_2 + nice_len); + // NOTE: len_test_2 may be greater than limit so the call to + // lzma_memcmplen() must be done conditionally. + if (len_test_2 < limit) + len_test_2 = lzma_memcmplen(buf, buf_back, len_test_2, limit); + + len_test_2 -= len_test + 1; + + if (len_test_2 >= 2) { + lzma_lzma_state state_2 = state; + update_long_rep(state_2); + + uint32_t pos_state_next = (position + len_test) & coder->pos_mask; + + const uint32_t cur_and_len_literal_price = price + + get_len_price(&coder->rep_len_encoder, + len_test, pos_state) + + rc_bit_0_price(coder->is_match[state_2][pos_state_next]) + + get_literal_price(coder, position + len_test, + buf[len_test - 1], true, + buf_back[len_test], buf[len_test]); + + update_literal(state_2); + + pos_state_next = (position + len_test + 1) & coder->pos_mask; + + const uint32_t next_rep_match_price = cur_and_len_literal_price + + rc_bit_1_price(coder->is_match[state_2][pos_state_next]) + + rc_bit_1_price(coder->is_rep[state_2]); + + //for(; len_test_2 >= 2; len_test_2--) { + const uint32_t offset = cur + len_test + 1 + len_test_2; + + while (len_end < offset) + coder->opts[++len_end].price = RC_INFINITY_PRICE; + + const uint32_t cur_and_len_price = next_rep_match_price + + get_rep_price(coder, 0, len_test_2, + state_2, pos_state_next); + + if (cur_and_len_price < coder->opts[offset].price) { + coder->opts[offset].price = cur_and_len_price; + coder->opts[offset].pos_prev = cur + len_test + 1; + coder->opts[offset].back_prev = 0; + coder->opts[offset].prev_1_is_literal = true; + coder->opts[offset].prev_2 = true; + coder->opts[offset].pos_prev_2 = cur; + coder->opts[offset].back_prev_2 = rep_index; + } + //} + } + } + + + //for (uint32_t len_test = 2; len_test <= new_len; ++len_test) + if (new_len > buf_avail) { + new_len = buf_avail; + + matches_count = 0; + while (new_len > coder->matches[matches_count].len) + ++matches_count; + + coder->matches[matches_count++].len = new_len; + } + + + if (new_len >= start_len) { + const uint32_t normal_match_price = match_price + + rc_bit_0_price(coder->is_rep[state]); + + while (len_end < cur + new_len) + coder->opts[++len_end].price = RC_INFINITY_PRICE; + + uint32_t i = 0; + while (start_len > coder->matches[i].len) + ++i; + + for (uint32_t len_test = start_len; ; ++len_test) { + const uint32_t cur_back = coder->matches[i].dist; + uint32_t cur_and_len_price = normal_match_price + + get_dist_len_price(coder, + cur_back, len_test, pos_state); + + if (cur_and_len_price < coder->opts[cur + len_test].price) { + coder->opts[cur + len_test].price = cur_and_len_price; + coder->opts[cur + len_test].pos_prev = cur; + coder->opts[cur + len_test].back_prev + = cur_back + REPS; + coder->opts[cur + len_test].prev_1_is_literal = false; + } + + if (len_test == coder->matches[i].len) { + // Try Match + Literal + Rep0 + const uint8_t *const buf_back = buf - cur_back - 1; + uint32_t len_test_2 = len_test + 1; + const uint32_t limit = my_min(buf_avail_full, + len_test_2 + nice_len); + + // NOTE: len_test_2 may be greater than limit + // so the call to lzma_memcmplen() must be + // done conditionally. + if (len_test_2 < limit) + len_test_2 = lzma_memcmplen(buf, buf_back, + len_test_2, limit); + + len_test_2 -= len_test + 1; + + if (len_test_2 >= 2) { + lzma_lzma_state state_2 = state; + update_match(state_2); + uint32_t pos_state_next + = (position + len_test) & coder->pos_mask; + + const uint32_t cur_and_len_literal_price = cur_and_len_price + + rc_bit_0_price( + coder->is_match[state_2][pos_state_next]) + + get_literal_price(coder, + position + len_test, + buf[len_test - 1], + true, + buf_back[len_test], + buf[len_test]); + + update_literal(state_2); + pos_state_next = (pos_state_next + 1) & coder->pos_mask; + + const uint32_t next_rep_match_price + = cur_and_len_literal_price + + rc_bit_1_price( + coder->is_match[state_2][pos_state_next]) + + rc_bit_1_price(coder->is_rep[state_2]); + + // for(; len_test_2 >= 2; --len_test_2) { + const uint32_t offset = cur + len_test + 1 + len_test_2; + + while (len_end < offset) + coder->opts[++len_end].price = RC_INFINITY_PRICE; + + cur_and_len_price = next_rep_match_price + + get_rep_price(coder, 0, len_test_2, + state_2, pos_state_next); + + if (cur_and_len_price < coder->opts[offset].price) { + coder->opts[offset].price = cur_and_len_price; + coder->opts[offset].pos_prev = cur + len_test + 1; + coder->opts[offset].back_prev = 0; + coder->opts[offset].prev_1_is_literal = true; + coder->opts[offset].prev_2 = true; + coder->opts[offset].pos_prev_2 = cur; + coder->opts[offset].back_prev_2 + = cur_back + REPS; + } + //} + } + + if (++i == matches_count) + break; + } + } + } + + return len_end; +} + + +extern void +lzma_lzma_optimum_normal(lzma_lzma1_encoder *restrict coder, + lzma_mf *restrict mf, + uint32_t *restrict back_res, uint32_t *restrict len_res, + uint32_t position) +{ + // If we have symbols pending, return the next pending symbol. + if (coder->opts_end_index != coder->opts_current_index) { + assert(mf->read_ahead > 0); + *len_res = coder->opts[coder->opts_current_index].pos_prev + - coder->opts_current_index; + *back_res = coder->opts[coder->opts_current_index].back_prev; + coder->opts_current_index = coder->opts[ + coder->opts_current_index].pos_prev; + return; + } + + // Update the price tables. In LZMA SDK <= 4.60 (and possibly later) + // this was done in both initialization function and in the main loop. + // In liblzma they were moved into this single place. + if (mf->read_ahead == 0) { + if (coder->match_price_count >= (1 << 7)) + fill_dist_prices(coder); + + if (coder->align_price_count >= ALIGN_SIZE) + fill_align_prices(coder); + } + + // TODO: This needs quite a bit of cleaning still. But splitting + // the original function into two pieces makes it at least a little + // more readable, since those two parts don't share many variables. + + uint32_t len_end = helper1(coder, mf, back_res, len_res, position); + if (len_end == UINT32_MAX) + return; + + uint32_t reps[REPS]; + memcpy(reps, coder->reps, sizeof(reps)); + + uint32_t cur; + for (cur = 1; cur < len_end; ++cur) { + assert(cur < OPTS); + + coder->longest_match_length = mf_find( + mf, &coder->matches_count, coder->matches); + + if (coder->longest_match_length >= mf->nice_len) + break; + + len_end = helper2(coder, reps, mf_ptr(mf) - 1, len_end, + position + cur, cur, mf->nice_len, + my_min(mf_avail(mf) + 1, OPTS - 1 - cur)); + } + + backward(coder, len_res, back_res, cur); + return; +} diff --git a/src/liblzma/lzma/lzma_encoder_presets.c b/src/liblzma/lzma/lzma_encoder_presets.c new file mode 100644 index 0000000000..e53483f995 --- /dev/null +++ b/src/liblzma/lzma/lzma_encoder_presets.c @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_encoder_presets.c +/// \brief Encoder presets +/// \note xz needs this even when only decoding is enabled. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" + + +extern LZMA_API(lzma_bool) +lzma_lzma_preset(lzma_options_lzma *options, uint32_t preset) +{ + const uint32_t level = preset & LZMA_PRESET_LEVEL_MASK; + const uint32_t flags = preset & ~LZMA_PRESET_LEVEL_MASK; + const uint32_t supported_flags = LZMA_PRESET_EXTREME; + + if (level > 9 || (flags & ~supported_flags)) + return true; + + options->preset_dict = NULL; + options->preset_dict_size = 0; + + options->lc = LZMA_LC_DEFAULT; + options->lp = LZMA_LP_DEFAULT; + options->pb = LZMA_PB_DEFAULT; + + static const uint8_t dict_pow2[] + = { 18, 20, 21, 22, 22, 23, 23, 24, 25, 26 }; + options->dict_size = UINT32_C(1) << dict_pow2[level]; + + if (level <= 3) { + options->mode = LZMA_MODE_FAST; + options->mf = level == 0 ? LZMA_MF_HC3 : LZMA_MF_HC4; + options->nice_len = level <= 1 ? 128 : 273; + static const uint8_t depths[] = { 4, 8, 24, 48 }; + options->depth = depths[level]; + } else { + options->mode = LZMA_MODE_NORMAL; + options->mf = LZMA_MF_BT4; + options->nice_len = level == 4 ? 16 : level == 5 ? 32 : 64; + options->depth = 0; + } + + if (flags & LZMA_PRESET_EXTREME) { + options->mode = LZMA_MODE_NORMAL; + options->mf = LZMA_MF_BT4; + if (level == 3 || level == 5) { + options->nice_len = 192; + options->depth = 0; + } else { + options->nice_len = 273; + options->depth = 512; + } + } + + return false; +} diff --git a/src/liblzma/lzma/lzma_encoder_private.h b/src/liblzma/lzma/lzma_encoder_private.h new file mode 100644 index 0000000000..eeea5e9c12 --- /dev/null +++ b/src/liblzma/lzma/lzma_encoder_private.h @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzma_encoder_private.h +/// \brief Private definitions for LZMA encoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZMA_ENCODER_PRIVATE_H +#define LZMA_LZMA_ENCODER_PRIVATE_H + +#include "lz_encoder.h" +#include "range_encoder.h" +#include "lzma_common.h" +#include "lzma_encoder.h" + + +// Macro to compare if the first two bytes in two buffers differ. This is +// needed in lzma_lzma_optimum_*() to test if the match is at least +// MATCH_LEN_MIN bytes. Unaligned access gives tiny gain so there's no +// reason to not use it when it is supported. +#ifdef TUKLIB_FAST_UNALIGNED_ACCESS +# define not_equal_16(a, b) (read16ne(a) != read16ne(b)) +#else +# define not_equal_16(a, b) \ + ((a)[0] != (b)[0] || (a)[1] != (b)[1]) +#endif + + +// Optimal - Number of entries in the optimum array. +#define OPTS (1 << 12) + + +typedef struct { + probability choice; + probability choice2; + probability low[POS_STATES_MAX][LEN_LOW_SYMBOLS]; + probability mid[POS_STATES_MAX][LEN_MID_SYMBOLS]; + probability high[LEN_HIGH_SYMBOLS]; + + uint32_t prices[POS_STATES_MAX][LEN_SYMBOLS]; + uint32_t table_size; + uint32_t counters[POS_STATES_MAX]; + +} lzma_length_encoder; + + +typedef struct { + lzma_lzma_state state; + + bool prev_1_is_literal; + bool prev_2; + + uint32_t pos_prev_2; + uint32_t back_prev_2; + + uint32_t price; + uint32_t pos_prev; // pos_next; + uint32_t back_prev; + + uint32_t backs[REPS]; + +} lzma_optimal; + + +struct lzma_lzma1_encoder_s { + /// Range encoder + lzma_range_encoder rc; + + /// Uncompressed size (doesn't include possible preset dictionary) + uint64_t uncomp_size; + + /// If non-zero, produce at most this much output. + /// Some input may then be missing from the output. + uint64_t out_limit; + + /// If the above out_limit is non-zero, *uncomp_size_ptr is set to + /// the amount of uncompressed data that we were able to fit + /// in the output buffer. + uint64_t *uncomp_size_ptr; + + /// State + lzma_lzma_state state; + + /// The four most recent match distances + uint32_t reps[REPS]; + + /// Array of match candidates + lzma_match matches[MATCH_LEN_MAX + 1]; + + /// Number of match candidates in matches[] + uint32_t matches_count; + + /// Variable to hold the length of the longest match between calls + /// to lzma_lzma_optimum_*(). + uint32_t longest_match_length; + + /// True if using getoptimumfast + bool fast_mode; + + /// True if the encoder has been initialized by encoding the first + /// byte as a literal. + bool is_initialized; + + /// True if the range encoder has been flushed, but not all bytes + /// have been written to the output buffer yet. + bool is_flushed; + + /// True if end of payload marker will be written. + bool use_eopm; + + uint32_t pos_mask; ///< (1 << pos_bits) - 1 + uint32_t literal_context_bits; + uint32_t literal_mask; + + // These are the same as in lzma_decoder.c. See comments there. + probability literal[LITERAL_CODERS_MAX * LITERAL_CODER_SIZE]; + probability is_match[STATES][POS_STATES_MAX]; + probability is_rep[STATES]; + probability is_rep0[STATES]; + probability is_rep1[STATES]; + probability is_rep2[STATES]; + probability is_rep0_long[STATES][POS_STATES_MAX]; + probability dist_slot[DIST_STATES][DIST_SLOTS]; + probability dist_special[FULL_DISTANCES - DIST_MODEL_END]; + probability dist_align[ALIGN_SIZE]; + + // These are the same as in lzma_decoder.c except that the encoders + // include also price tables. + lzma_length_encoder match_len_encoder; + lzma_length_encoder rep_len_encoder; + + // Price tables + uint32_t dist_slot_prices[DIST_STATES][DIST_SLOTS]; + uint32_t dist_prices[DIST_STATES][FULL_DISTANCES]; + uint32_t dist_table_size; + uint32_t match_price_count; + + uint32_t align_prices[ALIGN_SIZE]; + uint32_t align_price_count; + + // Optimal + uint32_t opts_end_index; + uint32_t opts_current_index; + lzma_optimal opts[OPTS]; +}; + + +extern void lzma_lzma_optimum_fast( + lzma_lzma1_encoder *restrict coder, lzma_mf *restrict mf, + uint32_t *restrict back_res, uint32_t *restrict len_res); + +extern void lzma_lzma_optimum_normal(lzma_lzma1_encoder *restrict coder, + lzma_mf *restrict mf, uint32_t *restrict back_res, + uint32_t *restrict len_res, uint32_t position); + +#endif diff --git a/src/liblzma/rangecoder/Makefile.inc b/src/liblzma/rangecoder/Makefile.inc new file mode 100644 index 0000000000..9a00e6e155 --- /dev/null +++ b/src/liblzma/rangecoder/Makefile.inc @@ -0,0 +1,17 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +EXTRA_DIST += rangecoder/price_tablegen.c + +liblzma_la_SOURCES += rangecoder/range_common.h + +if COND_ENCODER_LZMA1 +liblzma_la_SOURCES += \ + rangecoder/range_encoder.h \ + rangecoder/price.h \ + rangecoder/price_table.c +endif + +if COND_DECODER_LZMA1 +liblzma_la_SOURCES += rangecoder/range_decoder.h +endif diff --git a/src/liblzma/rangecoder/price.h b/src/liblzma/rangecoder/price.h new file mode 100644 index 0000000000..cce6bdae5f --- /dev/null +++ b/src/liblzma/rangecoder/price.h @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file price.h +/// \brief Probability price calculation +// +// Author: Igor Pavlov +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_PRICE_H +#define LZMA_PRICE_H + + +#define RC_MOVE_REDUCING_BITS 4 +#define RC_BIT_PRICE_SHIFT_BITS 4 +#define RC_PRICE_TABLE_SIZE (RC_BIT_MODEL_TOTAL >> RC_MOVE_REDUCING_BITS) + +#define RC_INFINITY_PRICE (UINT32_C(1) << 30) + + +/// Lookup table for the inline functions defined in this file. +lzma_attr_visibility_hidden +extern const uint8_t lzma_rc_prices[RC_PRICE_TABLE_SIZE]; + + +static inline uint32_t +rc_bit_price(const probability prob, const uint32_t bit) +{ + return lzma_rc_prices[(prob ^ ((UINT32_C(0) - bit) + & (RC_BIT_MODEL_TOTAL - 1))) >> RC_MOVE_REDUCING_BITS]; +} + + +static inline uint32_t +rc_bit_0_price(const probability prob) +{ + return lzma_rc_prices[prob >> RC_MOVE_REDUCING_BITS]; +} + + +static inline uint32_t +rc_bit_1_price(const probability prob) +{ + return lzma_rc_prices[(prob ^ (RC_BIT_MODEL_TOTAL - 1)) + >> RC_MOVE_REDUCING_BITS]; +} + + +static inline uint32_t +rc_bittree_price(const probability *const probs, + const uint32_t bit_levels, uint32_t symbol) +{ + uint32_t price = 0; + symbol += UINT32_C(1) << bit_levels; + + do { + const uint32_t bit = symbol & 1; + symbol >>= 1; + price += rc_bit_price(probs[symbol], bit); + } while (symbol != 1); + + return price; +} + + +static inline uint32_t +rc_bittree_reverse_price(const probability *const probs, + uint32_t bit_levels, uint32_t symbol) +{ + uint32_t price = 0; + uint32_t model_index = 1; + + do { + const uint32_t bit = symbol & 1; + symbol >>= 1; + price += rc_bit_price(probs[model_index], bit); + model_index = (model_index << 1) + bit; + } while (--bit_levels != 0); + + return price; +} + + +static inline uint32_t +rc_direct_price(const uint32_t bits) +{ + return bits << RC_BIT_PRICE_SHIFT_BITS; +} + +#endif diff --git a/src/liblzma/rangecoder/price_table.c b/src/liblzma/rangecoder/price_table.c new file mode 100644 index 0000000000..c33433f718 --- /dev/null +++ b/src/liblzma/rangecoder/price_table.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: 0BSD + +// This file has been generated by price_tablegen.c. + +#include "range_encoder.h" + +const uint8_t lzma_rc_prices[RC_PRICE_TABLE_SIZE] = { + 128, 103, 91, 84, 78, 73, 69, 66, + 63, 61, 58, 56, 54, 52, 51, 49, + 48, 46, 45, 44, 43, 42, 41, 40, + 39, 38, 37, 36, 35, 34, 34, 33, + 32, 31, 31, 30, 29, 29, 28, 28, + 27, 26, 26, 25, 25, 24, 24, 23, + 23, 22, 22, 22, 21, 21, 20, 20, + 19, 19, 19, 18, 18, 17, 17, 17, + 16, 16, 16, 15, 15, 15, 14, 14, + 14, 13, 13, 13, 12, 12, 12, 11, + 11, 11, 11, 10, 10, 10, 10, 9, + 9, 9, 9, 8, 8, 8, 8, 7, + 7, 7, 7, 6, 6, 6, 6, 5, + 5, 5, 5, 5, 4, 4, 4, 4, + 3, 3, 3, 3, 3, 2, 2, 2, + 2, 2, 2, 1, 1, 1, 1, 1 +}; diff --git a/src/liblzma/rangecoder/price_tablegen.c b/src/liblzma/rangecoder/price_tablegen.c new file mode 100644 index 0000000000..4b6ca37efa --- /dev/null +++ b/src/liblzma/rangecoder/price_tablegen.c @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file price_tablegen.c +/// \brief Probability price table generator +/// +/// Compiling: gcc -std=c99 -o price_tablegen price_tablegen.c +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include +#include + +// Make it compile without common.h. +#define BUILDING_PRICE_TABLEGEN +#define lzma_attr_visibility_hidden + +#include "range_common.h" +#include "price.h" + + +static uint32_t rc_prices[RC_PRICE_TABLE_SIZE]; + + +static void +init_price_table(void) +{ + for (uint32_t i = (UINT32_C(1) << RC_MOVE_REDUCING_BITS) / 2; + i < RC_BIT_MODEL_TOTAL; + i += (UINT32_C(1) << RC_MOVE_REDUCING_BITS)) { + const uint32_t cycles_bits = RC_BIT_PRICE_SHIFT_BITS; + uint32_t w = i; + uint32_t bit_count = 0; + + for (uint32_t j = 0; j < cycles_bits; ++j) { + w *= w; + bit_count <<= 1; + + while (w >= (UINT32_C(1) << 16)) { + w >>= 1; + ++bit_count; + } + } + + rc_prices[i >> RC_MOVE_REDUCING_BITS] + = (RC_BIT_MODEL_TOTAL_BITS << cycles_bits) + - 15 - bit_count; + } + + return; +} + + +static void +print_price_table(void) +{ + // Split the SPDX string so that it won't accidentally match + // when tools search for the string. + printf("// SPDX" "-License-Identifier" ": 0BSD\n\n" + "// This file has been generated by price_tablegen.c.\n\n" + "#include \"range_encoder.h\"\n\n" + "const uint8_t lzma_rc_prices[" + "RC_PRICE_TABLE_SIZE] = {"); + + const size_t array_size = sizeof(lzma_rc_prices) + / sizeof(lzma_rc_prices[0]); + for (size_t i = 0; i < array_size; ++i) { + if (i % 8 == 0) + printf("\n\t"); + + printf("%4" PRIu32, rc_prices[i]); + + if (i != array_size - 1) + printf(","); + } + + printf("\n};\n"); + + return; +} + + +int +main(void) +{ + init_price_table(); + print_price_table(); + return 0; +} diff --git a/src/liblzma/rangecoder/range_common.h b/src/liblzma/rangecoder/range_common.h new file mode 100644 index 0000000000..ac4dbe196f --- /dev/null +++ b/src/liblzma/rangecoder/range_common.h @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file range_common.h +/// \brief Common things for range encoder and decoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_RANGE_COMMON_H +#define LZMA_RANGE_COMMON_H + +// Skip common.h if building price_tablegen.c. +#ifndef BUILDING_PRICE_TABLEGEN +# include "common.h" +#endif + + +/////////////// +// Constants // +/////////////// + +#define RC_SHIFT_BITS 8 +#define RC_TOP_BITS 24 +#define RC_TOP_VALUE (UINT32_C(1) << RC_TOP_BITS) +#define RC_BIT_MODEL_TOTAL_BITS 11 +#define RC_BIT_MODEL_TOTAL (UINT32_C(1) << RC_BIT_MODEL_TOTAL_BITS) +#define RC_MOVE_BITS 5 + + +//////////// +// Macros // +//////////// + +// Resets the probability so that both 0 and 1 have probability of 50 % +#define bit_reset(prob) \ + prob = RC_BIT_MODEL_TOTAL >> 1 + +// This does the same for a complete bit tree. +// (A tree represented as an array.) +#define bittree_reset(probs, bit_levels) \ + for (uint32_t bt_i = 0; bt_i < (1 << (bit_levels)); ++bt_i) \ + bit_reset((probs)[bt_i]) + + +////////////////////// +// Type definitions // +////////////////////// + +/// \brief Type of probabilities used with range coder +/// +/// This needs to be at least 12-bit integer, so uint16_t is a logical choice. +/// However, on some architecture and compiler combinations, a bigger type +/// may give better speed, because the probability variables are accessed +/// a lot. On the other hand, bigger probability type increases cache +/// footprint, since there are 2 to 14 thousand probability variables in +/// LZMA (assuming the limit of lc + lp <= 4; with lc + lp <= 12 there +/// would be about 1.5 million variables). +/// +/// With malicious files, the initialization speed of the LZMA decoder can +/// become important. In that case, smaller probability variables mean that +/// there is less bytes to write to RAM, which makes initialization faster. +/// With big probability type, the initialization can become so slow that it +/// can be a problem e.g. for email servers doing virus scanning. +/// +/// I will be sticking to uint16_t unless some specific architectures +/// are *much* faster (20-50 %) with uint32_t. +/// +/// Update in 2024: The branchless C and x86-64 assembly was written so that +/// probability is assumed to be uint16_t. (In contrast, LZMA SDK 23.01 +/// assembly supports both types.) +typedef uint16_t probability; + +#endif diff --git a/src/liblzma/rangecoder/range_decoder.h b/src/liblzma/rangecoder/range_decoder.h new file mode 100644 index 0000000000..a8aca9077c --- /dev/null +++ b/src/liblzma/rangecoder/range_decoder.h @@ -0,0 +1,966 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file range_decoder.h +/// \brief Range Decoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_RANGE_DECODER_H +#define LZMA_RANGE_DECODER_H + +#include "range_common.h" + + +// Choose the range decoder variants to use using a bitmask. +// If no bits are set, only the basic version is used. +// If more than one version is selected for the same feature, +// the last one on the list below is used. +// +// Bitwise-or of the following enable branchless C versions: +// 0x01 normal bittrees +// 0x02 fixed-sized reverse bittrees +// 0x04 variable-sized reverse bittrees (not faster) +// 0x08 matched literal (not faster) +// +// GCC & Clang compatible x86-64 inline assembly: +// 0x010 normal bittrees +// 0x020 fixed-sized reverse bittrees +// 0x040 variable-sized reverse bittrees +// 0x080 matched literal +// 0x100 direct bits +// +// The default can be overridden at build time by defining +// LZMA_RANGE_DECODER_CONFIG to the desired mask. +// +// 2024-02-22: Feedback from benchmarks: +// - Brancless C (0x003) can be better than basic on x86-64 but often it's +// slightly worse on other archs. Since asm is much better on x86-64, +// branchless C is not used at all. +// - With x86-64 asm, there are slight differences between GCC and Clang +// and different processors. Overall 0x1F0 seems to be the best choice. +#ifndef LZMA_RANGE_DECODER_CONFIG +# if defined(__x86_64__) && !defined(__ILP32__) \ + && !defined(__NVCOMPILER) \ + && (defined(__GNUC__) || defined(__clang__)) +# define LZMA_RANGE_DECODER_CONFIG 0x1F0 +# else +# define LZMA_RANGE_DECODER_CONFIG 0 +# endif +#endif + + +// Negative RC_BIT_MODEL_TOTAL but the lowest RC_MOVE_BITS are flipped. +// This is useful for updating probability variables in branchless decoding: +// +// uint32_t decoded_bit = ...; +// probability tmp = RC_BIT_MODEL_OFFSET; +// tmp &= decoded_bit - 1; +// prob -= (prob + tmp) >> RC_MOVE_BITS; +#define RC_BIT_MODEL_OFFSET \ + ((UINT32_C(1) << RC_MOVE_BITS) - 1 - RC_BIT_MODEL_TOTAL) + + +typedef struct { + uint32_t range; + uint32_t code; + uint32_t init_bytes_left; +} lzma_range_decoder; + + +/// Reads the first five bytes to initialize the range decoder. +static inline lzma_ret +rc_read_init(lzma_range_decoder *rc, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size) +{ + while (rc->init_bytes_left > 0) { + if (*in_pos == in_size) + return LZMA_OK; + + // The first byte is always 0x00. It could have been omitted + // in LZMA2 but it wasn't, so one byte is wasted in every + // LZMA2 chunk. + if (rc->init_bytes_left == 5 && in[*in_pos] != 0x00) + return LZMA_DATA_ERROR; + + rc->code = (rc->code << 8) | in[*in_pos]; + ++*in_pos; + --rc->init_bytes_left; + } + + return LZMA_STREAM_END; +} + + +/// Makes local copies of range decoder and *in_pos variables. Doing this +/// improves speed significantly. The range decoder macros expect also +/// variables 'in' and 'in_size' to be defined. +#define rc_to_local(range_decoder, in_pos, fast_mode_in_required) \ + lzma_range_decoder rc = range_decoder; \ + const uint8_t *rc_in_ptr = in + (in_pos); \ + const uint8_t *rc_in_end = in + in_size; \ + const uint8_t *rc_in_fast_end \ + = (rc_in_end - rc_in_ptr) <= (fast_mode_in_required) \ + ? rc_in_ptr \ + : rc_in_end - (fast_mode_in_required); \ + (void)rc_in_fast_end; /* Silence a warning with HAVE_SMALL. */ \ + uint32_t rc_bound + + +/// Evaluates to true if there is enough input remaining to use fast mode. +#define rc_is_fast_allowed() (rc_in_ptr < rc_in_fast_end) + + +/// Stores the local copes back to the range decoder structure. +#define rc_from_local(range_decoder, in_pos) \ +do { \ + range_decoder = rc; \ + in_pos = (size_t)(rc_in_ptr - in); \ +} while (0) + + +/// Resets the range decoder structure. +#define rc_reset(range_decoder) \ +do { \ + (range_decoder).range = UINT32_MAX; \ + (range_decoder).code = 0; \ + (range_decoder).init_bytes_left = 5; \ +} while (0) + + +/// When decoding has been properly finished, rc.code is always zero unless +/// the input stream is corrupt. So checking this can catch some corrupt +/// files especially if they don't have any other integrity check. +#define rc_is_finished(range_decoder) \ + ((range_decoder).code == 0) + + +// Read the next input byte if needed. +#define rc_normalize() \ +do { \ + if (rc.range < RC_TOP_VALUE) { \ + rc.range <<= RC_SHIFT_BITS; \ + rc.code = (rc.code << RC_SHIFT_BITS) | *rc_in_ptr++; \ + } \ +} while (0) + + +/// If more input is needed but there is +/// no more input available, "goto out" is used to jump out of the main +/// decoder loop. The "_safe" macros are used in the Resumable decoder +/// mode in order to save the sequence to continue decoding from that +/// point later. +#define rc_normalize_safe(seq) \ +do { \ + if (rc.range < RC_TOP_VALUE) { \ + if (rc_in_ptr == rc_in_end) { \ + coder->sequence = seq; \ + goto out; \ + } \ + rc.range <<= RC_SHIFT_BITS; \ + rc.code = (rc.code << RC_SHIFT_BITS) | *rc_in_ptr++; \ + } \ +} while (0) + + +/// Start decoding a bit. This must be used together with rc_update_0() +/// and rc_update_1(): +/// +/// rc_if_0(prob) { +/// rc_update_0(prob); +/// // Do something +/// } else { +/// rc_update_1(prob); +/// // Do something else +/// } +/// +#define rc_if_0(prob) \ + rc_normalize(); \ + rc_bound = (rc.range >> RC_BIT_MODEL_TOTAL_BITS) * (prob); \ + if (rc.code < rc_bound) + + +#define rc_if_0_safe(prob, seq) \ + rc_normalize_safe(seq); \ + rc_bound = (rc.range >> RC_BIT_MODEL_TOTAL_BITS) * (prob); \ + if (rc.code < rc_bound) + + +/// Update the range decoder state and the used probability variable to +/// match a decoded bit of 0. +/// +/// The x86-64 assembly uses the commented method but it seems that, +/// at least on x86-64, the first version is slightly faster as C code. +#define rc_update_0(prob) \ +do { \ + rc.range = rc_bound; \ + prob += (RC_BIT_MODEL_TOTAL - (prob)) >> RC_MOVE_BITS; \ + /* prob -= ((prob) + RC_BIT_MODEL_OFFSET) >> RC_MOVE_BITS; */ \ +} while (0) + + +/// Update the range decoder state and the used probability variable to +/// match a decoded bit of 1. +#define rc_update_1(prob) \ +do { \ + rc.range -= rc_bound; \ + rc.code -= rc_bound; \ + prob -= (prob) >> RC_MOVE_BITS; \ +} while (0) + + +/// Decodes one bit and runs action0 or action1 depending on the decoded bit. +/// This macro is used as the last step in bittree reverse decoders since +/// those don't use "symbol" for anything else than indexing the probability +/// arrays. +#define rc_bit_last(prob, action0, action1) \ +do { \ + rc_if_0(prob) { \ + rc_update_0(prob); \ + action0; \ + } else { \ + rc_update_1(prob); \ + action1; \ + } \ +} while (0) + + +#define rc_bit_last_safe(prob, action0, action1, seq) \ +do { \ + rc_if_0_safe(prob, seq) { \ + rc_update_0(prob); \ + action0; \ + } else { \ + rc_update_1(prob); \ + action1; \ + } \ +} while (0) + + +/// Decodes one bit, updates "symbol", and runs action0 or action1 depending +/// on the decoded bit. +#define rc_bit(prob, action0, action1) \ + rc_bit_last(prob, \ + symbol <<= 1; action0, \ + symbol = (symbol << 1) + 1; action1); + + +#define rc_bit_safe(prob, action0, action1, seq) \ + rc_bit_last_safe(prob, \ + symbol <<= 1; action0, \ + symbol = (symbol << 1) + 1; action1, \ + seq); + +// Unroll fixed-sized bittree decoding. +// +// A compile-time constant in final_add can be used to get rid of the high bit +// from symbol that is used for the array indexing (1U << bittree_bits). +// final_add may also be used to add offset to the result (LZMA length +// decoder does that). +// +// The reason to have final_add here is that in the asm code the addition +// can be done for free: in x86-64 there is SBB instruction with -1 as +// the immediate value, and final_add is combined with that value. +#define rc_bittree_bit(prob) \ + rc_bit(prob, , ) + +#define rc_bittree3(probs, final_add) \ +do { \ + symbol = 1; \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + symbol += (uint32_t)(final_add); \ +} while (0) + +#define rc_bittree6(probs, final_add) \ +do { \ + symbol = 1; \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + symbol += (uint32_t)(final_add); \ +} while (0) + +#define rc_bittree8(probs, final_add) \ +do { \ + symbol = 1; \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + rc_bittree_bit(probs[symbol]); \ + symbol += (uint32_t)(final_add); \ +} while (0) + + +// Fixed-sized reverse bittree +#define rc_bittree_rev4(probs) \ +do { \ + symbol = 0; \ + rc_bit_last(probs[symbol + 1], , symbol += 1); \ + rc_bit_last(probs[symbol + 2], , symbol += 2); \ + rc_bit_last(probs[symbol + 4], , symbol += 4); \ + rc_bit_last(probs[symbol + 8], , symbol += 8); \ +} while (0) + + +// Decode one bit from variable-sized reverse bittree. The loop is done +// in the code that uses this macro. This could be changed if the assembly +// version benefited from having the loop done in assembly but it didn't +// seem so in early 2024. +// +// Also, if the loop was done here, the loop counter would likely be local +// to the macro so that it wouldn't modify yet another input variable. +// If a _safe version of a macro with a loop was done then a modifiable +// input variable couldn't be avoided though. +#define rc_bit_add_if_1(probs, dest, value_to_add_if_1) \ + rc_bit(probs[symbol], \ + , \ + dest += value_to_add_if_1); + + +// Matched literal +#define decode_with_match_bit \ + t_match_byte <<= 1; \ + t_match_bit = t_match_byte & t_offset; \ + t_subcoder_index = t_offset + t_match_bit + symbol; \ + rc_bit(probs[t_subcoder_index], \ + t_offset &= ~t_match_bit, \ + t_offset &= t_match_bit) + +#define rc_matched_literal(probs_base_var, match_byte) \ +do { \ + uint32_t t_match_byte = (match_byte); \ + uint32_t t_match_bit; \ + uint32_t t_subcoder_index; \ + uint32_t t_offset = 0x100; \ + symbol = 1; \ + decode_with_match_bit; \ + decode_with_match_bit; \ + decode_with_match_bit; \ + decode_with_match_bit; \ + decode_with_match_bit; \ + decode_with_match_bit; \ + decode_with_match_bit; \ + decode_with_match_bit; \ +} while (0) + + +/// Decode a bit without using a probability. +// +// NOTE: GCC 13 and Clang/LLVM 16 can, at least on x86-64, optimize the bound +// calculation to use an arithmetic right shift so there's no need to provide +// the alternative code which, according to C99/C11/C23 6.3.1.3-p3 isn't +// perfectly portable: rc_bound = (uint32_t)((int32_t)rc.code >> 31); +#define rc_direct(dest, count_var) \ +do { \ + dest = (dest << 1) + 1; \ + rc_normalize(); \ + rc.range >>= 1; \ + rc.code -= rc.range; \ + rc_bound = UINT32_C(0) - (rc.code >> 31); \ + dest += rc_bound; \ + rc.code += rc.range & rc_bound; \ +} while (--count_var > 0) + + + +#define rc_direct_safe(dest, count_var, seq) \ +do { \ + rc_normalize_safe(seq); \ + rc.range >>= 1; \ + rc.code -= rc.range; \ + rc_bound = UINT32_C(0) - (rc.code >> 31); \ + rc.code += rc.range & rc_bound; \ + dest = (dest << 1) + (rc_bound + 1); \ +} while (--count_var > 0) + + +////////////////// +// Branchless C // +////////////////// + +/// Decode a bit using a branchless method. This reduces the number of +/// mispredicted branches and thus can improve speed. +#define rc_c_bit(prob, action_bit, action_neg) \ +do { \ + probability *p = &(prob); \ + rc_normalize(); \ + rc_bound = (rc.range >> RC_BIT_MODEL_TOTAL_BITS) * *p; \ + uint32_t rc_mask = rc.code >= rc_bound; /* rc_mask = decoded bit */ \ + action_bit; /* action when rc_mask is 0 or 1 */ \ + /* rc_mask becomes 0 if bit is 0 and 0xFFFFFFFF if bit is 1: */ \ + rc_mask = 0U - rc_mask; \ + rc.range &= rc_mask; /* If bit 0: set rc.range = 0 */ \ + rc_bound ^= rc_mask; \ + rc_bound -= rc_mask; /* If bit 1: rc_bound = 0U - rc_bound */ \ + rc.range += rc_bound; \ + rc_bound &= rc_mask; \ + rc.code += rc_bound; \ + action_neg; /* action when rc_mask is 0 or 0xFFFFFFFF */ \ + rc_mask = ~rc_mask; /* If bit 0: all bits are set in rc_mask */ \ + rc_mask &= RC_BIT_MODEL_OFFSET; \ + *p -= (*p + rc_mask) >> RC_MOVE_BITS; \ +} while (0) + + +// Testing on x86-64 give an impression that only the normal bittrees and +// the fixed-sized reverse bittrees are worth the branchless C code. +// It should be tested on other archs for which there isn't assembly code +// in this file. + +// Using addition in "(symbol << 1) + rc_mask" allows use of x86 LEA +// or RISC-V SH1ADD instructions. Compilers might infer it from +// "(symbol << 1) | rc_mask" too if they see that mask is 0 or 1 but +// the use of addition doesn't require such analysis from compilers. +#if LZMA_RANGE_DECODER_CONFIG & 0x01 +#undef rc_bittree_bit +#define rc_bittree_bit(prob) \ + rc_c_bit(prob, \ + symbol = (symbol << 1) + rc_mask, \ + ) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x01 + +#if LZMA_RANGE_DECODER_CONFIG & 0x02 +#undef rc_bittree_rev4 +#define rc_bittree_rev4(probs) \ +do { \ + symbol = 0; \ + rc_c_bit(probs[symbol + 1], symbol += rc_mask, ); \ + rc_c_bit(probs[symbol + 2], symbol += rc_mask << 1, ); \ + rc_c_bit(probs[symbol + 4], symbol += rc_mask << 2, ); \ + rc_c_bit(probs[symbol + 8], symbol += rc_mask << 3, ); \ +} while (0) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x02 + +#if LZMA_RANGE_DECODER_CONFIG & 0x04 +#undef rc_bit_add_if_1 +#define rc_bit_add_if_1(probs, dest, value_to_add_if_1) \ + rc_c_bit(probs[symbol], \ + symbol = (symbol << 1) + rc_mask, \ + dest += (value_to_add_if_1) & rc_mask) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x04 + + +#if LZMA_RANGE_DECODER_CONFIG & 0x08 +#undef decode_with_match_bit +#define decode_with_match_bit \ + t_match_byte <<= 1; \ + t_match_bit = t_match_byte & t_offset; \ + t_subcoder_index = t_offset + t_match_bit + symbol; \ + rc_c_bit(probs[t_subcoder_index], \ + symbol = (symbol << 1) + rc_mask, \ + t_offset &= ~t_match_bit ^ rc_mask) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x08 + + +//////////// +// x86-64 // +//////////// + +#if LZMA_RANGE_DECODER_CONFIG & 0x1F0 + +// rc_asm_y and rc_asm_n are used as arguments to macros to control which +// strings to include or omit. +#define rc_asm_y(str) str +#define rc_asm_n(str) + +// There are a few possible variations for normalization. +// This is the smallest variant which is also used by LZMA SDK. +// +// - This has partial register write (the MOV from (%[in_ptr])). +// +// - INC saves one byte in code size over ADD. False dependency on +// partial flags from INC shouldn't become a problem on any processor +// because the instructions after normalization don't read the flags +// until SUB which sets all flags. +// +#define rc_asm_normalize \ + "cmp %[top_value], %[range]\n\t" \ + "jae 1f\n\t" \ + "shl %[shift_bits], %[code]\n\t" \ + "mov (%[in_ptr]), %b[code]\n\t" \ + "shl %[shift_bits], %[range]\n\t" \ + "inc %[in_ptr]\n" \ + "1:\n" + +// rc_asm_calc(prob) is roughly equivalent to the C version of rc_if_0(prob)... +// +// rc_bound = (rc.range >> RC_BIT_MODEL_TOTAL_BITS) * (prob); +// if (rc.code < rc_bound) +// +// ...but the bound is stored in "range": +// +// t0 = range; +// range = (range >> RC_BIT_MODEL_TOTAL_BITS) * (prob); +// t0 -= range; +// t1 = code; +// code -= range; +// +// The carry flag (CF) from the last subtraction holds the negation of +// the decoded bit (if CF==0 then the decoded bit is 1). +// The values in t0 and t1 are needed for rc_update_0(prob) and +// rc_update_1(prob). If the bit is 0, rc_update_0(prob)... +// +// rc.range = rc_bound; +// +// ...has already been done but the "code -= range" has to be reverted using +// the old value stored in t1. (Also, prob needs to be updated.) +// +// If the bit is 1, rc_update_1(prob)... +// +// rc.range -= rc_bound; +// rc.code -= rc_bound; +// +// ...is already done for "code" but the value for "range" needs to be taken +// from t0. (Also, prob needs to be updated here as well.) +// +// The assignments from t0 and t1 can be done in a branchless manner with CMOV +// after the instructions from this macro. The CF from SUB tells which moves +// are needed. +#define rc_asm_calc(prob) \ + "mov %[range], %[t0]\n\t" \ + "shr %[bit_model_total_bits], %[range]\n\t" \ + "imul %[" prob "], %[range]\n\t" \ + "sub %[range], %[t0]\n\t" \ + "mov %[code], %[t1]\n\t" \ + "sub %[range], %[code]\n\t" + +// Also, prob needs to be updated: The update math depends on the decoded bit. +// It can be expressed in a few slightly different ways but this is fairly +// convenient here: +// +// prob -= (prob + (bit ? 0 : RC_BIT_MODEL_OFFSET)) >> RC_MOVE_BITS; +// +// To do it in branchless way when the negation of the decoded bit is in CF, +// both "prob" and "prob + RC_BIT_MODEL_OFFSET" are needed. Then the desired +// value can be picked with CMOV. The addition can be done using LEA without +// affecting CF. +// +// (This prob update method is a tiny bit different from LZMA SDK 23.01. +// In the LZMA SDK a single register is reserved solely for a constant to +// be used with CMOV when updating prob. That is fine since there are enough +// free registers to do so. The method used here uses one fewer register, +// which is valuable with inline assembly.) +// +// * * * +// +// In bittree decoding, each (unrolled) loop iteration decodes one bit +// and needs one prob variable. To make it faster, the prob variable of +// the iteration N+1 is loaded during iteration N. There are two possible +// prob variables to choose from for N+1. Both are loaded from memory and +// the correct one is chosen with CMOV using the same CF as is used for +// other things described above. +// +// This preloading/prefetching requires an extra register. To avoid +// useless moves from "preloaded prob register" to "current prob register", +// the macros swap between the two registers for odd and even iterations. +// +// * * * +// +// Finally, the decoded bit has to be stored in "symbol". Since the negation +// of the bit is in CF, this can be done with SBB: symbol -= CF - 1. That is, +// if the decoded bit is 0 (CF==1) the operation is a no-op "symbol -= 0" +// and when bit is 1 (CF==0) the operation is "symbol -= 0 - 1" which is +// the same as "symbol += 1". +// +// The instructions for all things are intertwined for a few reasons: +// - freeing temporary registers for new use +// - not modifying CF too early +// - instruction scheduling +// +// The first and last iterations can cheat a little. For example, +// on the first iteration "symbol" is known to start from 1 so it +// doesn't need to be read; it can even be immediately initialized +// to 2 to prepare for the second iteration of the loop. +// +// * * * +// +// a = number of the current prob variable (0 or 1) +// b = number of the next prob variable (1 or 0) +// *_only = rc_asm_y or _n to include or exclude code marked with them +#define rc_asm_bittree(a, b, first_only, middle_only, last_only) \ + first_only( \ + "movzwl 2(%[probs_base]), %[prob" #a "]\n\t" \ + "mov $2, %[symbol]\n\t" \ + "movzwl 4(%[probs_base]), %[prob" #b "]\n\t" \ + ) \ + middle_only( \ + /* Note the scaling of 4 instead of 2: */ \ + "movzwl (%[probs_base], %q[symbol], 4), %[prob" #b "]\n\t" \ + ) \ + last_only( \ + "add %[symbol], %[symbol]\n\t" \ + ) \ + \ + rc_asm_normalize \ + rc_asm_calc("prob" #a) \ + \ + "cmovae %[t0], %[range]\n\t" \ + \ + first_only( \ + "movzwl 6(%[probs_base]), %[t0]\n\t" \ + "cmovae %[t0], %[prob" #b "]\n\t" \ + ) \ + middle_only( \ + "movzwl 2(%[probs_base], %q[symbol], 4), %[t0]\n\t" \ + "lea (%q[symbol], %q[symbol]), %[symbol]\n\t" \ + "cmovae %[t0], %[prob" #b "]\n\t" \ + ) \ + \ + "lea %c[bit_model_offset](%q[prob" #a "]), %[t0]\n\t" \ + "cmovb %[t1], %[code]\n\t" \ + "mov %[symbol], %[t1]\n\t" \ + "cmovae %[prob" #a "], %[t0]\n\t" \ + \ + first_only( \ + "sbb $-1, %[symbol]\n\t" \ + ) \ + middle_only( \ + "sbb $-1, %[symbol]\n\t" \ + ) \ + last_only( \ + "sbb %[last_sbb], %[symbol]\n\t" \ + ) \ + \ + "shr %[move_bits], %[t0]\n\t" \ + "sub %[t0], %[prob" #a "]\n\t" \ + /* Scaling of 1 instead of 2 because symbol <<= 1. */ \ + "mov %w[prob" #a "], (%[probs_base], %q[t1], 1)\n\t" + +// NOTE: The order of variables in __asm__ can affect speed and code size. +#define rc_asm_bittree_n(probs_base_var, final_add, asm_str) \ +do { \ + uint32_t t0; \ + uint32_t t1; \ + uint32_t t_prob0; \ + uint32_t t_prob1; \ + \ + __asm__( \ + asm_str \ + : \ + [range] "+&r"(rc.range), \ + [code] "+&r"(rc.code), \ + [t0] "=&r"(t0), \ + [t1] "=&r"(t1), \ + [prob0] "=&r"(t_prob0), \ + [prob1] "=&r"(t_prob1), \ + [symbol] "=&r"(symbol), \ + [in_ptr] "+&r"(rc_in_ptr) \ + : \ + [probs_base] "r"(probs_base_var), \ + [last_sbb] "n"(-1 - (final_add)), \ + [top_value] "n"(RC_TOP_VALUE), \ + [shift_bits] "n"(RC_SHIFT_BITS), \ + [bit_model_total_bits] "n"(RC_BIT_MODEL_TOTAL_BITS), \ + [bit_model_offset] "n"(RC_BIT_MODEL_OFFSET), \ + [move_bits] "n"(RC_MOVE_BITS) \ + : \ + "cc", "memory"); \ +} while (0) + + +#if LZMA_RANGE_DECODER_CONFIG & 0x010 +#undef rc_bittree3 +#define rc_bittree3(probs_base_var, final_add) \ + rc_asm_bittree_n(probs_base_var, final_add, \ + rc_asm_bittree(0, 1, rc_asm_y, rc_asm_n, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(0, 1, rc_asm_n, rc_asm_n, rc_asm_y) \ + ) + +#undef rc_bittree6 +#define rc_bittree6(probs_base_var, final_add) \ + rc_asm_bittree_n(probs_base_var, final_add, \ + rc_asm_bittree(0, 1, rc_asm_y, rc_asm_n, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(0, 1, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(0, 1, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_n, rc_asm_y) \ + ) + +#undef rc_bittree8 +#define rc_bittree8(probs_base_var, final_add) \ + rc_asm_bittree_n(probs_base_var, final_add, \ + rc_asm_bittree(0, 1, rc_asm_y, rc_asm_n, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(0, 1, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(0, 1, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(0, 1, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree(1, 0, rc_asm_n, rc_asm_n, rc_asm_y) \ + ) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x010 + + +// Fixed-sized reverse bittree +// +// This uses the indexing that constructs the final value in symbol directly. +// add = 1, 2, 4, 8 +// dcur = -, 4, 8, 16 +// dnext0 = 4, 8, 16, - +// dnext0 = 6, 12, 24, - +#define rc_asm_bittree_rev(a, b, add, dcur, dnext0, dnext1, \ + first_only, middle_only, last_only) \ + first_only( \ + "movzwl 2(%[probs_base]), %[prob" #a "]\n\t" \ + "xor %[symbol], %[symbol]\n\t" \ + "movzwl 4(%[probs_base]), %[prob" #b "]\n\t" \ + ) \ + middle_only( \ + "movzwl " #dnext0 "(%[probs_base], %q[symbol], 2), " \ + "%[prob" #b "]\n\t" \ + ) \ + \ + rc_asm_normalize \ + rc_asm_calc("prob" #a) \ + \ + "cmovae %[t0], %[range]\n\t" \ + \ + first_only( \ + "movzwl 6(%[probs_base]), %[t0]\n\t" \ + "cmovae %[t0], %[prob" #b "]\n\t" \ + ) \ + middle_only( \ + "movzwl " #dnext1 "(%[probs_base], %q[symbol], 2), %[t0]\n\t" \ + "cmovae %[t0], %[prob" #b "]\n\t" \ + ) \ + \ + "lea " #add "(%q[symbol]), %[t0]\n\t" \ + "cmovb %[t1], %[code]\n\t" \ + middle_only( \ + "mov %[symbol], %[t1]\n\t" \ + ) \ + last_only( \ + "mov %[symbol], %[t1]\n\t" \ + ) \ + "cmovae %[t0], %[symbol]\n\t" \ + "lea %c[bit_model_offset](%q[prob" #a "]), %[t0]\n\t" \ + "cmovae %[prob" #a "], %[t0]\n\t" \ + \ + "shr %[move_bits], %[t0]\n\t" \ + "sub %[t0], %[prob" #a "]\n\t" \ + first_only( \ + "mov %w[prob" #a "], 2(%[probs_base])\n\t" \ + ) \ + middle_only( \ + "mov %w[prob" #a "], " \ + #dcur "(%[probs_base], %q[t1], 2)\n\t" \ + ) \ + last_only( \ + "mov %w[prob" #a "], " \ + #dcur "(%[probs_base], %q[t1], 2)\n\t" \ + ) + +#if LZMA_RANGE_DECODER_CONFIG & 0x020 +#undef rc_bittree_rev4 +#define rc_bittree_rev4(probs_base_var) \ +rc_asm_bittree_n(probs_base_var, 4, \ + rc_asm_bittree_rev(0, 1, 1, -, 4, 6, rc_asm_y, rc_asm_n, rc_asm_n) \ + rc_asm_bittree_rev(1, 0, 2, 4, 8, 12, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree_rev(0, 1, 4, 8, 16, 24, rc_asm_n, rc_asm_y, rc_asm_n) \ + rc_asm_bittree_rev(1, 0, 8, 16, -, -, rc_asm_n, rc_asm_n, rc_asm_y) \ +) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x020 + + +#if LZMA_RANGE_DECODER_CONFIG & 0x040 +#undef rc_bit_add_if_1 +#define rc_bit_add_if_1(probs_base_var, dest_var, value_to_add_if_1) \ +do { \ + uint32_t t0; \ + uint32_t t1; \ + uint32_t t2 = (value_to_add_if_1); \ + uint32_t t_prob; \ + uint32_t t_index; \ + \ + __asm__( \ + "movzwl (%[probs_base], %q[symbol], 2), %[prob]\n\t" \ + "mov %[symbol], %[index]\n\t" \ + \ + "add %[dest], %[t2]\n\t" \ + "add %[symbol], %[symbol]\n\t" \ + \ + rc_asm_normalize \ + rc_asm_calc("prob") \ + \ + "cmovae %[t0], %[range]\n\t" \ + "lea %c[bit_model_offset](%q[prob]), %[t0]\n\t" \ + "cmovb %[t1], %[code]\n\t" \ + "cmovae %[prob], %[t0]\n\t" \ + \ + "cmovae %[t2], %[dest]\n\t" \ + "sbb $-1, %[symbol]\n\t" \ + \ + "sar %[move_bits], %[t0]\n\t" \ + "sub %[t0], %[prob]\n\t" \ + "mov %w[prob], (%[probs_base], %q[index], 2)" \ + : \ + [range] "+&r"(rc.range), \ + [code] "+&r"(rc.code), \ + [t0] "=&r"(t0), \ + [t1] "=&r"(t1), \ + [prob] "=&r"(t_prob), \ + [index] "=&r"(t_index), \ + [symbol] "+&r"(symbol), \ + [t2] "+&r"(t2), \ + [dest] "+&r"(dest_var), \ + [in_ptr] "+&r"(rc_in_ptr) \ + : \ + [probs_base] "r"(probs_base_var), \ + [top_value] "n"(RC_TOP_VALUE), \ + [shift_bits] "n"(RC_SHIFT_BITS), \ + [bit_model_total_bits] "n"(RC_BIT_MODEL_TOTAL_BITS), \ + [bit_model_offset] "n"(RC_BIT_MODEL_OFFSET), \ + [move_bits] "n"(RC_MOVE_BITS) \ + : \ + "cc", "memory"); \ +} while (0) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x040 + + +// Literal decoding uses a normal 8-bit bittree but literal with match byte +// is more complex in picking the probability variable from the correct +// subtree. This doesn't use preloading/prefetching of the next prob because +// there are four choices instead of two. +// +// FIXME? The first iteration starts with symbol = 1 so it could be optimized +// by a tiny amount. +#define rc_asm_matched_literal(nonlast_only) \ + "add %[offset], %[symbol]\n\t" \ + "and %[offset], %[match_bit]\n\t" \ + "add %[match_bit], %[symbol]\n\t" \ + \ + "movzwl (%[probs_base], %q[symbol], 2), %[prob]\n\t" \ + \ + "add %[symbol], %[symbol]\n\t" \ + \ + nonlast_only( \ + "xor %[match_bit], %[offset]\n\t" \ + "add %[match_byte], %[match_byte]\n\t" \ + ) \ + \ + rc_asm_normalize \ + rc_asm_calc("prob") \ + \ + "cmovae %[t0], %[range]\n\t" \ + "lea %c[bit_model_offset](%q[prob]), %[t0]\n\t" \ + "cmovb %[t1], %[code]\n\t" \ + "mov %[symbol], %[t1]\n\t" \ + "cmovae %[prob], %[t0]\n\t" \ + \ + nonlast_only( \ + "cmovae %[match_bit], %[offset]\n\t" \ + "mov %[match_byte], %[match_bit]\n\t" \ + ) \ + \ + "sbb $-1, %[symbol]\n\t" \ + \ + "shr %[move_bits], %[t0]\n\t" \ + /* Undo symbol += match_bit + offset: */ \ + "and $0x1FF, %[symbol]\n\t" \ + "sub %[t0], %[prob]\n\t" \ + \ + /* Scaling of 1 instead of 2 because symbol <<= 1. */ \ + "mov %w[prob], (%[probs_base], %q[t1], 1)\n\t" + + +#if LZMA_RANGE_DECODER_CONFIG & 0x080 +#undef rc_matched_literal +#define rc_matched_literal(probs_base_var, match_byte_value) \ +do { \ + uint32_t t0; \ + uint32_t t1; \ + uint32_t t_prob; \ + uint32_t t_match_byte = (uint32_t)(match_byte_value) << 1; \ + uint32_t t_match_bit = t_match_byte; \ + uint32_t t_offset = 0x100; \ + symbol = 1; \ + \ + __asm__( \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_y) \ + rc_asm_matched_literal(rc_asm_n) \ + : \ + [range] "+&r"(rc.range), \ + [code] "+&r"(rc.code), \ + [t0] "=&r"(t0), \ + [t1] "=&r"(t1), \ + [prob] "=&r"(t_prob), \ + [match_bit] "+&r"(t_match_bit), \ + [symbol] "+&r"(symbol), \ + [match_byte] "+&r"(t_match_byte), \ + [offset] "+&r"(t_offset), \ + [in_ptr] "+&r"(rc_in_ptr) \ + : \ + [probs_base] "r"(probs_base_var), \ + [top_value] "n"(RC_TOP_VALUE), \ + [shift_bits] "n"(RC_SHIFT_BITS), \ + [bit_model_total_bits] "n"(RC_BIT_MODEL_TOTAL_BITS), \ + [bit_model_offset] "n"(RC_BIT_MODEL_OFFSET), \ + [move_bits] "n"(RC_MOVE_BITS) \ + : \ + "cc", "memory"); \ +} while (0) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x080 + + +// Doing the loop in asm instead of C seems to help a little. +#if LZMA_RANGE_DECODER_CONFIG & 0x100 +#undef rc_direct +#define rc_direct(dest_var, count_var) \ +do { \ + uint32_t t0; \ + uint32_t t1; \ + \ + __asm__( \ + "2:\n\t" \ + "add %[dest], %[dest]\n\t" \ + "lea 1(%q[dest]), %[t1]\n\t" \ + \ + rc_asm_normalize \ + \ + "shr $1, %[range]\n\t" \ + "mov %[code], %[t0]\n\t" \ + "sub %[range], %[code]\n\t" \ + "cmovns %[t1], %[dest]\n\t" \ + "cmovs %[t0], %[code]\n\t" \ + "dec %[count]\n\t" \ + "jnz 2b\n\t" \ + : \ + [range] "+&r"(rc.range), \ + [code] "+&r"(rc.code), \ + [t0] "=&r"(t0), \ + [t1] "=&r"(t1), \ + [dest] "+&r"(dest_var), \ + [count] "+&r"(count_var), \ + [in_ptr] "+&r"(rc_in_ptr) \ + : \ + [top_value] "n"(RC_TOP_VALUE), \ + [shift_bits] "n"(RC_SHIFT_BITS) \ + : \ + "cc", "memory"); \ +} while (0) +#endif // LZMA_RANGE_DECODER_CONFIG & 0x100 + +#endif // x86_64 + +#endif diff --git a/src/liblzma/rangecoder/range_encoder.h b/src/liblzma/rangecoder/range_encoder.h new file mode 100644 index 0000000000..8f62a47ac0 --- /dev/null +++ b/src/liblzma/rangecoder/range_encoder.h @@ -0,0 +1,349 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file range_encoder.h +/// \brief Range Encoder +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_RANGE_ENCODER_H +#define LZMA_RANGE_ENCODER_H + +#include "range_common.h" +#include "price.h" + + +/// Maximum number of symbols that can be put pending into lzma_range_encoder +/// structure between calls to lzma_rc_encode(). For LZMA, 48+5 is enough +/// (match with big distance and length followed by range encoder flush). +#define RC_SYMBOLS_MAX 53 + + +typedef struct { + uint64_t low; + uint64_t cache_size; + uint32_t range; + uint8_t cache; + + /// Number of bytes written out by rc_encode() -> rc_shift_low() + uint64_t out_total; + + /// Number of symbols in the tables + size_t count; + + /// rc_encode()'s position in the tables + size_t pos; + + /// Symbols to encode + enum { + RC_BIT_0, + RC_BIT_1, + RC_DIRECT_0, + RC_DIRECT_1, + RC_FLUSH, + } symbols[RC_SYMBOLS_MAX]; + + /// Probabilities associated with RC_BIT_0 or RC_BIT_1 + probability *probs[RC_SYMBOLS_MAX]; + +} lzma_range_encoder; + + +static inline void +rc_reset(lzma_range_encoder *rc) +{ + rc->low = 0; + rc->cache_size = 1; + rc->range = UINT32_MAX; + rc->cache = 0; + rc->out_total = 0; + rc->count = 0; + rc->pos = 0; +} + + +static inline void +rc_forget(lzma_range_encoder *rc) +{ + // This must not be called when rc_encode() is partially done. + assert(rc->pos == 0); + rc->count = 0; +} + + +static inline void +rc_bit(lzma_range_encoder *rc, probability *prob, uint32_t bit) +{ + rc->symbols[rc->count] = bit; + rc->probs[rc->count] = prob; + ++rc->count; +} + + +static inline void +rc_bittree(lzma_range_encoder *rc, probability *probs, + uint32_t bit_count, uint32_t symbol) +{ + uint32_t model_index = 1; + + do { + const uint32_t bit = (symbol >> --bit_count) & 1; + rc_bit(rc, &probs[model_index], bit); + model_index = (model_index << 1) + bit; + } while (bit_count != 0); +} + + +static inline void +rc_bittree_reverse(lzma_range_encoder *rc, probability *probs, + uint32_t bit_count, uint32_t symbol) +{ + uint32_t model_index = 1; + + do { + const uint32_t bit = symbol & 1; + symbol >>= 1; + rc_bit(rc, &probs[model_index], bit); + model_index = (model_index << 1) + bit; + } while (--bit_count != 0); +} + + +static inline void +rc_direct(lzma_range_encoder *rc, + uint32_t value, uint32_t bit_count) +{ + do { + rc->symbols[rc->count++] + = RC_DIRECT_0 + ((value >> --bit_count) & 1); + } while (bit_count != 0); +} + + +static inline void +rc_flush(lzma_range_encoder *rc) +{ + for (size_t i = 0; i < 5; ++i) + rc->symbols[rc->count++] = RC_FLUSH; +} + + +static inline bool +rc_shift_low(lzma_range_encoder *rc, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + if ((uint32_t)(rc->low) < (uint32_t)(0xFF000000) + || (uint32_t)(rc->low >> 32) != 0) { + do { + if (*out_pos == out_size) + return true; + + out[*out_pos] = rc->cache + (uint8_t)(rc->low >> 32); + ++*out_pos; + ++rc->out_total; + rc->cache = 0xFF; + + } while (--rc->cache_size != 0); + + rc->cache = (rc->low >> 24) & 0xFF; + } + + ++rc->cache_size; + rc->low = (rc->low & 0x00FFFFFF) << RC_SHIFT_BITS; + + return false; +} + + +// NOTE: The last two arguments are uint64_t instead of size_t because in +// the dummy version these refer to the size of the whole range-encoded +// output stream, not just to the currently available output buffer space. +static inline bool +rc_shift_low_dummy(uint64_t *low, uint64_t *cache_size, uint8_t *cache, + uint64_t *out_pos, uint64_t out_size) +{ + if ((uint32_t)(*low) < (uint32_t)(0xFF000000) + || (uint32_t)(*low >> 32) != 0) { + do { + if (*out_pos == out_size) + return true; + + ++*out_pos; + *cache = 0xFF; + + } while (--*cache_size != 0); + + *cache = (*low >> 24) & 0xFF; + } + + ++*cache_size; + *low = (*low & 0x00FFFFFF) << RC_SHIFT_BITS; + + return false; +} + + +static inline bool +rc_encode(lzma_range_encoder *rc, + uint8_t *out, size_t *out_pos, size_t out_size) +{ + assert(rc->count <= RC_SYMBOLS_MAX); + + while (rc->pos < rc->count) { + // Normalize + if (rc->range < RC_TOP_VALUE) { + if (rc_shift_low(rc, out, out_pos, out_size)) + return true; + + rc->range <<= RC_SHIFT_BITS; + } + + // Encode a bit + switch (rc->symbols[rc->pos]) { + case RC_BIT_0: { + probability prob = *rc->probs[rc->pos]; + rc->range = (rc->range >> RC_BIT_MODEL_TOTAL_BITS) + * prob; + prob += (RC_BIT_MODEL_TOTAL - prob) >> RC_MOVE_BITS; + *rc->probs[rc->pos] = prob; + break; + } + + case RC_BIT_1: { + probability prob = *rc->probs[rc->pos]; + const uint32_t bound = prob * (rc->range + >> RC_BIT_MODEL_TOTAL_BITS); + rc->low += bound; + rc->range -= bound; + prob -= prob >> RC_MOVE_BITS; + *rc->probs[rc->pos] = prob; + break; + } + + case RC_DIRECT_0: + rc->range >>= 1; + break; + + case RC_DIRECT_1: + rc->range >>= 1; + rc->low += rc->range; + break; + + case RC_FLUSH: + // Prevent further normalizations. + rc->range = UINT32_MAX; + + // Flush the last five bytes (see rc_flush()). + do { + if (rc_shift_low(rc, out, out_pos, out_size)) + return true; + } while (++rc->pos < rc->count); + + // Reset the range encoder so we are ready to continue + // encoding if we weren't finishing the stream. + rc_reset(rc); + return false; + + default: + assert(0); + break; + } + + ++rc->pos; + } + + rc->count = 0; + rc->pos = 0; + + return false; +} + + +static inline bool +rc_encode_dummy(const lzma_range_encoder *rc, uint64_t out_limit) +{ + assert(rc->count <= RC_SYMBOLS_MAX); + + uint64_t low = rc->low; + uint64_t cache_size = rc->cache_size; + uint32_t range = rc->range; + uint8_t cache = rc->cache; + uint64_t out_pos = rc->out_total; + + size_t pos = rc->pos; + + while (true) { + // Normalize + if (range < RC_TOP_VALUE) { + if (rc_shift_low_dummy(&low, &cache_size, &cache, + &out_pos, out_limit)) + return true; + + range <<= RC_SHIFT_BITS; + } + + // This check is here because the normalization above must + // be done before flushing the last bytes. + if (pos == rc->count) + break; + + // Encode a bit + switch (rc->symbols[pos]) { + case RC_BIT_0: { + probability prob = *rc->probs[pos]; + range = (range >> RC_BIT_MODEL_TOTAL_BITS) + * prob; + break; + } + + case RC_BIT_1: { + probability prob = *rc->probs[pos]; + const uint32_t bound = prob * (range + >> RC_BIT_MODEL_TOTAL_BITS); + low += bound; + range -= bound; + break; + } + + case RC_DIRECT_0: + range >>= 1; + break; + + case RC_DIRECT_1: + range >>= 1; + low += range; + break; + + case RC_FLUSH: + default: + assert(0); + break; + } + + ++pos; + } + + // Flush the last bytes. This isn't in rc->symbols[] so we do + // it after the above loop to take into account the size of + // the flushing that will be done at the end of the stream. + for (pos = 0; pos < 5; ++pos) { + if (rc_shift_low_dummy(&low, &cache_size, + &cache, &out_pos, out_limit)) + return true; + } + + return false; +} + + +static inline uint64_t +rc_pending(const lzma_range_encoder *rc) +{ + return rc->cache_size + 5 - 1; +} + +#endif diff --git a/src/liblzma/simple/Makefile.inc b/src/liblzma/simple/Makefile.inc new file mode 100644 index 0000000000..f42ad1ba51 --- /dev/null +++ b/src/liblzma/simple/Makefile.inc @@ -0,0 +1,51 @@ +## SPDX-License-Identifier: 0BSD +## Author: Lasse Collin + +liblzma_la_SOURCES += \ + simple/simple_coder.c \ + simple/simple_coder.h \ + simple/simple_private.h + +if COND_ENCODER_SIMPLE +liblzma_la_SOURCES += \ + simple/simple_encoder.c \ + simple/simple_encoder.h +endif + +if COND_DECODER_SIMPLE +liblzma_la_SOURCES += \ + simple/simple_decoder.c \ + simple/simple_decoder.h +endif + +if COND_FILTER_X86 +liblzma_la_SOURCES += simple/x86.c +endif + +if COND_FILTER_POWERPC +liblzma_la_SOURCES += simple/powerpc.c +endif + +if COND_FILTER_IA64 +liblzma_la_SOURCES += simple/ia64.c +endif + +if COND_FILTER_ARM +liblzma_la_SOURCES += simple/arm.c +endif + +if COND_FILTER_ARMTHUMB +liblzma_la_SOURCES += simple/armthumb.c +endif + +if COND_FILTER_ARM64 +liblzma_la_SOURCES += simple/arm64.c +endif + +if COND_FILTER_SPARC +liblzma_la_SOURCES += simple/sparc.c +endif + +if COND_FILTER_RISCV +liblzma_la_SOURCES += simple/riscv.c +endif diff --git a/src/liblzma/simple/arm.c b/src/liblzma/simple/arm.c new file mode 100644 index 0000000000..f9d9c08b3c --- /dev/null +++ b/src/liblzma/simple/arm.c @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file arm.c +/// \brief Filter for ARM binaries +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +arm_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + size &= ~(size_t)3; + + size_t i; + for (i = 0; i < size; i += 4) { + if (buffer[i + 3] == 0xEB) { + uint32_t src = ((uint32_t)(buffer[i + 2]) << 16) + | ((uint32_t)(buffer[i + 1]) << 8) + | (uint32_t)(buffer[i + 0]); + src <<= 2; + + uint32_t dest; + if (is_encoder) + dest = now_pos + (uint32_t)(i) + 8 + src; + else + dest = src - (now_pos + (uint32_t)(i) + 8); + + dest >>= 2; + buffer[i + 2] = (dest >> 16); + buffer[i + 1] = (dest >> 8); + buffer[i + 0] = dest; + } + } + + return i; +} + + +static lzma_ret +arm_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &arm_code, 0, 4, 4, is_encoder); +} + + +#ifdef HAVE_ENCODER_ARM +extern lzma_ret +lzma_simple_arm_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return arm_coder_init(next, allocator, filters, true); +} +#endif + + +#ifdef HAVE_DECODER_ARM +extern lzma_ret +lzma_simple_arm_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return arm_coder_init(next, allocator, filters, false); +} +#endif diff --git a/src/liblzma/simple/arm64.c b/src/liblzma/simple/arm64.c new file mode 100644 index 0000000000..2ec10d937f --- /dev/null +++ b/src/liblzma/simple/arm64.c @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file arm64.c +/// \brief Filter for ARM64 binaries +/// +/// This converts ARM64 relative addresses in the BL and ADRP immediates +/// to absolute values to increase redundancy of ARM64 code. +/// +/// Converting B or ADR instructions was also tested but it's not useful. +/// A majority of the jumps for the B instruction are very small (+/- 0xFF). +/// These are typical for loops and if-statements. Encoding them to their +/// absolute address reduces redundancy since many of the small relative +/// jump values are repeated, but very few of the absolute addresses are. +// +// Authors: Lasse Collin +// Jia Tan +// Igor Pavlov +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +arm64_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + size &= ~(size_t)3; + + size_t i; + + // Clang 14.0.6 on x86-64 makes this four times bigger and 40 % slower + // with auto-vectorization that is enabled by default with -O2. + // Such vectorization bloat happens with -O2 when targeting ARM64 too + // but performance hasn't been tested. +#ifdef __clang__ +# pragma clang loop vectorize(disable) +#endif + for (i = 0; i < size; i += 4) { + uint32_t pc = (uint32_t)(now_pos + i); + uint32_t instr = read32le(buffer + i); + + if ((instr >> 26) == 0x25) { + // BL instruction: + // The full 26-bit immediate is converted. + // The range is +/-128 MiB. + // + // Using the full range helps quite a lot with + // big executables. Smaller range would reduce false + // positives in non-code sections of the input though + // so this is a compromise that slightly favors big + // files. With the full range, only six bits of the 32 + // need to match to trigger a conversion. + const uint32_t src = instr; + instr = 0x94000000; + + pc >>= 2; + if (!is_encoder) + pc = 0U - pc; + + instr |= (src + pc) & 0x03FFFFFF; + write32le(buffer + i, instr); + + } else if ((instr & 0x9F000000) == 0x90000000) { + // ADRP instruction: + // Only values in the range +/-512 MiB are converted. + // + // Using less than the full +/-4 GiB range reduces + // false positives on non-code sections of the input + // while being excellent for executables up to 512 MiB. + // The positive effect of ADRP conversion is smaller + // than that of BL but it also doesn't hurt so much in + // non-code sections of input because, with +/-512 MiB + // range, nine bits of 32 need to match to trigger a + // conversion (two 10-bit match choices = 9 bits). + const uint32_t src = ((instr >> 29) & 3) + | ((instr >> 3) & 0x001FFFFC); + + // With the addition only one branch is needed to + // check the +/- range. This is usually false when + // processing ARM64 code so branch prediction will + // handle it well in terms of performance. + // + //if ((src & 0x001E0000) != 0 + // && (src & 0x001E0000) != 0x001E0000) + if ((src + 0x00020000) & 0x001C0000) + continue; + + instr &= 0x9000001F; + + pc >>= 12; + if (!is_encoder) + pc = 0U - pc; + + const uint32_t dest = src + pc; + instr |= (dest & 3) << 29; + instr |= (dest & 0x0003FFFC) << 3; + instr |= (0U - (dest & 0x00020000)) & 0x00E00000; + write32le(buffer + i, instr); + } + } + + return i; +} + + +static lzma_ret +arm64_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &arm64_code, 0, 4, 4, is_encoder); +} + + +#ifdef HAVE_ENCODER_ARM64 +extern lzma_ret +lzma_simple_arm64_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return arm64_coder_init(next, allocator, filters, true); +} + + +extern LZMA_API(size_t) +lzma_bcj_arm64_encode(uint32_t start_offset, uint8_t *buf, size_t size) +{ + // start_offset must be a multiple of four. + start_offset &= ~UINT32_C(3); + return arm64_code(NULL, start_offset, true, buf, size); +} +#endif + + +#ifdef HAVE_DECODER_ARM64 +extern lzma_ret +lzma_simple_arm64_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return arm64_coder_init(next, allocator, filters, false); +} + + +extern LZMA_API(size_t) +lzma_bcj_arm64_decode(uint32_t start_offset, uint8_t *buf, size_t size) +{ + // start_offset must be a multiple of four. + start_offset &= ~UINT32_C(3); + return arm64_code(NULL, start_offset, false, buf, size); +} +#endif diff --git a/src/liblzma/simple/armthumb.c b/src/liblzma/simple/armthumb.c new file mode 100644 index 0000000000..368b51c7fe --- /dev/null +++ b/src/liblzma/simple/armthumb.c @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file armthumb.c +/// \brief Filter for ARM-Thumb binaries +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +armthumb_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + if (size < 4) + return 0; + + size -= 4; + + size_t i; + for (i = 0; i <= size; i += 2) { + if ((buffer[i + 1] & 0xF8) == 0xF0 + && (buffer[i + 3] & 0xF8) == 0xF8) { + uint32_t src = (((uint32_t)(buffer[i + 1]) & 7) << 19) + | ((uint32_t)(buffer[i + 0]) << 11) + | (((uint32_t)(buffer[i + 3]) & 7) << 8) + | (uint32_t)(buffer[i + 2]); + + src <<= 1; + + uint32_t dest; + if (is_encoder) + dest = now_pos + (uint32_t)(i) + 4 + src; + else + dest = src - (now_pos + (uint32_t)(i) + 4); + + dest >>= 1; + buffer[i + 1] = 0xF0 | ((dest >> 19) & 0x7); + buffer[i + 0] = (dest >> 11); + buffer[i + 3] = 0xF8 | ((dest >> 8) & 0x7); + buffer[i + 2] = (dest); + i += 2; + } + } + + return i; +} + + +static lzma_ret +armthumb_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &armthumb_code, 0, 4, 2, is_encoder); +} + + +#ifdef HAVE_ENCODER_ARMTHUMB +extern lzma_ret +lzma_simple_armthumb_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return armthumb_coder_init(next, allocator, filters, true); +} +#endif + + +#ifdef HAVE_DECODER_ARMTHUMB +extern lzma_ret +lzma_simple_armthumb_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return armthumb_coder_init(next, allocator, filters, false); +} +#endif diff --git a/src/liblzma/simple/ia64.c b/src/liblzma/simple/ia64.c new file mode 100644 index 0000000000..2a4aaebb47 --- /dev/null +++ b/src/liblzma/simple/ia64.c @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file ia64.c +/// \brief Filter for IA64 (Itanium) binaries +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +ia64_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + static const uint32_t BRANCH_TABLE[32] = { + 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, + 4, 4, 6, 6, 0, 0, 7, 7, + 4, 4, 0, 0, 4, 4, 0, 0 + }; + + size &= ~(size_t)15; + + size_t i; + for (i = 0; i < size; i += 16) { + const uint32_t instr_template = buffer[i] & 0x1F; + const uint32_t mask = BRANCH_TABLE[instr_template]; + uint32_t bit_pos = 5; + + for (size_t slot = 0; slot < 3; ++slot, bit_pos += 41) { + if (((mask >> slot) & 1) == 0) + continue; + + const size_t byte_pos = (bit_pos >> 3); + const uint32_t bit_res = bit_pos & 0x7; + uint64_t instruction = 0; + + for (size_t j = 0; j < 6; ++j) + instruction += (uint64_t)( + buffer[i + j + byte_pos]) + << (8 * j); + + uint64_t inst_norm = instruction >> bit_res; + + if (((inst_norm >> 37) & 0xF) == 0x5 + && ((inst_norm >> 9) & 0x7) == 0 + /* && (inst_norm & 0x3F)== 0 */ + ) { + uint32_t src = (uint32_t)( + (inst_norm >> 13) & 0xFFFFF); + src |= ((inst_norm >> 36) & 1) << 20; + + src <<= 4; + + uint32_t dest; + if (is_encoder) + dest = now_pos + (uint32_t)(i) + src; + else + dest = src - (now_pos + (uint32_t)(i)); + + dest >>= 4; + + inst_norm &= ~((uint64_t)(0x8FFFFF) << 13); + inst_norm |= (uint64_t)(dest & 0xFFFFF) << 13; + inst_norm |= (uint64_t)(dest & 0x100000) + << (36 - 20); + + instruction &= (1U << bit_res) - 1; + instruction |= (inst_norm << bit_res); + + for (size_t j = 0; j < 6; j++) + buffer[i + j + byte_pos] = (uint8_t)( + instruction + >> (8 * j)); + } + } + } + + return i; +} + + +static lzma_ret +ia64_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &ia64_code, 0, 16, 16, is_encoder); +} + + +#ifdef HAVE_ENCODER_IA64 +extern lzma_ret +lzma_simple_ia64_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return ia64_coder_init(next, allocator, filters, true); +} +#endif + + +#ifdef HAVE_DECODER_IA64 +extern lzma_ret +lzma_simple_ia64_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return ia64_coder_init(next, allocator, filters, false); +} +#endif diff --git a/src/liblzma/simple/powerpc.c b/src/liblzma/simple/powerpc.c new file mode 100644 index 0000000000..ea47d14d4c --- /dev/null +++ b/src/liblzma/simple/powerpc.c @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file powerpc.c +/// \brief Filter for PowerPC (big endian) binaries +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +powerpc_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + size &= ~(size_t)3; + + size_t i; + for (i = 0; i < size; i += 4) { + // PowerPC branch 6(48) 24(Offset) 1(Abs) 1(Link) + if ((buffer[i] >> 2) == 0x12 + && ((buffer[i + 3] & 3) == 1)) { + + const uint32_t src + = (((uint32_t)(buffer[i + 0]) & 3) << 24) + | ((uint32_t)(buffer[i + 1]) << 16) + | ((uint32_t)(buffer[i + 2]) << 8) + | ((uint32_t)(buffer[i + 3]) & ~UINT32_C(3)); + + uint32_t dest; + if (is_encoder) + dest = now_pos + (uint32_t)(i) + src; + else + dest = src - (now_pos + (uint32_t)(i)); + + buffer[i + 0] = 0x48 | ((dest >> 24) & 0x03); + buffer[i + 1] = (dest >> 16); + buffer[i + 2] = (dest >> 8); + buffer[i + 3] &= 0x03; + buffer[i + 3] |= dest; + } + } + + return i; +} + + +static lzma_ret +powerpc_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &powerpc_code, 0, 4, 4, is_encoder); +} + + +#ifdef HAVE_ENCODER_POWERPC +extern lzma_ret +lzma_simple_powerpc_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return powerpc_coder_init(next, allocator, filters, true); +} +#endif + + +#ifdef HAVE_DECODER_POWERPC +extern lzma_ret +lzma_simple_powerpc_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return powerpc_coder_init(next, allocator, filters, false); +} +#endif diff --git a/src/liblzma/simple/riscv.c b/src/liblzma/simple/riscv.c new file mode 100644 index 0000000000..bc97ebdbb0 --- /dev/null +++ b/src/liblzma/simple/riscv.c @@ -0,0 +1,773 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file riscv.c +/// \brief Filter for 32-bit/64-bit little/big endian RISC-V binaries +/// +/// This converts program counter relative addresses in function calls +/// (JAL, AUIPC+JALR), address calculation of functions and global +/// variables (AUIPC+ADDI), loads (AUIPC+load), and stores (AUIPC+store). +/// +/// For AUIPC+inst2 pairs, the paired instruction checking is fairly relaxed. +/// The paired instruction opcode must only have its lowest two bits set, +/// meaning it will convert any paired instruction that is not a 16-bit +/// compressed instruction. This was shown to be enough to keep the number +/// of false matches low while improving code size and speed. +// +// Authors: Lasse Collin +// Jia Tan +// +// Special thanks: +// +// - Chien Wong provided a few early versions of RISC-V +// filter variants along with test files and benchmark results. +// +// - Igor Pavlov helped a lot in the filter design, getting it both +// faster and smaller. The implementation here is still independently +// written, not based on LZMA SDK. +// +/////////////////////////////////////////////////////////////////////////////// + +/* + +RISC-V filtering +================ + + RV32I and RV64I, possibly combined with extensions C, Zfh, F, D, + and Q, are identical enough that the same filter works for both. + + The instruction encoding is always little endian, even on systems + with big endian data access. Thus the same filter works for both + endiannesses. + + The following instructions have program counter relative + (pc-relative) behavior: + +JAL +--- + + JAL is used for function calls (including tail calls) and + unconditional jumps within functions. Jumps within functions + aren't useful to filter because the absolute addresses often + appear only once or at most a few times. Tail calls and jumps + within functions look the same to a simple filter so neither + are filtered, that is, JAL x0 is ignored (the ABI name of the + register x0 is "zero"). + + Almost all calls store the return address to register x1 (ra) + or x5 (t0). To reduce false matches when the filter is applied + to non-code data, only the JAL instructions that use x1 or x5 + are converted. JAL has pc-relative range of +/-1 MiB so longer + calls and jumps need another method (AUIPC+JALR). + +C.J and C.JAL +------------- + + C.J and C.JAL have pc-relative range of +/-2 KiB. + + C.J is for tail calls and jumps within functions and isn't + filtered for the reasons mentioned for JAL x0. + + C.JAL is an RV32C-only instruction. Its encoding overlaps with + RV64C-only C.ADDIW which is a common instruction. So if filtering + C.JAL was useful (it wasn't tested) then a separate filter would + be needed for RV32 and RV64. Also, false positives would be a + significant problem when the filter is applied to non-code data + because C.JAL needs only five bits to match. Thus, this filter + doesn't modify C.JAL instructions. + +BEQ, BNE, BLT, BGE, BLTU, BGEU, C.BEQZ, and C.BNEZ +-------------------------------------------------- + + These are conditional branches with pc-relative range + of +/-4 KiB (+/-256 B for C.*). The absolute addresses often + appear only once and very short distances are the most common, + so filtering these instructions would make compression worse. + +AUIPC with rd != x0 +------------------- + + AUIPC is paired with a second instruction (inst2) to do + pc-relative jumps, calls, loads, stores, and for taking + an address of a symbol. AUIPC has a 20-bit immediate and + the possible inst2 choices have a 12-bit immediate. + + AUIPC stores pc + 20-bit signed immediate to a register. + The immediate encodes a multiple of 4 KiB so AUIPC itself + has a pc-relative range of +/-2 GiB. AUIPC does *NOT* set + the lowest 12 bits of the result to zero! This means that + the 12-bit immediate in inst2 cannot just include the lowest + 12 bits of the absolute address as is; the immediate has to + compensate for the lowest 12 bits that AUIPC copies from the + program counter. This means that a good filter has to convert + not only AUIPC but also the paired inst2. + + A strict filter would focus on filtering the following + AUIPC+inst2 pairs: + + - AUIPC+JALR: Function calls, including tail calls. + + - AUIPC+ADDI: Calculating the address of a function + or a global variable. + + - AUIPC+load/store from the base instruction sets + (RV32I, RV64I) or from the floating point extensions + Zfh, F, D, and Q: + * RV32I: LB, LH, LW, LBU, LHU, SB, SH, SW + * RV64I has also: LD, LWU, SD + * Zfh: FLH, FSH + * F: FLW, FSW + * D: FLD, FSD + * Q: FLQ, FSQ + + NOTE: AUIPC+inst2 can only be a pair if AUIPC's rd specifies + the same register as inst2's rs1. + + Instead of strictly accepting only the above instructions as inst2, + this filter uses a much simpler condition: the lowest two bits of + inst2 must be set, that is, inst2 must not be a 16-bit compressed + instruction. So this will accept all 32-bit and possible future + extended instructions as a pair to AUIPC if the bits in AUIPC's + rd [11:7] match the bits [19:15] in inst2 (the bits that I-type and + S-type instructions use for rs1). Testing showed that this relaxed + condition for inst2 did not consistently or significantly affect + compression ratio but it reduced code size and improved speed. + + Additionally, the paired instruction is always treated as an I-type + instruction. The S-type instructions used by stores (SB, SH, SW, + etc.) place the lowest 5 bits of the immediate in a different + location than I-type instructions. AUIPC+store pairs are less + common than other pairs, and testing showed that the extra + code required to handle S-type instructions was not worth the + compression ratio gained. + + AUIPC+inst2 don't necessarily appear sequentially next to each + other although very often they do. Especially AUIPC+JALR are + sequential as that may allow instruction fusion in processors + (and perhaps help branch prediction as a fused AUIPC+JALR is + a direct branch while JALR alone is an indirect branch). + + Clang 16 can generate code where AUIPC+inst2 is split: + + - AUIPC is outside a loop and inst2 (load/store) is inside + the loop. This way the AUIPC instruction needs to be + executed only once. + + - Load-modify-store may have AUIPC for the load and the same + AUIPC-result is used for the store too. This may get combined + with AUIPC being outside the loop. + + - AUIPC is before a conditional branch and inst2 is hundreds + of bytes away at the branch target. + + - Inner and outer pair: + + auipc a1,0x2f + auipc a2,0x3d + ld a2,-500(a2) + addi a1,a1,-233 + + - Many split pairs with an untaken conditional branch between: + + auipc s9,0x1613 # Pair 1 + auipc s4,0x1613 # Pair 2 + auipc s6,0x1613 # Pair 3 + auipc s10,0x1613 # Pair 4 + beqz a5,a3baae + ld a0,0(a6) + ld a6,246(s9) # Pair 1 + ld a1,250(s4) # Pair 2 + ld a3,254(s6) # Pair 3 + ld a4,258(s10) # Pair 4 + + It's not possible to find all split pairs in a filter like this. + At least in 2024, simple sequential pairs are 99 % of AUIPC uses + so filtering only such pairs gives good results and makes the + filter simpler. However, it's possible that future compilers will + produce different code where sequential pairs aren't as common. + + This filter doesn't convert AUIPC instructions alone because: + + (1) The conversion would be off-by-one (or off-by-4096) half the + time because the lowest 12 bits from inst2 (inst2_imm12) + aren't known. We only know that the absolute address is + pc + AUIPC_imm20 + [-2048, +2047] but there is no way to + know the exact 4096-byte multiple (or 4096 * n + 2048): + there are always two possibilities because AUIPC copies + the 12 lowest bits from pc instead of zeroing them. + + NOTE: The sign-extension of inst2_imm12 adds a tiny bit + of extra complexity to AUIPC math in general but it's not + the reason for this problem. The sign-extension only changes + the relative position of the pc-relative 4096-byte window. + + (2) Matching AUIPC instruction alone requires only seven bits. + When the filter is applied to non-code data, that leads + to many false positives which make compression worse. + As long as most AUIPC+inst2 pairs appear as two consecutive + instructions, converting only such pairs gives better results. + + In assembly, AUIPC+inst2 tend to look like this: + + # Call: + auipc ra, 0x12345 + jalr ra, -42(ra) + + # Tail call: + auipc t1, 0x12345 + jalr zero, -42(t1) + + # Getting the absolute address: + auipc a0, 0x12345 + addi a0, a0, -42 + + # rd of inst2 isn't necessarily the same as rs1 even + # in cases where there is no reason to preserve rs1. + auipc a0, 0x12345 + addi a1, a0, -42 + + As of 2024, 16-bit instructions from the C extension don't + appear as inst2. The RISC-V psABI doesn't list AUIPC+C.* as + a linker relaxation type explicitly but it's not disallowed + either. Usefulness is limited as most of the time the lowest + 12 bits won't fit in a C instruction. This filter doesn't + support AUIPC+C.* combinations because this makes the filter + simpler, there are no test files, and it hopefully will never + be needed anyway. + + (Compare AUIPC to ARM64 where ADRP does set the lowest 12 bits + to zero. The paired instruction has the lowest 12 bits of the + absolute address as is in a zero-extended immediate. Thus the + ARM64 filter doesn't need to care about the instructions that + are paired with ADRP. An off-by-4096 issue can still occur if + the code section isn't aligned with the filter's start offset. + It's not a problem with standalone ELF files but Windows PE + files need start_offset=3072 for best results. Also, a .tar + stores files with 512-byte alignment so most of the time it + won't be the best for ARM64.) + +AUIPC with rd == x0 +------------------- + + AUIPC instructions with rd=x0 are reserved for HINTs in the base + instruction set. Such AUIPC instructions are never filtered. + + As of January 2024, it seems likely that AUIPC with rd=x0 will + be used for landing pads (pseudoinstruction LPAD). LPAD is used + to mark valid targets for indirect jumps (for JALR), for example, + beginnings of functions. The 20-bit immediate in LPAD instruction + is a label, not a pc-relative address. Thus it would be + counterproductive to convert AUIPC instructions with rd=x0. + + Often the next instruction after LPAD won't have rs1=x0 and thus + the filtering would be skipped for that reason alone. However, + it's not good to rely on this. For example, consider a function + that begins like this: + + int foo(int i) + { + if (i <= 234) { + ... + } + + A compiler may generate something like this: + + lpad 0x54321 + li a5, 234 + bgt a0, a5, .L2 + + Converting the pseudoinstructions to raw instructions: + + auipc x0, 0x54321 + addi x15, x0, 234 + blt x15, x10, .L2 + + In this case the filter would undesirably convert the AUIPC+ADDI + pair if the filter didn't explicitly skip AUIPC instructions + that have rd=x0. + +*/ + + +#include "simple_private.h" + + +// This checks two conditions at once: +// - AUIPC rd == inst2 rs1. +// - inst2 opcode has the lowest two bits set. +// +// The 8 bit left shift aligns the rd of AUIPC with the rs1 of inst2. +// By XORing the registers, any non-zero value in those bits indicates the +// registers are not equal and thus not an AUIPC pair. Subtracting 3 from +// inst2 will zero out the first two opcode bits only when they are set. +// The mask tests if any of the register or opcode bits are set (and thus +// not an AUIPC pair). +// +// Alternative expression: (((((auipc) << 8) ^ (inst2)) & 0xF8003) != 3) +#define NOT_AUIPC_PAIR(auipc, inst2) \ + ((((auipc) << 8) ^ ((inst2) - 3)) & 0xF8003) + +// This macro checks multiple conditions: +// (1) AUIPC rd [11:7] == x2 (special rd value). +// (2) AUIPC bits 12 and 13 set (the lowest two opcode bits of packed inst2). +// (3) inst2_rs1 doesn't equal x0 or x2 because the opposite +// conversion is only done when +// auipc_rd != x0 && +// auipc_rd != x2 && +// auipc_rd == inst2_rs1. +// +// The left-hand side takes care of (1) and (2). +// (a) The lowest 7 bits are already known to be AUIPC so subtracting 0x17 +// makes those bits zeros. +// (b) If AUIPC rd equals x2, subtracting 0x100 makes bits [11:7] zeros. +// If rd doesn't equal x2, then there will be at least one non-zero bit +// and the next step (c) is irrelevant. +// (c) If the lowest two opcode bits of the packed inst2 are set in [13:12], +// then subtracting 0x3000 will make those bits zeros. Otherwise there +// will be at least one non-zero bit. +// +// The shift by 18 removes the high bits from the final '>=' comparison and +// ensures that any non-zero result will be larger than any possible result +// from the right-hand side of the comparison. The cast ensures that the +// left-hand side didn't get promoted to a larger type than uint32_t. +// +// On the right-hand side, inst2_rs1 & 0x1D will be non-zero as long as +// inst2_rs1 is not x0 or x2. +// +// The final '>=' comparison will make the expression true if: +// - The subtraction caused any bits to be set (special AUIPC rd value not +// used or inst2 opcode bits not set). (non-zero >= non-zero or 0) +// - The subtraction did not cause any bits to be set but inst2_rs1 was +// x0 or x2. (0 >= 0) +#define NOT_SPECIAL_AUIPC(auipc, inst2_rs1) \ + ((uint32_t)(((auipc) - 0x3117) << 18) >= ((inst2_rs1) & 0x1D)) + + +// The encode and decode functions are split for this filter because of the +// AUIPC+inst2 filtering. This filter design allows a decoder-only +// implementation to be smaller than alternative designs. + +#ifdef HAVE_ENCODER_RISCV +static size_t +riscv_encode(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, + bool is_encoder lzma_attribute((__unused__)), + uint8_t *buffer, size_t size) +{ + // Avoid using i + 8 <= size in the loop condition. + // + // NOTE: If there is a JAL in the last six bytes of the stream, it + // won't be converted. This is intentional to keep the code simpler. + if (size < 8) + return 0; + + size -= 8; + + size_t i; + + // The loop is advanced by 2 bytes every iteration since the + // instruction stream may include 16-bit instructions (C extension). + for (i = 0; i <= size; i += 2) { + uint32_t inst = buffer[i]; + + if (inst == 0xEF) { + // JAL + const uint32_t b1 = buffer[i + 1]; + + // Only filter rd=x1(ra) and rd=x5(t0). + if ((b1 & 0x0D) != 0) + continue; + + // The 20-bit immediate is in four pieces. + // The encoder stores it in big endian form + // since it improves compression slightly. + const uint32_t b2 = buffer[i + 2]; + const uint32_t b3 = buffer[i + 3]; + const uint32_t pc = now_pos + (uint32_t)i; + +// The following chart shows the highest three bytes of JAL, focusing on +// the 20-bit immediate field [31:12]. The first row of numbers is the +// bit position in a 32-bit little endian instruction. The second row of +// numbers shows the order of the immediate field in a J-type instruction. +// The last row is the bit number in each byte. +// +// To determine the amount to shift each bit, subtract the value in +// the last row from the value in the second last row. If the number +// is positive, shift left. If negative, shift right. +// +// For example, at the rightmost side of the chart, the bit 4 in b1 is +// the bit 12 of the address. Thus that bit needs to be shifted left +// by 12 - 4 = 8 bits to put it in the right place in the addr variable. +// +// NOTE: The immediate of a J-type instruction holds bits [20:1] of +// the address. The bit [0] is always 0 and not part of the immediate. +// +// | b3 | b2 | b1 | +// | 31 30 29 28 27 26 25 24 | 23 22 21 20 19 18 17 16 | 15 14 13 12 x x x x | +// | 20 10 9 8 7 6 5 4 | 3 2 1 11 19 18 17 16 | 15 14 13 12 x x x x | +// | 7 6 5 4 3 2 1 0 | 7 6 5 4 3 2 1 0 | 7 6 5 4 x x x x | + + uint32_t addr = ((b1 & 0xF0) << 8) + | ((b2 & 0x0F) << 16) + | ((b2 & 0x10) << 7) + | ((b2 & 0xE0) >> 4) + | ((b3 & 0x7F) << 4) + | ((b3 & 0x80) << 13); + + addr += pc; + + buffer[i + 1] = (uint8_t)((b1 & 0x0F) + | ((addr >> 13) & 0xF0)); + + buffer[i + 2] = (uint8_t)(addr >> 9); + buffer[i + 3] = (uint8_t)(addr >> 1); + + // The "-2" is included because the for-loop will + // always increment by 2. In this case, we want to + // skip an extra 2 bytes since we used 4 bytes + // of input. + i += 4 - 2; + + } else if ((inst & 0x7F) == 0x17) { + // AUIPC + inst |= (uint32_t)buffer[i + 1] << 8; + inst |= (uint32_t)buffer[i + 2] << 16; + inst |= (uint32_t)buffer[i + 3] << 24; + + // Branch based on AUIPC's rd. The bitmask test does + // the same thing as this: + // + // const uint32_t auipc_rd = (inst >> 7) & 0x1F; + // if (auipc_rd != 0 && auipc_rd != 2) { + if (inst & 0xE80) { + // AUIPC's rd doesn't equal x0 or x2. + + // Check if AUIPC+inst2 are a pair. + uint32_t inst2 = read32le(buffer + i + 4); + + if (NOT_AUIPC_PAIR(inst, inst2)) { + // The NOT_AUIPC_PAIR macro allows + // a false AUIPC+AUIPC pair if the + // bits [19:15] (where rs1 would be) + // in the second AUIPC match the rd + // of the first AUIPC. + // + // We must skip enough forward so + // that the first two bytes of the + // second AUIPC cannot get converted. + // Such a conversion could make the + // current pair become a valid pair + // which would desync the decoder. + // + // Skipping six bytes is enough even + // though the above condition looks + // at the lowest four bits of the + // buffer[i + 6] too. This is safe + // because this filter never changes + // those bits if a conversion at + // that position is done. + i += 6 - 2; + continue; + } + + // Convert AUIPC+inst2 to a special format: + // + // - The lowest 7 bits [6:0] retain the + // AUIPC opcode. + // + // - The rd [11:7] is set to x2(sp). x2 is + // used as the stack pointer so AUIPC with + // rd=x2 should be very rare in real-world + // executables. + // + // - The remaining 20 bits [31:12] (that + // normally hold the pc-relative immediate) + // are used to store the lowest 20 bits of + // inst2. That is, the 12-bit immediate of + // inst2 is not included. + // + // - The location of the original inst2 is + // used to store the 32-bit absolute + // address in big endian format. Compared + // to the 20+12-bit split encoding, this + // results in a longer uninterrupted + // sequence of identical common bytes + // when the same address is referred + // with different instruction pairs + // (like AUIPC+LD vs. AUIPC+ADDI) or + // when the occurrences of the same + // pair use different registers. When + // referring to adjacent memory locations + // (like function calls that go via the + // ELF PLT), in big endian order only the + // last 1-2 bytes differ; in little endian + // the differing 1-2 bytes would be in the + // middle of the 8-byte sequence. + // + // When reversing the transformation, the + // original rd of AUIPC can be restored + // from inst2's rs1 as they are required to + // be the same. + + // Arithmetic right shift makes sign extension + // trivial but (1) it's implementation-defined + // behavior (C99/C11/C23 6.5.7-p5) and so is + // (2) casting unsigned to signed (6.3.1.3-p3). + // + // One can check for (1) with + // + // if ((-1 >> 1) == -1) ... + // + // but (2) has to be checked from the + // compiler docs. GCC promises that (1) + // and (2) behave in the common expected + // way and thus + // + // addr += (uint32_t)( + // (int32_t)inst2 >> 20); + // + // does the same as the code below. But since + // the 100 % portable way is only a few bytes + // bigger code and there is no real speed + // difference, let's just use that, especially + // since the decoder doesn't need this at all. + uint32_t addr = inst & 0xFFFFF000; + addr += (inst2 >> 20) + - ((inst2 >> 19) & 0x1000); + + addr += now_pos + (uint32_t)i; + + // Construct the first 32 bits: + // [6:0] AUIPC opcode + // [11:7] Special AUIPC rd = x2 + // [31:12] The lowest 20 bits of inst2 + inst = 0x17 | (2 << 7) | (inst2 << 12); + + write32le(buffer + i, inst); + + // The second 32 bits store the absolute + // address in big endian order. + write32be(buffer + i + 4, addr); + } else { + // AUIPC's rd equals x0 or x2. + // + // x0 indicates a landing pad (LPAD). + // It's always skipped. + // + // AUIPC with rd == x2 is used for the special + // format as explained above. When the input + // contains a byte sequence that matches the + // special format, "fake" decoding must be + // done to keep the filter bijective (that + // is, safe to apply on arbitrary data). + // + // See the "x0 or x2" section in riscv_decode() + // for how the "real" decoding is done. The + // "fake" decoding is a simplified version + // of "real" decoding with the following + // differences (these reduce code size of + // the decoder): + // (1) The lowest 12 bits aren't sign-extended. + // (2) No address conversion is done. + // (3) Big endian format isn't used (the fake + // address is in little endian order). + + // Check if inst matches the special format. + const uint32_t fake_rs1 = inst >> 27; + + if (NOT_SPECIAL_AUIPC(inst, fake_rs1)) { + i += 4 - 2; + continue; + } + + const uint32_t fake_addr = + read32le(buffer + i + 4); + + // Construct the second 32 bits: + // [19:0] Upper 20 bits from AUIPC + // [31:20] The lowest 12 bits of fake_addr + const uint32_t fake_inst2 = (inst >> 12) + | (fake_addr << 20); + + // Construct new first 32 bits from: + // [6:0] AUIPC opcode + // [11:7] Fake AUIPC rd = fake_rs1 + // [31:12] The highest 20 bits of fake_addr + inst = 0x17 | (fake_rs1 << 7) + | (fake_addr & 0xFFFFF000); + + write32le(buffer + i, inst); + write32le(buffer + i + 4, fake_inst2); + } + + i += 8 - 2; + } + } + + return i; +} + + +extern lzma_ret +lzma_simple_riscv_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return lzma_simple_coder_init(next, allocator, filters, + &riscv_encode, 0, 8, 2, true); +} + + +extern LZMA_API(size_t) +lzma_bcj_riscv_encode(uint32_t start_offset, uint8_t *buf, size_t size) +{ + // start_offset must be a multiple of two. + start_offset &= ~UINT32_C(1); + return riscv_encode(NULL, start_offset, true, buf, size); +} +#endif + + +#ifdef HAVE_DECODER_RISCV +static size_t +riscv_decode(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, + bool is_encoder lzma_attribute((__unused__)), + uint8_t *buffer, size_t size) +{ + if (size < 8) + return 0; + + size -= 8; + + size_t i; + for (i = 0; i <= size; i += 2) { + uint32_t inst = buffer[i]; + + if (inst == 0xEF) { + // JAL + const uint32_t b1 = buffer[i + 1]; + + // Only filter rd=x1(ra) and rd=x5(t0). + if ((b1 & 0x0D) != 0) + continue; + + const uint32_t b2 = buffer[i + 2]; + const uint32_t b3 = buffer[i + 3]; + const uint32_t pc = now_pos + (uint32_t)i; + +// | b3 | b2 | b1 | +// | 31 30 29 28 27 26 25 24 | 23 22 21 20 19 18 17 16 | 15 14 13 12 x x x x | +// | 20 10 9 8 7 6 5 4 | 3 2 1 11 19 18 17 16 | 15 14 13 12 x x x x | +// | 7 6 5 4 3 2 1 0 | 7 6 5 4 3 2 1 0 | 7 6 5 4 x x x x | + + uint32_t addr = ((b1 & 0xF0) << 13) + | (b2 << 9) | (b3 << 1); + + addr -= pc; + + buffer[i + 1] = (uint8_t)((b1 & 0x0F) + | ((addr >> 8) & 0xF0)); + + buffer[i + 2] = (uint8_t)(((addr >> 16) & 0x0F) + | ((addr >> 7) & 0x10) + | ((addr << 4) & 0xE0)); + + buffer[i + 3] = (uint8_t)(((addr >> 4) & 0x7F) + | ((addr >> 13) & 0x80)); + + i += 4 - 2; + + } else if ((inst & 0x7F) == 0x17) { + // AUIPC + uint32_t inst2; + + inst |= (uint32_t)buffer[i + 1] << 8; + inst |= (uint32_t)buffer[i + 2] << 16; + inst |= (uint32_t)buffer[i + 3] << 24; + + if (inst & 0xE80) { + // AUIPC's rd doesn't equal x0 or x2. + + // Check if it is a "fake" AUIPC+inst2 pair. + inst2 = read32le(buffer + i + 4); + + if (NOT_AUIPC_PAIR(inst, inst2)) { + i += 6 - 2; + continue; + } + + // Decode (or more like re-encode) the "fake" + // pair. The "fake" format doesn't do + // sign-extension, address conversion, or + // use big endian. (The use of little endian + // allows sharing the write32le() calls in + // the decoder to reduce code size when + // unaligned access isn't supported.) + uint32_t addr = inst & 0xFFFFF000; + addr += inst2 >> 20; + + inst = 0x17 | (2 << 7) | (inst2 << 12); + inst2 = addr; + } else { + // AUIPC's rd equals x0 or x2. + + // Check if inst matches the special format + // used by the encoder. + const uint32_t inst2_rs1 = inst >> 27; + + if (NOT_SPECIAL_AUIPC(inst, inst2_rs1)) { + i += 4 - 2; + continue; + } + + // Decode the "real" pair. + uint32_t addr = read32be(buffer + i + 4); + + addr -= now_pos + (uint32_t)i; + + // The second instruction: + // - Get the lowest 20 bits from inst. + // - Add the lowest 12 bits of the address + // as the immediate field. + inst2 = (inst >> 12) | (addr << 20); + + // AUIPC: + // - rd is the same as inst2_rs1. + // - The sign extension of the lowest 12 bits + // must be taken into account. + inst = 0x17 | (inst2_rs1 << 7) + | ((addr + 0x800) & 0xFFFFF000); + } + + // Both decoder branches write in little endian order. + write32le(buffer + i, inst); + write32le(buffer + i + 4, inst2); + + i += 8 - 2; + } + } + + return i; +} + + +extern lzma_ret +lzma_simple_riscv_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return lzma_simple_coder_init(next, allocator, filters, + &riscv_decode, 0, 8, 2, false); +} + + +extern LZMA_API(size_t) +lzma_bcj_riscv_decode(uint32_t start_offset, uint8_t *buf, size_t size) +{ + // start_offset must be a multiple of two. + start_offset &= ~UINT32_C(1); + return riscv_decode(NULL, start_offset, false, buf, size); +} +#endif diff --git a/src/liblzma/simple/simple_coder.c b/src/liblzma/simple/simple_coder.c new file mode 100644 index 0000000000..5cbfa82270 --- /dev/null +++ b/src/liblzma/simple/simple_coder.c @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_coder.c +/// \brief Wrapper for simple filters +/// +/// Simple filters don't change the size of the data i.e. number of bytes +/// in equals the number of bytes out. +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +/// Copied or encodes/decodes more data to out[]. +static lzma_ret +copy_or_code(lzma_simple_coder *coder, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + assert(!coder->end_was_reached); + + if (coder->next.code == NULL) { + lzma_bufcpy(in, in_pos, in_size, out, out_pos, out_size); + + // Check if end of stream was reached. + if (coder->is_encoder && action == LZMA_FINISH + && *in_pos == in_size) + coder->end_was_reached = true; + + } else { + // Call the next coder in the chain to provide us some data. + const lzma_ret ret = coder->next.code( + coder->next.coder, allocator, + in, in_pos, in_size, + out, out_pos, out_size, action); + + if (ret == LZMA_STREAM_END) { + assert(!coder->is_encoder + || action == LZMA_FINISH); + coder->end_was_reached = true; + + } else if (ret != LZMA_OK) { + return ret; + } + } + + return LZMA_OK; +} + + +static size_t +call_filter(lzma_simple_coder *coder, uint8_t *buffer, size_t size) +{ + const size_t filtered = coder->filter(coder->simple, + coder->now_pos, coder->is_encoder, + buffer, size); + coder->now_pos += filtered; + return filtered; +} + + +static lzma_ret +simple_code(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_simple_coder *coder = coder_ptr; + + // TODO: Add partial support for LZMA_SYNC_FLUSH. We can support it + // in cases when the filter is able to filter everything. With most + // simple filters it can be done at offset that is a multiple of 2, + // 4, or 16. With x86 filter, it needs good luck, and thus cannot + // be made to work predictably. + if (action == LZMA_SYNC_FLUSH) + return LZMA_OPTIONS_ERROR; + + // Flush already filtered data from coder->buffer[] to out[]. + if (coder->pos < coder->filtered) { + lzma_bufcpy(coder->buffer, &coder->pos, coder->filtered, + out, out_pos, out_size); + + // If we couldn't flush all the filtered data, return to + // application immediately. + if (coder->pos < coder->filtered) + return LZMA_OK; + + if (coder->end_was_reached) { + assert(coder->filtered == coder->size); + return LZMA_STREAM_END; + } + } + + // If we get here, there is no filtered data left in the buffer. + coder->filtered = 0; + + assert(!coder->end_was_reached); + + // If there is more output space left than there is unfiltered data + // in coder->buffer[], flush coder->buffer[] to out[], and copy/code + // more data to out[] hopefully filling it completely. Then filter + // the data in out[]. This step is where most of the data gets + // filtered if the buffer sizes used by the application are reasonable. + const size_t out_avail = out_size - *out_pos; + const size_t buf_avail = coder->size - coder->pos; + if (out_avail > buf_avail || buf_avail == 0) { + // Store the old position so that we know from which byte + // to start filtering. + const size_t out_start = *out_pos; + + // Flush data from coder->buffer[] to out[], but don't reset + // coder->pos and coder->size yet. This way the coder can be + // restarted if the next filter in the chain returns e.g. + // LZMA_MEM_ERROR. + // + // Do the memcpy() conditionally because out can be NULL + // (in which case buf_avail is always 0). Calling memcpy() + // with a null-pointer is undefined even if the third + // argument is 0. + if (buf_avail > 0) + memcpy(out + *out_pos, coder->buffer + coder->pos, + buf_avail); + + *out_pos += buf_avail; + + // Copy/Encode/Decode more data to out[]. + { + const lzma_ret ret = copy_or_code(coder, allocator, + in, in_pos, in_size, + out, out_pos, out_size, action); + assert(ret != LZMA_STREAM_END); + if (ret != LZMA_OK) + return ret; + } + + // Filter out[] unless there is nothing to filter. + // This way we avoid null pointer + 0 (undefined behavior) + // when out == NULL. + const size_t size = *out_pos - out_start; + const size_t filtered = size == 0 ? 0 : call_filter( + coder, out + out_start, size); + + const size_t unfiltered = size - filtered; + assert(unfiltered <= coder->allocated / 2); + + // Now we can update coder->pos and coder->size, because + // the next coder in the chain (if any) was successful. + coder->pos = 0; + coder->size = unfiltered; + + if (coder->end_was_reached) { + // The last byte has been copied to out[] already. + // They are left as is. + coder->size = 0; + + } else if (unfiltered > 0) { + // There is unfiltered data left in out[]. Copy it to + // coder->buffer[] and rewind *out_pos appropriately. + *out_pos -= unfiltered; + memcpy(coder->buffer, out + *out_pos, unfiltered); + } + } else if (coder->pos > 0) { + memmove(coder->buffer, coder->buffer + coder->pos, buf_avail); + coder->size -= coder->pos; + coder->pos = 0; + } + + assert(coder->pos == 0); + + // If coder->buffer[] isn't empty, try to fill it by copying/decoding + // more data. Then filter coder->buffer[] and copy the successfully + // filtered data to out[]. It is probable, that some filtered and + // unfiltered data will be left to coder->buffer[]. + if (coder->size > 0) { + { + const lzma_ret ret = copy_or_code(coder, allocator, + in, in_pos, in_size, + coder->buffer, &coder->size, + coder->allocated, action); + assert(ret != LZMA_STREAM_END); + if (ret != LZMA_OK) + return ret; + } + + coder->filtered = call_filter( + coder, coder->buffer, coder->size); + + // Everything is considered to be filtered if coder->buffer[] + // contains the last bytes of the data. + if (coder->end_was_reached) + coder->filtered = coder->size; + + // Flush as much as possible. + lzma_bufcpy(coder->buffer, &coder->pos, coder->filtered, + out, out_pos, out_size); + } + + // Check if we got everything done. + if (coder->end_was_reached && coder->pos == coder->size) + return LZMA_STREAM_END; + + return LZMA_OK; +} + + +static void +simple_coder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_simple_coder *coder = coder_ptr; + lzma_next_end(&coder->next, allocator); + lzma_free(coder->simple, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +simple_coder_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters_null lzma_attribute((__unused__)), + const lzma_filter *reversed_filters) +{ + lzma_simple_coder *coder = coder_ptr; + + // No update support, just call the next filter in the chain. + return lzma_next_filter_update( + &coder->next, allocator, reversed_filters + 1); +} + + +extern lzma_ret +lzma_simple_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, + size_t (*filter)(void *simple, uint32_t now_pos, + bool is_encoder, uint8_t *buffer, size_t size), + size_t simple_size, size_t unfiltered_max, + uint32_t alignment, bool is_encoder) +{ + // Allocate memory for the lzma_simple_coder structure if needed. + lzma_simple_coder *coder = next->coder; + if (coder == NULL) { + // Here we allocate space also for the temporary buffer. We + // need twice the size of unfiltered_max, because then it + // is always possible to filter at least unfiltered_max bytes + // more data in coder->buffer[] if it can be filled completely. + coder = lzma_alloc(sizeof(lzma_simple_coder) + + 2 * unfiltered_max, allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &simple_code; + next->end = &simple_coder_end; + next->update = &simple_coder_update; + + coder->next = LZMA_NEXT_CODER_INIT; + coder->filter = filter; + coder->allocated = 2 * unfiltered_max; + + // Allocate memory for filter-specific data structure. + if (simple_size > 0) { + coder->simple = lzma_alloc(simple_size, allocator); + if (coder->simple == NULL) + return LZMA_MEM_ERROR; + } else { + coder->simple = NULL; + } + } + + if (filters[0].options != NULL) { + const lzma_options_bcj *simple = filters[0].options; + coder->now_pos = simple->start_offset; + if (coder->now_pos & (alignment - 1)) + return LZMA_OPTIONS_ERROR; + } else { + coder->now_pos = 0; + } + + // Reset variables. + coder->is_encoder = is_encoder; + coder->end_was_reached = false; + coder->pos = 0; + coder->filtered = 0; + coder->size = 0; + + return lzma_next_filter_init(&coder->next, allocator, filters + 1); +} diff --git a/src/liblzma/simple/simple_coder.h b/src/liblzma/simple/simple_coder.h new file mode 100644 index 0000000000..2b762d5007 --- /dev/null +++ b/src/liblzma/simple/simple_coder.h @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_coder.h +/// \brief Wrapper for simple filters +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_SIMPLE_CODER_H +#define LZMA_SIMPLE_CODER_H + +#include "common.h" + + +extern lzma_ret lzma_simple_x86_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_x86_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_powerpc_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_powerpc_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_ia64_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_ia64_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_arm_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_arm_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_armthumb_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_armthumb_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_arm64_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_arm64_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_sparc_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_sparc_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + +extern lzma_ret lzma_simple_riscv_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_riscv_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +#endif diff --git a/src/liblzma/simple/simple_decoder.c b/src/liblzma/simple/simple_decoder.c new file mode 100644 index 0000000000..d9820ee8ed --- /dev/null +++ b/src/liblzma/simple/simple_decoder.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_decoder.c +/// \brief Properties decoder for simple filters +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_decoder.h" + + +extern lzma_ret +lzma_simple_props_decode(void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size) +{ + if (props_size == 0) + return LZMA_OK; + + if (props_size != 4) + return LZMA_OPTIONS_ERROR; + + lzma_options_bcj *opt = lzma_alloc( + sizeof(lzma_options_bcj), allocator); + if (opt == NULL) + return LZMA_MEM_ERROR; + + opt->start_offset = read32le(props); + + // Don't leave an options structure allocated if start_offset is zero. + if (opt->start_offset == 0) + lzma_free(opt, allocator); + else + *options = opt; + + return LZMA_OK; +} diff --git a/src/liblzma/simple/simple_decoder.h b/src/liblzma/simple/simple_decoder.h new file mode 100644 index 0000000000..2ae87bb863 --- /dev/null +++ b/src/liblzma/simple/simple_decoder.h @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_decoder.h +/// \brief Properties decoder for simple filters +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_SIMPLE_DECODER_H +#define LZMA_SIMPLE_DECODER_H + +#include "simple_coder.h" + +extern lzma_ret lzma_simple_props_decode( + void **options, const lzma_allocator *allocator, + const uint8_t *props, size_t props_size); + +#endif diff --git a/src/liblzma/simple/simple_encoder.c b/src/liblzma/simple/simple_encoder.c new file mode 100644 index 0000000000..d1f35096e2 --- /dev/null +++ b/src/liblzma/simple/simple_encoder.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_encoder.c +/// \brief Properties encoder for simple filters +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_encoder.h" + + +extern lzma_ret +lzma_simple_props_size(uint32_t *size, const void *options) +{ + const lzma_options_bcj *const opt = options; + *size = (opt == NULL || opt->start_offset == 0) ? 0 : 4; + return LZMA_OK; +} + + +extern lzma_ret +lzma_simple_props_encode(const void *options, uint8_t *out) +{ + const lzma_options_bcj *const opt = options; + + // The default start offset is zero, so we don't need to store any + // options unless the start offset is non-zero. + if (opt == NULL || opt->start_offset == 0) + return LZMA_OK; + + write32le(out, opt->start_offset); + + return LZMA_OK; +} diff --git a/src/liblzma/simple/simple_encoder.h b/src/liblzma/simple/simple_encoder.h new file mode 100644 index 0000000000..bf5edbb1c3 --- /dev/null +++ b/src/liblzma/simple/simple_encoder.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_encoder.c +/// \brief Properties encoder for simple filters +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_SIMPLE_ENCODER_H +#define LZMA_SIMPLE_ENCODER_H + +#include "simple_coder.h" + + +extern lzma_ret lzma_simple_props_size(uint32_t *size, const void *options); + +extern lzma_ret lzma_simple_props_encode(const void *options, uint8_t *out); + +#endif diff --git a/src/liblzma/simple/simple_private.h b/src/liblzma/simple/simple_private.h new file mode 100644 index 0000000000..7aa360ff49 --- /dev/null +++ b/src/liblzma/simple/simple_private.h @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file simple_private.h +/// \brief Private definitions for so called simple filters +// +// Author: Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_SIMPLE_PRIVATE_H +#define LZMA_SIMPLE_PRIVATE_H + +#include "simple_coder.h" + + +typedef struct { + /// Next filter in the chain + lzma_next_coder next; + + /// True if the next coder in the chain has returned LZMA_STREAM_END. + bool end_was_reached; + + /// True if filter() should encode the data; false to decode. + /// Currently all simple filters use the same function for encoding + /// and decoding, because the difference between encoders and decoders + /// is very small. + bool is_encoder; + + /// Pointer to filter-specific function, which does + /// the actual filtering. + size_t (*filter)(void *simple, uint32_t now_pos, + bool is_encoder, uint8_t *buffer, size_t size); + + /// Pointer to filter-specific data, or NULL if filter doesn't need + /// any extra data. + void *simple; + + /// The lowest 32 bits of the current position in the data. Most + /// filters need this to do conversions between absolute and relative + /// addresses. + uint32_t now_pos; + + /// Size of the memory allocated for the buffer. + size_t allocated; + + /// Flushing position in the temporary buffer. buffer[pos] is the + /// next byte to be copied to out[]. + size_t pos; + + /// buffer[filtered] is the first unfiltered byte. When pos is smaller + /// than filtered, there is unflushed filtered data in the buffer. + size_t filtered; + + /// Total number of bytes (both filtered and unfiltered) currently + /// in the temporary buffer. + size_t size; + + /// Temporary buffer + uint8_t buffer[]; +} lzma_simple_coder; + + +extern lzma_ret lzma_simple_coder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters, + size_t (*filter)(void *simple, uint32_t now_pos, + bool is_encoder, uint8_t *buffer, size_t size), + size_t simple_size, size_t unfiltered_max, + uint32_t alignment, bool is_encoder); + +#endif diff --git a/src/liblzma/simple/sparc.c b/src/liblzma/simple/sparc.c new file mode 100644 index 0000000000..1fa4850458 --- /dev/null +++ b/src/liblzma/simple/sparc.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file sparc.c +/// \brief Filter for SPARC binaries +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +sparc_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + size &= ~(size_t)3; + + size_t i; + for (i = 0; i < size; i += 4) { + if ((buffer[i] == 0x40 && (buffer[i + 1] & 0xC0) == 0x00) + || (buffer[i] == 0x7F + && (buffer[i + 1] & 0xC0) == 0xC0)) { + + uint32_t src = ((uint32_t)buffer[i + 0] << 24) + | ((uint32_t)buffer[i + 1] << 16) + | ((uint32_t)buffer[i + 2] << 8) + | ((uint32_t)buffer[i + 3]); + + src <<= 2; + + uint32_t dest; + if (is_encoder) + dest = now_pos + (uint32_t)(i) + src; + else + dest = src - (now_pos + (uint32_t)(i)); + + dest >>= 2; + + dest = (((0 - ((dest >> 22) & 1)) << 22) & 0x3FFFFFFF) + | (dest & 0x3FFFFF) + | 0x40000000; + + buffer[i + 0] = (uint8_t)(dest >> 24); + buffer[i + 1] = (uint8_t)(dest >> 16); + buffer[i + 2] = (uint8_t)(dest >> 8); + buffer[i + 3] = (uint8_t)(dest); + } + } + + return i; +} + + +static lzma_ret +sparc_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &sparc_code, 0, 4, 4, is_encoder); +} + + +#ifdef HAVE_ENCODER_SPARC +extern lzma_ret +lzma_simple_sparc_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return sparc_coder_init(next, allocator, filters, true); +} +#endif + + +#ifdef HAVE_DECODER_SPARC +extern lzma_ret +lzma_simple_sparc_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return sparc_coder_init(next, allocator, filters, false); +} +#endif diff --git a/src/liblzma/simple/x86.c b/src/liblzma/simple/x86.c new file mode 100644 index 0000000000..dffa786313 --- /dev/null +++ b/src/liblzma/simple/x86.c @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: 0BSD + +/////////////////////////////////////////////////////////////////////////////// +// +/// \file x86.c +/// \brief Filter for x86 binaries (BCJ filter) +/// +// Authors: Igor Pavlov +// Lasse Collin +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +#define Test86MSByte(b) ((b) == 0 || (b) == 0xFF) + + +typedef struct { + uint32_t prev_mask; + uint32_t prev_pos; +} lzma_simple_x86; + + +static size_t +x86_code(void *simple_ptr, uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + static const uint32_t MASK_TO_BIT_NUMBER[5] = { 0, 1, 2, 2, 3 }; + + lzma_simple_x86 *simple = simple_ptr; + uint32_t prev_mask = simple->prev_mask; + uint32_t prev_pos = simple->prev_pos; + + if (size < 5) + return 0; + + if (now_pos - prev_pos > 5) + prev_pos = now_pos - 5; + + const size_t limit = size - 5; + size_t buffer_pos = 0; + + while (buffer_pos <= limit) { + uint8_t b = buffer[buffer_pos]; + if (b != 0xE8 && b != 0xE9) { + ++buffer_pos; + continue; + } + + const uint32_t offset = now_pos + (uint32_t)(buffer_pos) + - prev_pos; + prev_pos = now_pos + (uint32_t)(buffer_pos); + + if (offset > 5) { + prev_mask = 0; + } else { + for (uint32_t i = 0; i < offset; ++i) { + prev_mask &= 0x77; + prev_mask <<= 1; + } + } + + b = buffer[buffer_pos + 4]; + + if (Test86MSByte(b) && (prev_mask >> 1) <= 4 + && (prev_mask >> 1) != 3) { + + uint32_t src = ((uint32_t)(b) << 24) + | ((uint32_t)(buffer[buffer_pos + 3]) << 16) + | ((uint32_t)(buffer[buffer_pos + 2]) << 8) + | (buffer[buffer_pos + 1]); + + uint32_t dest; + while (true) { + if (is_encoder) + dest = src + (now_pos + (uint32_t)( + buffer_pos) + 5); + else + dest = src - (now_pos + (uint32_t)( + buffer_pos) + 5); + + if (prev_mask == 0) + break; + + const uint32_t i = MASK_TO_BIT_NUMBER[ + prev_mask >> 1]; + + b = (uint8_t)(dest >> (24 - i * 8)); + + if (!Test86MSByte(b)) + break; + + src = dest ^ ((1U << (32 - i * 8)) - 1); + } + + buffer[buffer_pos + 4] + = (uint8_t)(~(((dest >> 24) & 1) - 1)); + buffer[buffer_pos + 3] = (uint8_t)(dest >> 16); + buffer[buffer_pos + 2] = (uint8_t)(dest >> 8); + buffer[buffer_pos + 1] = (uint8_t)(dest); + buffer_pos += 5; + prev_mask = 0; + + } else { + ++buffer_pos; + prev_mask |= 1; + if (Test86MSByte(b)) + prev_mask |= 0x10; + } + } + + simple->prev_mask = prev_mask; + simple->prev_pos = prev_pos; + + return buffer_pos; +} + + +static lzma_ret +x86_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + const lzma_ret ret = lzma_simple_coder_init(next, allocator, filters, + &x86_code, sizeof(lzma_simple_x86), 5, 1, is_encoder); + + if (ret == LZMA_OK) { + lzma_simple_coder *coder = next->coder; + lzma_simple_x86 *simple = coder->simple; + simple->prev_mask = 0; + simple->prev_pos = (uint32_t)(-5); + } + + return ret; +} + + +#ifdef HAVE_ENCODER_X86 +extern lzma_ret +lzma_simple_x86_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return x86_coder_init(next, allocator, filters, true); +} + + +extern LZMA_API(size_t) +lzma_bcj_x86_encode(uint32_t start_offset, uint8_t *buf, size_t size) +{ + lzma_simple_x86 simple = { + .prev_mask = 0, + .prev_pos = (uint32_t)(-5), + }; + + return x86_code(&simple, start_offset, true, buf, size); +} +#endif + + +#ifdef HAVE_DECODER_X86 +extern lzma_ret +lzma_simple_x86_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return x86_coder_init(next, allocator, filters, false); +} + + +extern LZMA_API(size_t) +lzma_bcj_x86_decode(uint32_t start_offset, uint8_t *buf, size_t size) +{ + lzma_simple_x86 simple = { + .prev_mask = 0, + .prev_pos = (uint32_t)(-5), + }; + + return x86_code(&simple, start_offset, false, buf, size); +} +#endif diff --git a/src/liblzma/validate_map.sh b/src/liblzma/validate_map.sh new file mode 100644 index 0000000000..dd1589d236 --- /dev/null +++ b/src/liblzma/validate_map.sh @@ -0,0 +1,163 @@ +#!/bin/sh +# SPDX-License-Identifier: 0BSD + +############################################################################### +# +# Check liblzma_*.map for certain types of errors. +# +# liblzma_generic.map is for FreeBSD and Solaris and possibly others +# except GNU/Linux. +# +# liblzma_linux.map is for GNU/Linux only. This and the matching extra code +# in the .c files make liblzma >= 5.2.7 compatible with binaries that were +# linked against ill-patched liblzma in RHEL/CentOS 7. By providing the +# compatibility in official XZ Utils release will hopefully prevent people +# from further copying the broken patch to other places when they want +# compatibility with binaries linked on RHEL/CentOS 7. The long version +# of the story: +# +# RHEL/CentOS 7 shipped with 5.1.2alpha, including the threaded +# encoder that is behind #ifdef LZMA_UNSTABLE in the API headers. +# In 5.1.2alpha these symbols are under XZ_5.1.2alpha in liblzma.map. +# API/ABI compatibility tracking isn't done between development +# releases so newer releases didn't have XZ_5.1.2alpha anymore. +# +# Later RHEL/CentOS 7 updated xz to 5.2.2 but they wanted to keep +# the exported symbols compatible with 5.1.2alpha. After checking +# the ABI changes it turned out that >= 5.2.0 ABI is backward +# compatible with the threaded encoder functions from 5.1.2alpha +# (but not vice versa as fixes and extensions to these functions +# were made between 5.1.2alpha and 5.2.0). +# +# In RHEL/CentOS 7, XZ Utils 5.2.2 was patched with +# xz-5.2.2-compat-libs.patch to modify liblzma.map: +# +# - XZ_5.1.2alpha was added with lzma_stream_encoder_mt and +# lzma_stream_encoder_mt_memusage. This matched XZ Utils 5.1.2alpha. +# +# - XZ_5.2 was replaced with XZ_5.2.2. It is clear that this was +# an error; the intention was to keep using XZ_5.2 (XZ_5.2.2 +# has never been used in XZ Utils). So XZ_5.2.2 lists all +# symbols that were listed under XZ_5.2 before the patch. +# lzma_stream_encoder_mt and _mt_memusage are included too so +# they are listed both here and under XZ_5.1.2alpha. +# +# The patch didn't add any __asm__(".symver ...") lines to the .c +# files. Thus the resulting liblzma.so exports the threaded encoder +# functions under XZ_5.1.2alpha only. Listing the two functions +# also under XZ_5.2.2 in liblzma.map has no effect without +# matching .symver lines. +# +# The lack of XZ_5.2 in RHEL/CentOS 7 means that binaries linked +# against unpatched XZ Utils 5.2.x won't run on RHEL/CentOS 7. +# This is unfortunate but this alone isn't too bad as the problem +# is contained within RHEL/CentOS 7 and doesn't affect users +# of other distributions. It could also be fixed internally in +# RHEL/CentOS 7. +# +# The second problem is more serious: In XZ Utils 5.2.2 the API +# headers don't have #ifdef LZMA_UNSTABLE for obvious reasons. +# This is true in RHEL/CentOS 7 version too. Thus now programs +# using new APIs can be compiled without an extra #define. However, +# the programs end up depending on symbol version XZ_5.1.2alpha +# (and possibly also XZ_5.2.2) instead of XZ_5.2 as they would +# with an unpatched XZ Utils 5.2.2. This means that such binaries +# won't run on other distributions shipping XZ Utils >= 5.2.0 as +# they don't provide XZ_5.1.2alpha or XZ_5.2.2; they only provide +# XZ_5.2 (and XZ_5.0). (This includes RHEL/CentOS 8 as the patch +# luckily isn't included there anymore with XZ Utils 5.2.4.) +# +# Binaries built by RHEL/CentOS 7 users get distributed and then +# people wonder why they don't run on some other distribution. +# Seems that people have found out about the patch and been copying +# it to some build scripts, seemingly curing the symptoms but +# actually spreading the illness further and outside RHEL/CentOS 7. +# Adding compatibility in an official XZ Utils release should work +# as a vaccine against this ill patch and stop it from spreading. +# The vaccine is kept GNU/Linux-only as other OSes should be immune +# (hopefully it hasn't spread via some build script to other OSes). +# +# Author: Lasse Collin +# +############################################################################### + +LC_ALL=C +export LC_ALL + +STATUS=0 + +cd "$(dirname "$0")" + +# Get the list of symbols that aren't defined in liblzma_generic.map. +SYMS=$(sed -n 's/^extern LZMA_API([^)]*) \([a-z0-9_]*\)(.*$/\1;/p' \ + api/lzma/*.h \ + | sort \ + | grep -Fve "$(sed '/[{}:*]/d;/^$/d;s/^ //' liblzma_generic.map)") + +# Check that there are no old alpha or beta versions listed. +VER=$(cd ../.. && sh build-aux/version.sh) +NAMES= +case $VER in + *alpha | *beta) + NAMES=$(sed -n 's/^.*XZ_\([^ ]*\)\(alpha\|beta\) .*$/\1\2/p' \ + liblzma_generic.map | grep -Fv "$VER") + ;; +esac + +# Check for duplicate lines. It can catch missing dependencies. +DUPS=$(sort liblzma_generic.map | sed '/^$/d;/^global:$/d' | uniq -d) + +# Check that liblzma_linux.map is in sync with liblzma_generic.map. +# The RHEL/CentOS 7 compatibility symbols are in a fixed location +# so it makes it easy to remove them for comparison with liblzma_generic.map. +# +# NOTE: Putting XZ_5.2 before the compatibility symbols XZ_5.1.2alpha +# and XZ_5.2.2 in liblzma_linux.map is important: If liblzma_linux.map is +# incorrectly used without #define HAVE_SYMBOL_VERSIONS_LINUX, only the first +# occurrence of each function name will be used from liblzma_linux.map; +# the rest are ignored by the linker. Thus having XZ_5.2 before the +# compatibility symbols means that @@XZ_5.2 will be used for the symbols +# listed under XZ_5.2 {...} and the same function names later in +# the file under XZ_5.1.2alpha {...} and XZ_5.2.2 {...} will be +# ignored (@XZ_5.1.2alpha or @XZ_5.2.2 won't be added at all when +# the #define HAVE_SYMBOL_VERSIONS_LINUX isn't used). +IN_SYNC= +if ! sed '111,125d' liblzma_linux.map \ + | cmp -s - liblzma_generic.map; then + IN_SYNC=no +fi + +# Print error messages if needed. +if test -n "$SYMS$NAMES$DUPS$IN_SYNC"; then + echo + echo 'validate_map.sh found problems from liblzma_*.map:' + echo + + if test -n "$SYMS"; then + echo 'liblzma_generic.map lacks the following symbols:' + echo "$SYMS" + echo + fi + + if test -n "$NAMES"; then + echo 'Obsolete alpha or beta version names:' + echo "$NAMES" + echo + fi + + if test -n "$DUPS"; then + echo 'Duplicate lines:' + echo "$DUPS" + echo + fi + + if test -n "$IN_SYNC"; then + echo "liblzma_generic.map and liblzma_linux.map aren't in sync" + echo + fi + + STATUS=1 +fi + +# Exit status is 1 if problems were found, 0 otherwise. +exit "$STATUS" diff --git a/windows/config.h b/windows/config.h new file mode 100644 index 0000000000..81ddf6b7f6 --- /dev/null +++ b/windows/config.h @@ -0,0 +1,67 @@ +/* Configuration extracted from CMake'd project files. + +This is used by CPython, and is not part of the regular xz release. +*/ + +#define HAVE_CHECK_CRC32 1 +#define HAVE_CHECK_CRC64 1 +#define HAVE_CHECK_SHA256 1 + +#define HAVE_DECODERS 1 +#define HAVE_DECODER_ARM 1 +#define HAVE_DECODER_ARM64 1 +#define HAVE_DECODER_ARMTHUMB 1 +#define HAVE_DECODER_DELTA 1 +#define HAVE_DECODER_IA64 1 +#define HAVE_DECODER_POWERPC 1 +#define HAVE_DECODER_LZMA1 1 +#define HAVE_DECODER_LZMA2 1 +#define HAVE_DECODER_SPARC 1 +#define HAVE_DECODER_X86 1 + +#define HAVE_ENCODERS 1 +#define HAVE_ENCODER_ARM 1 +#define HAVE_ENCODER_ARM64 1 +#define HAVE_ENCODER_ARMTHUMB 1 +#define HAVE_ENCODER_DELTA 1 +#define HAVE_ENCODER_IA64 1 +#define HAVE_ENCODER_POWERPC 1 +#define HAVE_ENCODER_LZMA1 1 +#define HAVE_ENCODER_LZMA2 1 +#define HAVE_ENCODER_SPARC 1 +#define HAVE_ENCODER_X86 1 + +#if defined(_M_ARM64) + +#undef HAVE_IMMINTRIN_H +#undef HAVE_USABLE_CLMUL + +#else + +#define HAVE_IMMINTRIN_H 1 +#define HAVE_USABLE_CLMUL 1 +#define HAVE__MM_MOVEMASK_EPI8 1 +#define TUKLIB_FAST_UNALIGNED_ACCESS 1 + +#endif + +#define HAVE___BUILTIN_ASSUME_ALIGNED 1 +#define HAVE__BOOL 1 + +#define HAVE_INTTYPES_H 1 +#define HAVE_MF_BT2 1 +#define HAVE_MF_BT3 1 +#define HAVE_MF_BT4 1 +#define HAVE_MF_HC3 1 +#define HAVE_MF_HC4 1 +#define HAVE_STDBOOL_H 1 +#define HAVE_STDINT_H 1 +#define HAVE_VISIBILITY 0 + +#define MYTHREAD_VISTA 1 + +#define PACKAGE_BUGREPORT "xz@tukaani.org" +#define PACKAGE_NAME "XZ Utils" +#define PACKAGE_URL "https://tukaani.org/xz/" + +#define TUKLIB_SYMBOL_PREFIX lzma_