From 4aa440c0b2b39922e112522e45388948a2d92d39 Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Mon, 17 Mar 2025 15:29:14 +0100 Subject: [PATCH 1/3] update documentation on reproducible tarballs for EasyBuild 5.0 --- docs/easybuild-v5/enhancements.md | 47 ++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 10 deletions(-) diff --git a/docs/easybuild-v5/enhancements.md b/docs/easybuild-v5/enhancements.md index b265543401..7c3d245384 100644 --- a/docs/easybuild-v5/enhancements.md +++ b/docs/easybuild-v5/enhancements.md @@ -76,16 +76,43 @@ status codes are reported in the corresponding logs. ## Reproducible tarballs for sources created via `git_config` { : #reproducible-tarballs-git_config } EasyBuild can now generate reproducible tarballs of sources cloned from Git -repositories. This means that those sources using the `git_config` option will -now have consistent contents across different systems and across time, allowing -to reliably validate them with checksums. EasyBuild follows the -[archival guidelines from reproducible-builds.org](https://reproducible-builds.org/docs/archives/) -to generate those reproducible tarballs. - -This new feature does not apply to sources cloned with `keep_git_dir` enabled. -Including the `.git` folder in the sources is inherently time-dependent as it -contains information about the clone action itself, which hinders the creation -of a reproducible tarball. +repositories. This means that easyconfigs with sources using the `git_config` +option can now have consistent contents across different systems and across +time, allowing to reliably validate them with checksums. + +EasyBuild follows the [archival guidelines from reproducible-builds.org](https://reproducible-builds.org/docs/archives/) +to generate reproducible tarballs. The new method to create archives in +EasyBuild 5.0 is fully implemented in Python, which removes our dependency on +external tools such as [GNU Tar](https://www.gnu.org/software/tar/) or file +compressors for this tasks. However, extraction of archives continues to work +by executing external commands on the host system. + +Reproducible tarballs have the following restrictions: + +* Sources cloned with `keep_git_dir` enabled cannot be archived in a +reproducible manner. Including the `.git` folder in the sources is inherently +time-dependent as it contains information about the clone action itself, which +hinders the creation of a reproducible tarball. Hence, EasyBuild 5.0 will +create the archive of sources with `keep_git_dir`, but their checksums cannot +be validated across systems. + +* Reproducible archives are supported in uncompressed TAR format (`.tar`) or +for tarballs compressed with [XZ compression](https://en.wikipedia.org/wiki/XZ_Utils) +(`.tar.zx`). The wide-spread [GZip compression](https://en.wikipedia.org/wiki/Gzip) +is not currently supported because its implementation injects metadata in the +compressed archive that is time dependent. + +* Systems running EasyBuild with Python < 3.9 will skip checksum validation for +sources from Git repos. Due to changes in the low-level code of the `tarfile` +module in the Python base distribution, tarballs generated before version 3.9 +result in archives with different contents than those generated in Python 3.9+. + +Easyconfigs found in the repository of EasyBuild that contain sources from Git +repos without `keep_git_dir` have already been updated to use reproducible +tarballs. Archives will be created in `.tar.xz` format and checksums will be +validated on Python 3.9+. Therefore, beware that EasyBuild 5.0 might generate +new archives for sources that were already cloned in your system due to this +changes in format. --- From 2dbb403e7cdbdc2abe310eb04e7fb959c83cc3bd Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Tue, 18 Mar 2025 09:10:23 +0100 Subject: [PATCH 2/3] replace asterisks with dashes in list --- docs/easybuild-v5/enhancements.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/easybuild-v5/enhancements.md b/docs/easybuild-v5/enhancements.md index 7c3d245384..ae1522a7bd 100644 --- a/docs/easybuild-v5/enhancements.md +++ b/docs/easybuild-v5/enhancements.md @@ -89,20 +89,20 @@ by executing external commands on the host system. Reproducible tarballs have the following restrictions: -* Sources cloned with `keep_git_dir` enabled cannot be archived in a +- Sources cloned with `keep_git_dir` enabled cannot be archived in a reproducible manner. Including the `.git` folder in the sources is inherently time-dependent as it contains information about the clone action itself, which hinders the creation of a reproducible tarball. Hence, EasyBuild 5.0 will create the archive of sources with `keep_git_dir`, but their checksums cannot be validated across systems. -* Reproducible archives are supported in uncompressed TAR format (`.tar`) or +- Reproducible archives are supported in uncompressed TAR format (`.tar`) or for tarballs compressed with [XZ compression](https://en.wikipedia.org/wiki/XZ_Utils) (`.tar.zx`). The wide-spread [GZip compression](https://en.wikipedia.org/wiki/Gzip) is not currently supported because its implementation injects metadata in the compressed archive that is time dependent. -* Systems running EasyBuild with Python < 3.9 will skip checksum validation for +- Systems running EasyBuild with Python < 3.9 will skip checksum validation for sources from Git repos. Due to changes in the low-level code of the `tarfile` module in the Python base distribution, tarballs generated before version 3.9 result in archives with different contents than those generated in Python 3.9+. From 2a3531be6f7e82f3067d191496922bcaf5f15d4d Mon Sep 17 00:00:00 2001 From: Simon Branford <4967+branfosj@users.noreply.github.com> Date: Tue, 18 Mar 2025 15:45:05 +0000 Subject: [PATCH 3/3] Update docs/easybuild-v5/enhancements.md --- docs/easybuild-v5/enhancements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/easybuild-v5/enhancements.md b/docs/easybuild-v5/enhancements.md index ae1522a7bd..64d56052a4 100644 --- a/docs/easybuild-v5/enhancements.md +++ b/docs/easybuild-v5/enhancements.md @@ -84,7 +84,7 @@ EasyBuild follows the [archival guidelines from reproducible-builds.org](https:/ to generate reproducible tarballs. The new method to create archives in EasyBuild 5.0 is fully implemented in Python, which removes our dependency on external tools such as [GNU Tar](https://www.gnu.org/software/tar/) or file -compressors for this tasks. However, extraction of archives continues to work +compressors for this task. However, extraction of archives continues to work by executing external commands on the host system. Reproducible tarballs have the following restrictions: