Makefile always regenerates build-details.json
#131556
Comments
colesbury
added a commit
to colesbury/cpython
that referenced
this issue
Mar 21, 2025
Use the contents of `pybuilddir.txt` as a prefix for the build-details.json path, if it exists.
FFY00
added a commit
to FFY00/cpython
that referenced
this issue
Mar 26, 2025
Right now, this value is being calculated during the build process when calling `python -m sysconfig --generate-posix-vars`. This command, ammong other things, calculates the PYBUILDDIR value and writes it to a pybuilddir.txt, making it available to the Makefile this way. This is problematic because it makes it impossible to write Makefile rules with targets under PYBUILDDIR — the target and prerequisites of rule are always expanded immediatily as the Makefile is read, only the rule recipe is deferred [1], meaning that all targets must be known before any rule is executed. Since PYBUILDDIR is only known in the middle of the build process — after the target list has been built — we cannot write rules for files under it. We have had to worked around this limitation in a couple ways: - Extension modules, which need to be present in PYBUILDDIR to execute the interpreter in-tree, are built in the source tree, and afterwards symlinked to PYBUILDDIR once its value is known. - Instead of the sysconfigdata module and the sysconfig vars JSON file, instead of having theirn own target, are generated by the pybuilddir.txt target. Additionally, this limitation is also prone to cause issues such as pythonGH-131556. That said, on top of the Makefile issues, PYBUILDDIR being calculated by sysconfig also creates its own additional complications, necessitating more workarounds and introducing unecessary complexity to the sysconfig data generation code — there's a chicken-and-egg problem in certain systems, where we need to know PYBUILDDIR to install the sysconfigdata module, but the sysconfigdata module is necessary to be avaible to calculate the value PYBUILDDIR [2]. We currently handle this by manually building a module object for sysconfigdata and inject it to sys.modules. By defining PYBUILDDIR directly in Makefile.pre.in, we solve all these problems. The current build system design is the result of a sucession small fixes adapting the original sysconfig code from over 15 years ago to work with the rest of codebase as it evolved. That is to say — I don't think there's any technical resoning behind this particular design, I believe it is simply the result of technical debt. Therefore, I don't see any reason why not to move the PYBUILDDIR definition over to Makefile.pre.in, which to me seems to make more sense [1] https://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html [2] https://github.com/python/cpython/blob/898e6b395e63ad7f8bbe421adf0af8947d429925/Lib/sysconfig/__main__.py#L206-L221 Signed-off-by: Filipe Laíns <lains@riseup.net>
FFY00
added a commit
to FFY00/cpython
that referenced
this issue
Mar 26, 2025
Right now, this value is being calculated during the build process when calling `python -m sysconfig --generate-posix-vars`. This command, ammong other things, calculates the PYBUILDDIR value and writes it to a pybuilddir.txt, making it available to the Makefile this way. This is problematic because it makes it impossible to write Makefile rules with targets under PYBUILDDIR — the target and prerequisites of rule are always expanded immediatily as the Makefile is read, only the rule recipe is deferred [1], meaning that all targets must be known before any rule is executed. Since PYBUILDDIR is only known in the middle of the build process — after the target list has been built — we cannot write rules for files under it. We have had to worked around this limitation in a couple ways: - Extension modules, which need to be present in PYBUILDDIR to execute the interpreter in-tree, are built in the source tree, and afterwards symlinked to PYBUILDDIR once its value is known. - Instead of the sysconfigdata module and the sysconfig vars JSON file, instead of having theirn own target, are generated by the pybuilddir.txt target. Additionally, this limitation is also prone to cause issues such as pythonGH-131556. That said, on top of the Makefile issues, PYBUILDDIR being calculated by sysconfig also creates its own additional complications, necessitating more workarounds and introducing unecessary complexity to the sysconfig data generation code — there's a chicken-and-egg problem in certain systems, where we need to know PYBUILDDIR to install the sysconfigdata module, but the sysconfigdata module is necessary to be avaible to calculate the value PYBUILDDIR [2]. We currently handle this by manually building a module object for sysconfigdata and inject it to sys.modules. By defining PYBUILDDIR directly in Makefile.pre.in, we solve all these problems. The current build system design is the result of a sucession small fixes adapting the original sysconfig code from over 15 years ago to work with the rest of codebase as it evolved. That is to say — I don't think there's any technical resoning behind this particular design, I believe it is simply the result of technical debt. Therefore, I don't see any reason why not to move the PYBUILDDIR definition over to Makefile.pre.in, which to me seems to make more sense [1] https://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html [2] https://github.com/python/cpython/blob/898e6b395e63ad7f8bbe421adf0af8947d429925/Lib/sysconfig/__main__.py#L206-L221 Signed-off-by: Filipe Laíns <lains@riseup.net>
FFY00
added a commit
to FFY00/cpython
that referenced
this issue
Mar 26, 2025
Right now, this value is being calculated during the build process when calling `python -m sysconfig --generate-posix-vars`. This command, ammong other things, calculates the PYBUILDDIR value and writes it to a pybuilddir.txt, making it available to the Makefile this way. This is problematic because it makes it impossible to write Makefile rules with targets under PYBUILDDIR — the target and prerequisites of rule are always expanded immediatily as the Makefile is read, only the rule recipe is deferred [1], meaning that all targets must be known before any rule is executed. Since PYBUILDDIR is only known in the middle of the build process — after the target list has been built — we cannot write rules for files under it. We have had to worked around this limitation in a couple ways: - Extension modules, which need to be present in PYBUILDDIR to execute the interpreter in-tree, are built in the source tree, and afterwards symlinked to PYBUILDDIR once its value is known. - Instead of the sysconfigdata module and the sysconfig vars JSON file, instead of having theirn own target, are generated by the pybuilddir.txt target. Additionally, this limitation is also prone to cause issues such as pythonGH-131556. That said, on top of the Makefile issues, PYBUILDDIR being calculated by sysconfig also creates its own additional complications, necessitating more workarounds and introducing unecessary complexity to the sysconfig data generation code — there's a chicken-and-egg problem in certain systems, where we need to know PYBUILDDIR to install the sysconfigdata module, but the sysconfigdata module is necessary to be avaible to calculate the value PYBUILDDIR [2]. We currently handle this by manually building a module object for sysconfigdata and inject it to sys.modules. By defining PYBUILDDIR directly in Makefile.pre.in, we solve all these problems. The current build system design is the result of a sucession small fixes adapting the original sysconfig code from over 15 years ago to work with the rest of codebase as it evolved. That is to say — I don't think there's any technical resoning behind this particular design, I believe it is simply the result of technical debt. Therefore, I don't see any reason why not to move the PYBUILDDIR definition over to Makefile.pre.in, which to me seems to make more sense [1] https://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html [2] https://github.com/python/cpython/blob/898e6b395e63ad7f8bbe421adf0af8947d429925/Lib/sysconfig/__main__.py#L206-L221 Signed-off-by: Filipe Laíns <lains@riseup.net>
FFY00
added a commit
to FFY00/cpython
that referenced
this issue
Mar 26, 2025
Right now, this value is being calculated during the build process when calling `python -m sysconfig --generate-posix-vars`. This command, ammong other things, calculates the PYBUILDDIR value and writes it to a pybuilddir.txt, making it available to the Makefile this way. This is problematic because it makes it impossible to write Makefile rules with targets under PYBUILDDIR — the target and prerequisites of rule are always expanded immediatily as the Makefile is read, only the rule recipe is deferred [1], meaning that all targets must be known before any rule is executed. Since PYBUILDDIR is only known in the middle of the build process — after the target list has been built — we cannot write rules for files under it. We have had to worked around this limitation in a couple ways: - Extension modules, which need to be present in PYBUILDDIR to execute the interpreter in-tree, are built in the source tree, and afterwards symlinked to PYBUILDDIR once its value is known. - Instead of the sysconfigdata module and the sysconfig vars JSON file, instead of having theirn own target, are generated by the pybuilddir.txt target. Additionally, this limitation is also prone to cause issues such as pythonGH-131556. That said, on top of the Makefile issues, PYBUILDDIR being calculated by sysconfig also creates its own additional complications, necessitating more workarounds and introducing unecessary complexity to the sysconfig data generation code — there's a chicken-and-egg problem in certain systems, where we need to know PYBUILDDIR to install the sysconfigdata module, but the sysconfigdata module is necessary to be avaible to calculate the value PYBUILDDIR [2]. We currently handle this by manually building a module object for sysconfigdata and inject it to sys.modules. By defining PYBUILDDIR directly in Makefile.pre.in, we solve all these problems. The current build system design is the result of a sucession small fixes adapting the original sysconfig code from over 15 years ago to work with the rest of codebase as it evolved. That is to say — I don't think there's any technical resoning behind this particular design, I believe it is simply the result of technical debt. Therefore, I don't see any reason why not to move the PYBUILDDIR definition over to Makefile.pre.in, which to me seems to make more sense [1] https://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html [2] https://github.com/python/cpython/blob/898e6b395e63ad7f8bbe421adf0af8947d429925/Lib/sysconfig/__main__.py#L206-L221 Signed-off-by: Filipe Laíns <lains@riseup.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Makefile target that generates
build-details.jsonhas an incorrect target so it's re-run for every action. The generated file is in:For example, on my machine that evaluates to:
cpython/Makefile.pre.in
Lines 939 to 940 in 6131707
Linked PRs
build-details.jsonMakefile target #131558The text was updated successfully, but these errors were encountered: