gh-158948: Generate `Modules/getbuildinfo.h` by vstinner · Pull Request #158951 · python/cpython

vstinner · 2026-10-07T02:56:45Z

Add Tools/build/generate_getbuildinfo.py to generate a new Module/getbuildinfo.h header file which contains build information strings. It replaces C code which generates these strings at runtime.

Merge Python/getcompiler.c, Python/getcopyright.c, Python/getplatform.c and Python/getversion.c into Modules/getbuildinfo.c.

Truncate long Clang version string to just keep "Clang x.y.z" or "Apple Clang x.y.z".

COMPILER and _Py_COMPILER macros no longer contain surrounding square brackets.

Add _getcompiler project to the Visual Studio solution.

Issue: Rework Modules/getbuildinfo.c to build strings at build time, instead of runtime #158948

Merge Python/getcompiler.c, Python/getcopyright.c, Python/getplatform.c and Python/getversion.c into Modules/getbuildinfo.c. Add Tools/build/generate_getbuildinfo.py to auto-generate a new Module/getbuildinfo.h header file which contains build information strings.

vstinner · 2026-10-07T02:57:51Z

This PR is a draft. I didn't investigate Windows support yet. I didn't try cross-compilation, it may break.

vstinner · 2026-10-07T03:02:10Z

This change may fix the regression described by @itamaro: #158608 (comment).

vstinner · 2026-10-07T03:24:22Z

Tools/build/generate_getbuildinfo.py uses the SOURCE_DATE_EPOCH environment variable for reproducible build: https://reproducible-builds.org/specs/source-date-epoch/. If it's set, use it for the date and time.

vstinner · 2026-10-07T03:30:33Z

Ok, I fixed the out-of-tree build (used by the Ubuntu CI for example).

It's only needed in the Python script. Python.h defines PY_VERSION (via patchlevel.h).

vstinner · 2026-10-07T03:36:19Z

Example of auto-generated Modules/getbuildinfo.h header file:

// Header file auto-generated by Tools/build/generate_getbuildinfo.py

#define DATE "Oct  7 2026"
#define TIME "05:35:34"
#define COMPILER "[GCC 16.2.1 20260819 (Red Hat 16.2.1-2)]"

#define PLATFORM "linux"

#define GITBRANCH "getbuildinfo"
#define GITTAG "heads/getbuildinfo"
#define GITVERSION "22d1a6b7c1e"
#define GIT_IDENTIFIER "heads/getbuildinfo"

#define BUILDINFO "heads/getbuildinfo:22d1a6b7c1e, Oct  7 2026, 05:35:34"
#define GET_VERSION "3.16.0a0 (heads/getbuildinfo:22d1a6b7c1e, Oct  7 2026, 05:35:34) [GCC 16.2.1 20260819 (Red Hat 16.2.1-2)]"

vstinner · 2026-10-07T03:46:22Z

Ah, the "WASI / build and test" job does cross-compile! Running the Programs/_getcompiler program to get the compiler version fails with Exec format error.

/home/runner/work/cpython/cpython/cross-build/aarch64-unknown-linux-gnu/python ../../Tools/build/generate_getbuildinfo.py Modules/getbuildinfo.h \
		"wasi" \
		`LC_ALL=C git --git-dir ../../.git rev-parse --short HEAD` \
		`LC_ALL=C git --git-dir ../../.git describe --all --always --dirty` \
		`LC_ALL=C git --git-dir ../../.git name-rev --name-only HEAD` \
		"`Programs/_getcompiler`"
(...)
/bin/sh: 1: Programs/_getcompiler: Exec format error

Later, running test.pythinfo fails on collect_platform() because sys.version lacks the compiler:

ERROR: collect_platform() failed
Traceback (most recent call last):
  File "/Lib/test/pythoninfo.py", line 1418, in collect_info
    collect_func(info_add)
    ~~~~~~~~~~~~^^^^^^^^^^
  File "/Lib/test/pythoninfo.py", line 185, in collect_platform
    platform.python_implementation())
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Lib/platform.py", line 1199, in python_implementation
    return _sys_version()[0]
           ~~~~~~~~~~~~^^
  File "/Lib/platform.py", line 1159, in _sys_version
    raise ValueError(
        'failed to parse CPython sys.version: %s' %
        repr(sys_version))
ValueError: failed to parse CPython sys.version: '3.16.0a0 (remotes/pull/158951/merge-dirty:00e222b, Oct  7 2026, 03:37:58) '

Fix also dependencies.

vstinner · 2026-10-07T19:06:04Z

With the help of @zware, I managed to update the PCbuild project, to build Programs_getcompiler.c and generate Modules\getbuildinfo.h on Windows.

vstinner · 2026-10-07T19:15:03Z

Aha, test_freeze_simple_script() of test_tools fails on Ubuntu / build and test (ubuntu-26.04-arm) with:


LC_ALL=C sed -e 's,\$(\([A-Za-z0-9_]*\)),\$\{\1\},g' < Misc/python-config.sh >python-config
python3.14 /tmp/test_python_0l4z8_yx/tmpt900bnty/cpython/Tools/build/generate_getbuildinfo.py Modules/getbuildinfo.h \
		"linux" \
		`LC_ALL=C ` \
		`LC_ALL=C ` \
		`LC_ALL=C ` \
		"`Programs/_getcompiler || echo COMPILER`" \
		use_pyconfig
usage: generate_getbuildinfo.py OUTPUT PLATFORM GITVERSION GITTAG GITBRANCH COMPILER GIL_ENABLED
make[1]: Leaving directory '/tmp/test_python_0l4z8_yx/tmpt900bnty/python-build'

vstinner · 2026-10-07T19:26:49Z

Aha, test_freeze_simple_script() of test_tools fails on Ubuntu / build and test (ubuntu-26.04-arm)

It's just missing quotes: I fixed it.

* Use argparse for command line parsing. * Run _getcompiler if --compiler option is omitted. * Rename getbuildinfo.h variables.

vstinner · 2026-10-08T02:50:10Z

See also #144124 which adds UNVERSIONED_COMPILER environment variable to override the compiler string. Using this change, it will be easier to override variables such as the compiler, but if we add this feature, I would prefer to do it in a separated change.

vstinner · 2026-10-08T03:04:12Z

See also #157368 where the Clang version was too long (Clang 22.1.3 (https://github.com/llvm/llvm-project e9846648fd6183ee6d8cbdb4502213fcf902a211)) on Windows, and has been replaced with "Clang x.y.z" (PR #157369).

Add the script name as prefix to all logged messages.

bedevere-bot · 2026-10-08T03:40:24Z

🤖 New build scheduled with the buildbot fleet by @vstinner for commit ebaadbc 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F158951%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

Adjust also getbuildinfo.h indentation

bedevere-bot · 2026-10-08T04:21:07Z

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 8431d21 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F158951%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

vstinner · 2026-10-08T04:47:55Z

Logs of truncated Clang versions on buildbots:

aarch64 Android (build Python): Truncate compiler string Clang 16.0.0 (clang-1600.0.26.6) to Clang 16.0.0
AMD64 Android (host Python):Truncate compiler string Android (13691557, +pgo, +bolt, +lto, +mlgo, based on r522817d) clang version 18.0.4 (https://android.googlesource.com/toolchain/llvm-project d8003a456d14a3deb8054cdaa529ffbf02d9b262) to Android Clang 18.0.4
AMD64 FreeBSD Refleaks: Truncate compiler string Clang 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) to Clang 19.1.7
wasm32-wasi: Truncate compiler string Clang 23.1.0-wasi-sdk (https://github.com/llvm/llvm-project 895aa2c896ada719451be2e3673c83da8ddf1141) to Clang 23.1.0
iOS ARM64 Simulator (build Python): Truncate compiler string Clang 16.0.0 (clang-1600.0.26.6) to Clang 16.0.0
iOS ARM64 Simulator (host Python): Truncate compiler string Apple clang version 16.0.0 (clang-1600.0.26.6) to Apple Clang 16.0.0
WASM Emscripten: Truncate compiler string Clang 24.0.0git (https:/github.com/llvm/llvm-project a06d9165905ce89b5ffef2bbb84c886d60a9b8bf) to Clang 24.0.0git
aarch64 Fedora Stable Clang: Truncate compiler string Clang 21.1.8 (Fedora 21.1.8-6.fc43) to Clang 21.1.8
x86-64 macOS: Truncate compiler string Clang 14.0.0 (clang-1400.0.29.202) to Clang 14.0.0

Note: Clang 23.1.0-wasi-sdk truncated to Clang 23.1.0 is a bug: the -wasi-sdk suffix should be kept. I will fix it in my next change.

zooba · 2026-10-08T13:42:46Z

Windows also does cross-compilation, and presumably you're running the generator using whichever host Python is available, which means there's 0% guarantee it'll be the same as the target.

I'd also be very disappointed if generating the build info invalidates the cleanness of version control - if everything is committed, it should show as clean, and overwriting Modules/getbuildinfo.h will break that (equally, having the file be absent/ignored until created is annoying for editors/IDEs, as is putting it outside of the source tree).

Is there a real reason to rework this into a script rather than just avoiding the fixed-length arrays? Between some external commands and the C preprocessor, this should be straightforward.

vstinner · 2026-10-08T15:57:06Z

@zooba:

Windows also does cross-compilation, and presumably you're running the generator using whichever host Python is available, which means there's 0% guarantee it'll be the same as the target.

Tools/build/generate_getbuildinfo.py doesn't get any information from the Python used to run the script. It only collects data from its command line arguments, Makefile, pyconfig.h, $(srcdir)/Include/patchlevel.h, the _getcompiler program, and SOURCE_DATE_EPOCH environment variable.

On Windows, the _getcompiler program is run by the Visual studio project in PCbuild/_freeze_module.vcxproj.

I didn't think about cross-compilation on Windows. On Unix, running _getprogram can fail on cross-compilation: in this case, the C compiler is retrieve from the CC variable of Makefile and $CC --version is run. I suppose that we can do the same on Windows, run <compiler> --version (or something like that). But I'm not sure of the retrieve the C compiler? Any advice is welcomed :-)

I'd also be very disappointed if generating the build info invalidates the cleanness of version control - if everything is committed, it should show as clean, and overwriting Modules/getbuildinfo.h will break that (equally, having the file be absent/ignored until created is annoying for editors/IDEs, as is putting it outside of the source tree).

Modules/getbuildinfo.h is generated at each build. I'm not sure when it should be updated. I have to check how it works currently. So far, I spent a lot of time to just make the build work in all platforms supported by Python (GitHub Actions and now buildbots) :-)

I didn't get your remark on version control: Modules/getbuildinfo.h is not tracked by Git on purpose.

Is there a real reason to rework this into a script rather than just avoiding the fixed-length arrays? Between some external commands and the C preprocessor, this should be straightforward.

The Clang version string is normalized (truncated).
Add support for SOURCE_DATE_EPOCH environment variable for reproducible build.
Py_GetBuildInfo() and Py_GetVersion() no longer have to format the string at the first call: it's now computed statically during Python build.
Later, it will be easiler to modify the code to allow overridding the compiler string for reproducible build: see PR gh-144121: Allow overriding COMPILER with UNVERSIONED_COMPILER #144124.

To me, it seems cleaner to have the Tools/build/generate_getbuildinfo.py script to process, cleanup, and validate all build information.

vstinner · 2026-10-08T16:29:36Z

I created https://discuss.python.org/t/generate-modules-getbuildinfo-h-when-building-python-to-get-build-information-statically/109401 to discuss this build change.

vstinner · 2026-10-08T16:48:45Z

I wasn't sure if PYTHON_FOR_BUILD is actually needed to build Python on Unix. I tried PYTHON_FOR_BUILD=false and the build fails on running false ./Tools/build/generate-build-details.py none/build-details.json. Ah yes, we do run a Python script PYTHON_FOR_BUILD to build Python! So my change doesn't introduce a new dependency on Python.

On Windows, the FindPythonForBuild target downloads Python if Python is not already installed. It is especially needed by "regen" targets.

zooba · 2026-10-08T18:21:48Z

It is especially needed by "regen" targets

Which are optional, and if the generated files are present, it shouldn't run.

(From one of the buildbots):

py -3.14 Tools/build/generate_getbuildinfo.py --platform=win32 --git-version="00401067d0" --git-tag="heads/refs/pull/158951/merge" --git-branch="refs/pull/158951/merge" --compiler="MSC v.1951 32 bit (Intel)" --free-threading=1

This command line bothers me, there seems to be some very easy ways to inject extra commands here if you control the branch name or compiler name (and as we know, some compiler names can be a bit silly). I'd rather see the values passed as environment variables to avoid any risk

I haven't dug through the change yet to figure it out, but it's worth being aware that our release builds don't always run the two PGO builds on the same machine. We do the instrumented build first, maybe start a new machine to run the profile, and then start another new machine to merge the profile. This involves copying $(IntDir) from the first machine and download it on the third one, so if the generated file is in that directory it should be fine, but if it lives somewhere else then we may have to generate it again later. It shouldn't matter (output should be identical), but it'd be nice to know ahead of time whether it's going to reuse the file or not.

chris-eibl · 2026-10-08T18:59:53Z

See also #157368 where the Clang version was too long (Clang 22.1.3 (https://github.com/llvm/llvm-project e9846648fd6183ee6d8cbdb4502213fcf902a211)) on Windows, and has been replaced with "Clang x.y.z" (PR #157369).

The Windows clang build bot now shows (https://buildbot.python.org/?#/builders/2269/builds/34/steps/3/logs/stdio) ¹

sys.version: 3.16.0a0 (heads/refs/pull/158951/merge:00401067d0, Oct  8 2026, 00:02:24) [Clang 22.1.3]

vs https://buildbot.python.org/#/builders/2269/builds/17/steps/3/logs/stdio

sys.version: 3.16.0a0 (heads/refs/pull/157934/merge:a83720779a, Oct  6 2026, 10:19:52) [Clang 22.1.3 64 bit (AMD64) with MSC v.1951 CRT]

Quote from #157368

A lot of code depends on being able to parse the now missing 64 bit (AMD64) part (#145410, #154200, #154199, ...)

which the associated PRs #154210 and #154553 "masked away" on main (and hence the buildbot stayed green). 3rd party code relying on being able to parse the 64 bit (AMD64) part will now be broken again?

Also, I'd miss the MSC CRT version ...

confirmed on a local build as well, and the build log shows generate_getbuildinfo.py: Truncate compiler string 'Clang 22.1.3 64 bit (AMD64) with MSC v.1951 CRT' to 'Clang 22.1.3' ↩

chris-eibl · 2026-10-08T19:11:33Z

I also share @zooba's concern regarding cross-compiling on Windows in case of Arm: ¹
Windows / Build and test (arm64, switch-case) CI now shows (https://github.com/python/cpython/actions/runs/37726403262/job/113145665440?pr=158951)

sys.version: 3.16.0a0 (remotes/pull/158951/merge:0040106, Oct  7 2026, 21:14:56) [MSC v.1951 32 bit (Intel)]

vs https://github.com/python/cpython/actions/runs/37815471280/job/113442970056

sys.version: 3.16.0a0 (remotes/pull/158969/merge:ab6938f, Oct  8 2026, 10:21:06) [MSC v.1951 64 bit (ARM64)]

We do this definititely for the official releases. And currently also in CI, since we do not use the native Arm compiler tool chain, see https://github.com/python/cpython/issues/158244 for details ↩

bedevere-app Bot mentioned this pull request Oct 7, 2026

Rework Modules/getbuildinfo.c to build strings at build time, instead of runtime #158948

Open

vstinner mentioned this pull request Oct 7, 2026

gh-158445: Allocate memory in the heap in Py_GetVersion() #158608

Merged

Fix and reforamt generate_getbuildinfo.py

508e3ca

vstinner added the skip news label Oct 7, 2026

vstinner added 3 commits October 7, 2026 05:17

Look for pyconfig.h in the current working directory

8e7fe03

Fix out-of-tree build

06cc348

Short comment

7c10d0b

Update PCbuild/_freeze_module.vcxproj

8c2dcce

vstinner added 2 commits October 7, 2026 05:32

Update Tools/c-analyzer/cpython/ignored.tsv

7882331

Don't redefine PY_VERSION

22d1a6b

It's only needed in the Python script. Python.h defines PY_VERSION (via patchlevel.h).

Remove useless getbuildinfo.h dependencies

61ddc42

vstinner added 3 commits October 7, 2026 19:51

Add _getcompiler project on Windows

1ae6b29

Temporary workaround for cross-compilation

cd00cdb

Update PCbuild/pcbuild.proj

9a56153

Fix also dependencies.

Fix build when there is not .git/ directory (test_tools)

eab1b0e

vstinner added 5 commits October 7, 2026 23:39

Enhance generate_getbuildinfo.py

a0780d4

* Use argparse for command line parsing. * Run _getcompiler if --compiler option is omitted. * Rename getbuildinfo.h variables.

Run directly the compiler (--version) if _getcompiler failed

0c8ad79

Reuse sysconfig parsers

0d50e2e

Simplify _getcompiler build command

98519e1

Fix typo

05a9779

vstinner marked this pull request as ready for review October 8, 2026 02:46

vstinner requested review from a team, AA-Turner, FFY00, ZeroIntensity, emmatyping, ericsnowcurrently, erlend-aasland and itamaro as code owners October 8, 2026 02:47

bedevere-app Bot added the awaiting core review label Oct 8, 2026

vstinner mentioned this pull request Oct 8, 2026

gh-144121: Allow overriding COMPILER with UNVERSIONED_COMPILER #144124

Open

Log the truncated compiler string

ebaadbc

Add the script name as prefix to all logged messages.

vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 8, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 8, 2026

vstinner added 2 commits October 8, 2026 05:58

Support old sysconfig versions

2c9bafb

Fix Makefile (_getcompiler) for BSD make

8431d21

Adjust also getbuildinfo.h indentation

vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 8, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 8, 2026

Conversation

vstinner commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 7, 2026

Uh oh!

vstinner commented Oct 8, 2026

Uh oh!

vstinner commented Oct 8, 2026

Uh oh!

bedevere-bot commented Oct 8, 2026

Uh oh!

bedevere-bot commented Oct 8, 2026

Uh oh!

vstinner commented Oct 8, 2026

Uh oh!

zooba commented Oct 8, 2026

Uh oh!

vstinner commented Oct 8, 2026

Uh oh!

vstinner commented Oct 8, 2026

Uh oh!

vstinner commented Oct 8, 2026

Uh oh!

zooba commented Oct 8, 2026

Uh oh!

chris-eibl commented Oct 8, 2026

Footnotes

Uh oh!

chris-eibl commented Oct 8, 2026

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vstinner commented Oct 7, 2026 •

edited

Loading

vstinner commented Oct 7, 2026 •

edited

Loading

vstinner commented Oct 7, 2026 •

edited

Loading