Convert _ctypes extension module to multi-phase init

[neonene]

Feature or enhancement

Proposal:

This issue is available to keep track of PRs, following the heap type conversion completed at #114314.

TODO:

• [ ] Make free_list in malloc_closure.c per-module variables. rejected
• Fix intentional memory leaks of StgInfo during a finalization: comment.
• Enable Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED(DONE): comment.
  FUTURE?: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED after a compatible PyGILState_Ensure() is introduced, see also the links below.

Links to documents:

• libffi: a foreign function interface library
• libffi - GitHub

Links to previous discussion of this feature:

• Isolate Stdlib Extension Modules #103092
• Convert _ctypes static types (and the "meta class") to heap types #114314
• PyGILState_Ensure() for sub-interpreters:
  • [subinterpreters] Make the PyGILState API compatible with subinterpreters #59956
  • [subinterpreters] Make the PyGILState API compatible with multiple interpreters #55124
    • _ctypes patch (2011)

Linked PRs

• gh-117142: ctypes: Unify meta tp slot functions #117143
• gh-117142: Port _ctypes to multi-phase init #117181
• gh-117142: ctypes: Migrate global vars to module state #117189
• gh-117142: ctypes: Clean up globals-to-fix.tsv #117544
• gh-117142: ctypes: Migrate global closure freelist to thunk-type state #117874
• gh-117142: ctypes: Migrate global closure freelist to module state #117875
• gh-117142: ctypes: Fix memory leak of StgInfo #118139
• gh-117142: Slightly hacky fix for memory leak of StgInfo #119424
• [3.13] gh-117142: Slightly hacky fix for memory leak of StgInfo (GH-119424) #119468
• gh-117142: Support Importing ctypes in Isolated Interpreters #119991
• [3.13] gh-117142: Support Importing ctypes in Isolated Interpreters (gh-119991) #120008

[neonene]

Could anyone add this open issue to the base issue's todo (port _ctypes branch) as a checked (finished) leaf? Thanks.

[ericsnowcurrently]

@encukou, this is done now, right?

[encukou]

I'll wait for more green buildbots before merging the 3.13 backport. Then it'll be done :)

[ericsnowcurrently]

Unless I was looking at an old main HEAD, _ctypes is still has py_mod_multiple_interpreters set to Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED. Is there are reason we haven't set it to Py_MOD_PER_INTERPRETER_GIL_SUPPORTED?

CC @neonene

[neonene]

In my understanding of #117181 (comment), the closure_fcn function in callbacks.c needs a new compatible PyGILState_Ensure() faimily like #59956 (comment) to enable Py_MOD_PER_INTERPRETER_GIL_SUPPORTED?

Issue #55124 in 2011 has an experimantal patch: https://bugs.python.org/file20417/gilstateinterp.patch

[encukou]

Also mmap-based closure allocation needs a mutex, and I don't know how thread-safe the libffi API is. There might be other things.

[ericsnowcurrently]

FYI, PyGILState_Ensure() should work fine with subinterpreters, as of 3.12. It was necessary to fix that for per-interpreter GIL.

[ericsnowcurrently]

Regarding mmap and libffi, aren't those resolved for free-threading builds?