Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV with generators in free-threaded build #120321

Open
rostan-t opened this issue Jun 10, 2024 · 6 comments
Open

SIGSEGV with generators in free-threaded build #120321

rostan-t opened this issue Jun 10, 2024 · 6 comments
Labels
topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@rostan-t
Copy link

rostan-t commented Jun 10, 2024 •
edited by Fidget-Spinner
Loading

Crash report

What happened?

Trying to iterate on a generator from multiple threads under the free-threaded build results in a segmentation fault.

Here is a minimal repro:

import concurrent.futures


def gen():
    while True:
        yield


it = gen()
with concurrent.futures.ThreadPoolExecutor() as executor:
    while True:
        _ = executor.submit(lambda: next(it))

The issue seems to be specific to generators as other kinds of iterators work well in parallel.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a0 experimental free-threading build (heads/main:c3b6dbff2c, Jun 10 2024, 16:54:16) [GCC 11.4.0]

Linked PRs
@rostan-t rostan-t added the type-crash A hard crash of the interpreter, possibly with a core dump label Jun 10, 2024
@Fidget-Spinner
Copy link
Member

Fidget-Spinner commented Jun 10, 2024 •
edited
Loading

Thanks for the repro. Looking into this.

@Fidget-Spinner Fidget-Spinner self-assigned this Jun 10, 2024
@Fidget-Spinner
Copy link
Member

Confirmed, with pydebug and tsan.

@Zheaoli
Copy link
Contributor

Zheaoli commented Jun 10, 2024

May I take a chance to dive into this issue?

@Fidget-Spinner
Copy link
Member

@Zheaoli sorry I assigned myself on this one I have a fix already and will put it up shortly. Do you have any specific issues in mind you would like to work on? I could help mentor/guide you on stuff you're interested in if I can help!

@bedevere-app bedevere-app bot mentioned this issue Jun 10, 2024
Closed
@Zheaoli
Copy link
Contributor

Zheaoli commented Jun 10, 2024 •
edited
Loading

sorry I assigned myself on this one I have a fix already and will put it up shortly

No problem!

Do you have any specific issues in mind you would like to work on? I could help mentor/guide you on stuff you're interested in if I can help!

For now, I'm a fresh man try to be more familiar with the CPython codebase.

I'm now working on #104280 to make dtrace more functional to help debug the process. I'm also interested in JIT and Tier2(especially to enhance the observerability for the Python process). But I'm still trying to find the issue I can fix and the thing I can do(lol

Thanks for knowing that you will be willing to mentor/guide me lol!

@Fidget-Spinner
Copy link
Member

@Zheaoli please send me an email I'll help give some pointers privately

@bedevere-app bedevere-app bot mentioned this issue Jun 14, 2024
Merged
@vstinner vstinner mentioned this issue Nov 6, 2024
Open
@Fidget-Spinner Fidget-Spinner removed their assignment Nov 15, 2024
colesbury added a commit to colesbury/cpython that referenced this issue Mar 13, 2025
@colesbury
pythongh-120321: Use atomics for PyGenObject.gi_frame_state
b67dc73 
In the free threading build, use atomics to change the generator
s state. Generators can only be executed by one thread at a time and
it's an error to try to iterate over or send a value into a currently
executing generator.

The `gen_send_ex2` function needs to determine if the
`_PyEval_EvalFrame()` call resulted in the generator suspended or
returned. We can't rely on `gi_frame_state` because if the generator
suspended, a different thread could have (in theory) immediately started
executing it. Instead, route the resulting frame state through a new
`gi_out_frame_state` pointer.
colesbury added a commit to colesbury/cpython that referenced this issue Mar 27, 2025
@colesbury
pythongh-120321: Use atomics for PyGenObject.gi_frame_state
a6e0872 
In the free threading build, use atomics to change the generator
s state. Generators can only be executed by one thread at a time and
it's an error to try to iterate over or send a value into a currently
executing generator.

The `gen_send_ex2` function needs to determine if the
`_PyEval_EvalFrame()` call resulted in the generator suspended or
returned. We can't rely on `gi_frame_state` because if the generator
suspended, a different thread could have (in theory) immediately started
executing it. Instead, route the resulting frame state through a new
`gi_out_frame_state` pointer.
colesbury added a commit to colesbury/cpython that referenced this issue Mar 27, 2025
@colesbury
pythongh-120321: Use atomics for PyGenObject.gi_frame_state
6f50a92 
In the free threading build, use atomics to change the generator
s state. Generators can only be executed by one thread at a time and
it's an error to try to iterate over or send a value into a currently
executing generator.

The `gen_send_ex2` function needs to determine if the
`_PyEval_EvalFrame()` call resulted in the generator suspended or
returned. We can't rely on `gi_frame_state` because if the generator
suspended, a different thread could have (in theory) immediately started
executing it. Instead, route the resulting frame state through a new
`gi_out_frame_state` pointer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Zheaoli @Fidget-Spinner @rostan-t