TSAN data race in _PyErr_Restore #131401

kumaraditya303 · 2025-03-18T10:59:51Z

When running asyncio tests with forever mode, the following race is detected:

0:04:28 load avg: 13.03 [121/1] test_asyncio.test_tasks worker non-zero exit code (Exit code 66) -- running (3): test_asyncio.test_events (1 min 4 sec), test_asyncio.test_sendfile (32.5 sec), test_asyncio.test_tasks (1 min 28 sec)
==================
WARNING: ThreadSanitizer: data race (pid=4663)
  Read of size 8 at 0x7f754ba233f8 by main thread:
    #0 PyException_GetTraceback Objects/exceptions.c:524 (python+0x1800f4) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #1 PyTraceBack_Here Python/traceback.c:276 (python+0x473168) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #2 _PyEval_EvalFrameDefault Python/generated_cases.c.h:12134 (python+0x87bcf) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #3 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #4 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #5 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #6 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #7 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #8 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #9 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #10 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #11 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #12 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #13 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #14 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #15 _PyObject_VectorcallDictTstate Objects/call.c:135 (python+0x151d9e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #16 _PyObject_Call_Prepend Objects/call.c:504 (python+0x1520b3) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #17 slot_tp_call Objects/typeobject.c:10061 (python+0x27af43) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #18 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #19 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #20 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #21 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #22 _PyEval_EvalFrameDefault Python/generated_cases.c.h:3837 (python+0x8bba6) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #23 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #24 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #25 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #26 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #27 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #28 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #29 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #30 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #31 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #32 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #33 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #34 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #35 _PyObject_VectorcallDictTstate Objects/call.c:135 (python+0x151d9e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #36 _PyObject_Call_Prepend Objects/call.c:504 (python+0x1520b3) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #37 slot_tp_call Objects/typeobject.c:10061 (python+0x27af43) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #38 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #39 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #40 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #41 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #42 _PyEval_EvalFrameDefault Python/generated_cases.c.h:3837 (python+0x8bba6) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #43 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #44 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #45 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #46 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #47 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #48 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #49 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #50 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #51 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #52 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #53 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #54 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #55 _PyObject_VectorcallDictTstate Objects/call.c:135 (python+0x151d9e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #56 _PyObject_Call_Prepend Objects/call.c:504 (python+0x1520b3) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #57 slot_tp_call Objects/typeobject.c:10061 (python+0x27af43) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #58 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #59 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #60 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #61 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #62 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1375 (python+0x8bfa8) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #63 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x381d04) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #64 _PyEval_Vector Python/ceval.c:1903 (python+0x381d04)
    #65 PyEval_EvalCode Python/ceval.c:831 (python+0x381d04)
    #66 builtin_exec_impl Python/bltinmodule.c:1156 (python+0x36f006) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #67 builtin_exec Python/clinic/bltinmodule.c.h:560 (python+0x36f006)
    #68 cfunction_vectorcall_FASTCALL_KEYWORDS Objects/methodobject.c:452 (python+0x203bd9) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #69 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x14f921) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #70 PyObject_Vectorcall Objects/call.c:327 (python+0x14f921)
    #71 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1375 (python+0x8bfa8) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #72 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #73 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #74 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #75 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #76 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #77 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #78 pymain_run_module Modules/main.c:337 (python+0x48ee2c) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #79 pymain_run_python Modules/main.c:673 (python+0x48fb8c) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #80 Py_RunMain Modules/main.c:760 (python+0x491042) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #81 pymain_main Modules/main.c:790 (python+0x491042)
    #82 Py_BytesMain Modules/main.c:814 (python+0x491042)
    #83 main Programs/python.c:15 (python+0x85ce2) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)

  Previous write of size 8 at 0x7f754ba233f8 by thread T6:
    #0 _PyErr_Restore Python/errors.c:103 (python+0x3c0b47) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #1 _PyErr_SetObject Python/errors.c:248 (python+0x3be492) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #2 do_raise Python/ceval.c:2069 (python+0x377a04) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #3 _PyEval_EvalFrameDefault Python/generated_cases.c.h:10137 (python+0x99acb) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #4 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #5 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #6 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #7 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x1557f8) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #8 method_vectorcall Objects/classobject.c:72 (python+0x1557f8)
    #9 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #10 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #11 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #12 thread_run Modules/_threadmodule.c:352 (python+0x537935) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #13 pythread_wrapper Python/thread_pthread.h:242 (python+0x46f499) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)

  Thread T6 'asyncio_0' (tid=7712, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 do_start_joinable_thread Python/thread_pthread.h:289 (python+0x46f92e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #2 PyThread_start_joinable_thread Python/thread_pthread.h:313 (python+0x46f92e)
    #3 ThreadHandle_start Modules/_threadmodule.c:437 (python+0x538cb8) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #4 do_start_new_thread Modules/_threadmodule.c:1822 (python+0x538cb8)
    #5 thread_PyThread_start_joinable_thread Modules/_threadmodule.c:1945 (python+0x5395d1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #6 cfunction_call Objects/methodobject.c:551 (python+0x203dd6) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #7 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #8 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #9 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #10 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #11 _PyEval_EvalFrameDefault Python/generated_cases.c.h:3012 (python+0x8d3aa) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #12 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #13 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #14 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #15 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #16 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #17 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #18 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #19 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #20 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #21 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #22 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #23 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #24 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #25 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #26 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #27 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #28 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #29 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #30 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #31 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #32 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #33 _PyObject_VectorcallDictTstate Objects/call.c:135 (python+0x151d9e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #34 _PyObject_Call_Prepend Objects/call.c:504 (python+0x1520b3) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #35 slot_tp_call Objects/typeobject.c:10061 (python+0x27af43) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #36 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #37 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #38 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #39 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #40 _PyEval_EvalFrameDefault Python/generated_cases.c.h:3837 (python+0x8bba6) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #41 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #42 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #43 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #44 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #45 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #46 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #47 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #48 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #49 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #50 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #51 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #52 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #53 _PyObject_VectorcallDictTstate Objects/call.c:135 (python+0x151d9e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #54 _PyObject_Call_Prepend Objects/call.c:504 (python+0x1520b3) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #55 slot_tp_call Objects/typeobject.c:10061 (python+0x27af43) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #56 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #57 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #58 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #59 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #60 _PyEval_EvalFrameDefault Python/generated_cases.c.h:3837 (python+0x8bba6) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #61 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #62 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #63 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #64 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x15567d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #65 method_vectorcall Objects/classobject.c:94 (python+0x15567d)
    #66 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #67 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #68 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #69 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2424 (python+0x8f52d) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #70 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #71 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #72 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #73 _PyObject_VectorcallDictTstate Objects/call.c:135 (python+0x151d9e) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #74 _PyObject_Call_Prepend Objects/call.c:504 (python+0x1520b3) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #75 slot_tp_call Objects/typeobject.c:10061 (python+0x27af43) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #76 _PyObject_MakeTpCall Objects/call.c:242 (python+0x14ea92) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #77 _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 (python+0x14fa18) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #78 _PyObject_VectorcallTstate Include/internal/pycore_call.h:153 (python+0x14fa18)
    #79 PyObject_Vectorcall Objects/call.c:327 (python+0x14fa18)
    #80 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1375 (python+0x8bfa8) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #81 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x381d04) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #82 _PyEval_Vector Python/ceval.c:1903 (python+0x381d04)
    #83 PyEval_EvalCode Python/ceval.c:831 (python+0x381d04)
    #84 builtin_exec_impl Python/bltinmodule.c:1156 (python+0x36f006) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #85 builtin_exec Python/clinic/bltinmodule.c.h:560 (python+0x36f006)
    #86 cfunction_vectorcall_FASTCALL_KEYWORDS Objects/methodobject.c:452 (python+0x203bd9) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #87 _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 (python+0x14f921) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #88 PyObject_Vectorcall Objects/call.c:327 (python+0x14f921)
    #89 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1375 (python+0x8bfa8) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #90 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x382348) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #91 _PyEval_Vector Python/ceval.c:1903 (python+0x382348)
    #92 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14fd3b) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #93 _PyVectorcall_Call Objects/call.c:273 (python+0x152137) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #94 _PyObject_Call Objects/call.c:348 (python+0x1526f1) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #95 PyObject_Call Objects/call.c:373 (python+0x1526f1)
    #96 pymain_run_module Modules/main.c:337 (python+0x48ee2c) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #97 pymain_run_python Modules/main.c:673 (python+0x48fb8c) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #98 Py_RunMain Modules/main.c:760 (python+0x491042) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)
    #99 pymain_main Modules/main.c:790 (python+0x491042)
    #100 Py_BytesMain Modules/main.c:814 (python+0x491042)
    #101 main Programs/python.c:15 (python+0x85ce2) (BuildId: 53ed23022782f7933a3af0cc0267268b7d97fe42)

SUMMARY: ThreadSanitizer: data race Objects/exceptions.c:524 in PyException_GetTraceback
==================

The fix here is to set the traceback of the exception while holding the critical section so it should use PyException_SetTraceback API.

Linked PRs

The text was updated successfully, but these errors were encountered:

colesbury · 2025-03-18T13:41:02Z

I think this needs a bit more investigation before reaching a conclusion. What test case is this running? What's the Python code? If possible, it'd be helpful to extract some Python code that reproduces the race.

I'm concerned that some of these reported races are a symptom of some other bug. For example, the recent asyncio bug with _sendfile_fallback was reported as a race in Py_buffer. In that case adding locks to Py_buffer might have avoided the TSAN reported race, but it wouldn't have fixed the actual underlying race condition due to accidentally overwriting a buffer that was currently being read.

kumaraditya303 · 2025-03-18T14:09:30Z

I think this needs a bit more investigation before reaching a conclusion. What test case is this running? What's the Python code? If possible, it'd be helpful to extract some Python code that reproduces the race.

I'll try to find the test case which causes this rare data race.

I think this needs a bit more investigation before reaching a conclusion. What test case is this running? What's the Python code? If possible, it'd be helpful to extract some Python code that reproduces the race.

I agree, but I think this case is different, PyException_GetTraceback acquires critical section before accessing the fields but at some places the traceback is being accessed without acquiring the critical section.

kumaraditya303 · 2025-03-18T14:32:32Z

Minimal reproducer:

from threading import Thread

exc = Exception("foo")

def exc_tb():
    while True:
        exc.__traceback__

def exc_raise():
    raise exc

def exc_catch():
    while True:
        try:
            exc_raise()
        except Exception as e:
            pass


def main():
    t1 = Thread(target=exc_tb)
    t1.start()
    t2 = Thread(target=exc_catch)
    t2.start()
    t1.join()


if __name__ == "__main__":
    main()

colesbury · 2025-03-18T18:24:25Z

Here is what I've seen so far:

The data race occurs in test_run_coroutine_threadsafe_task_factory_exception:

cpython/Lib/test/test_asyncio/test_tasks.py

Lines 3586 to 3612 in f819900

    
               def test_run_coroutine_threadsafe_task_factory_exception(self): 
        
                   """Test coroutine submission from a thread to an event loop 
        
                   when the task factory raise an exception.""" 
        
                   def task_factory(loop, coro): 
        
                       raise NameError 
        
                   run = self.loop.run_in_executor( 
        
                       None, lambda: self.target(advance_coro=True)) 
        
                   # Set exception handler 
        
                   callback = test_utils.MockCallback() 
        
                   self.loop.set_exception_handler(callback) 
        
                   # Set corrupted task factory 
        
                   self.addCleanup(self.loop.set_task_factory, 
        
                                   self.loop.get_task_factory()) 
        
                   self.loop.set_task_factory(task_factory) 
        
                   # Run event loop 
        
                   with self.assertRaises(NameError) as exc_context: 
        
                       self.loop.run_until_complete(run) 
        
                   # Check exceptions 
        
                   self.assertEqual(len(callback.call_args_list), 1) 
        
                   (loop, context), kwargs = callback.call_args 
        
                   self.assertEqual(context['exception'], exc_context.exception)

The NameError is first raised in main thread in task_factory (see above)

Python Stack Trace

Traceback (most recent call first):
  File "/raid/sgross/cpython/Lib/test/test_asyncio/test_tasks.py", line 3597, in task_factory
    raise SamsError
  File "/raid/sgross/cpython/Lib/asyncio/base_events.py", line 468, in create_task
    return self._task_factory(self, coro, **kwargs)
  File "/raid/sgross/cpython/Lib/asyncio/tasks.py", line 738, in ensure_future
    return loop.create_task(coro_or_future)
  File "/raid/sgross/cpython/Lib/asyncio/tasks.py", line 1002, in callback
    futures._chain_future(ensure_future(coro, loop=loop), future)
  <built-in method run of _contextvars.Context object at remote 0x7fffb80c0370>
  File "/raid/sgross/cpython/Lib/asyncio/events.py", line 98, in _run
    self._context.run(self._callback, *self._args)
  File "/raid/sgross/cpython/Lib/asyncio/events.py", line 142, in _run
    return super()._run()
  File "/raid/sgross/cpython/Lib/asyncio/base_events.py", line 2037, in _run_once
    handle._run()
  File "/raid/sgross/cpython/Lib/asyncio/base_events.py", line 677, in run_forever
    self._run_once()
  File "/raid/sgross/cpython/Lib/asyncio/base_events.py", line 706, in run_until_complete
    self.run_forever()
  File "/raid/sgross/cpython/Lib/test/test_asyncio/test_tasks.py", line 3613, in test_run_coroutine_threadsafe_task_factory_exception
    self.loop.run_until_complete(run)
  File "/raid/sgross/cpython/Lib/unittest/case.py", line 606, in _callTestMethod
    result = method()
  File "/raid/sgross/cpython/Lib/unittest/case.py", line 660, in run
    self._callTestMethod(testMethod)
  File "/raid/sgross/cpython/Lib/unittest/case.py", line 716, in __call__
    return self.run(*args, **kwds)
  File "/raid/sgross/cpython/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/raid/sgross/cpython/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/raid/sgross/cpython/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/raid/sgross/cpython/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/raid/sgross/cpython/Lib/unittest/runner.py", line 259, in run
    test(result)
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 84, in _run_suite
    result = runner.run(suite)
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 210, in _runtest_env_changed_exc
    _load_run_test(result, runtests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 319, in _runtest
    _runtest_env_changed_exc(result, runtests,
  File "/raid/sgross/cpython/Lib/test/libregrtest/single.py", line 348, in run_single_test
    _runtest(result, runtests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/main.py", line 378, in run_test
    result = run_single_test(test_name, runtests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/main.py", line 408, in run_tests_sequentially
    result = self.run_test(test_name, runtests, tracer)
  File "/raid/sgross/cpython/Lib/test/libregrtest/main.py", line 550, in _run_tests
    self.run_tests_sequentially(runtests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/main.py", line 585, in run_tests
    return self._run_tests(selected, tests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/main.py", line 757, in main
    exitcode = self.run_tests(selected, tests)
  File "/raid/sgross/cpython/Lib/test/libregrtest/main.py", line 765, in main
    Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests)
  File "/raid/sgross/cpython/Lib/test/__main__.py", line 2, in <module>
    main(_add_python_opts=True)
  <built-in method exec of module object at remote 0x7fffb425c720>
  File "/raid/sgross/cpython/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/raid/sgross/cpython/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,

Main thread unwinds to:

cpython/Lib/asyncio/events.py

Lines 96 to 114 in f819900

    
           def _run(self): 
        
               try: 
        
                   self._context.run(self._callback, *self._args) 
        
               except (SystemExit, KeyboardInterrupt): 
        
                   raise 
        
               except BaseException as exc: 
        
                   cb = format_helpers._format_callback_source( 
        
                       self._callback, self._args, 
        
                       debug=self._loop.get_debug()) 
        
                   msg = f'Exception in callback {cb}' 
        
                   context = { 
        
                       'message': msg, 
        
                       'exception': exc, 
        
                       'handle': self, 
        
                   } 
        
                   if self._source_traceback: 
        
                       context['source_traceback'] = self._source_traceback 
        
                   self._loop.call_exception_handler(context) 
        
               self = None  # Needed to break cycles when an exception occurs.

The error is then raised again in the executor thread in Future.__get_result:

cpython/Lib/concurrent/futures/_base.py

Lines 392 to 400 in f819900

    
           def __get_result(self): 
        
               if self._exception: 
        
                   try: 
        
                       raise self._exception 
        
                   finally: 
        
                       # Break a reference cycle with the exception in self._exception 
        
                       self = None 
        
               else: 
        
                   return self._result

kumaraditya303 added the topic-free-threading label Mar 18, 2025

bedevere-app bot mentioned this issue Mar 18, 2025

gh-131401: fix data race in _PyErr_Restore #131406

Merged

picnixz added type-bug An unexpected behavior, bug, or error interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Mar 18, 2025

kumaraditya303 added a commit that referenced this issue Mar 19, 2025

gh-131401: fix data race in _PyErr_Restore (#131406)

74b8751

bedevere-app bot mentioned this issue Mar 19, 2025

gh-131401: fix data races in exception handling #131447

Merged

kumaraditya303 added a commit that referenced this issue Mar 20, 2025

gh-131401: fix data races in exception handling (#131447)

b12af0a

colesbury pushed a commit to colesbury/cpython that referenced this issue Mar 20, 2025

pythongh-131401: fix data race in _PyErr_Restore (python#131406)

33ac5cf

kumaraditya303 closed this as completed Mar 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TSAN data race in _PyErr_Restore #131401

TSAN data race in _PyErr_Restore #131401

kumaraditya303 commented Mar 18, 2025 •

edited by bedevere-app bot

Loading

colesbury commented Mar 18, 2025

kumaraditya303 commented Mar 18, 2025

kumaraditya303 commented Mar 18, 2025 •

edited

Loading

colesbury commented Mar 18, 2025

TSAN data race in _PyErr_Restore #131401

TSAN data race in _PyErr_Restore #131401

Comments

kumaraditya303 commented Mar 18, 2025 • edited by bedevere-app bot Loading

Linked PRs

colesbury commented Mar 18, 2025

kumaraditya303 commented Mar 18, 2025

kumaraditya303 commented Mar 18, 2025 • edited Loading

colesbury commented Mar 18, 2025

kumaraditya303 commented Mar 18, 2025 •

edited by bedevere-app bot

Loading

kumaraditya303 commented Mar 18, 2025 •

edited

Loading