Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-126491: GC: Mark objects reachable from roots before doing cycle collection #127110

Merged
merged 18 commits into from
Dec 2, 2024

Conversation

markshannon
Copy link
Member

@markshannon markshannon commented Nov 21, 2024

This is an updated version of #126502

It differs as follows:

  • The removal of lazy dict untracking has already been merged, so isn't present
  • More care is taken to ensure the consistency of the spaces
  • Some adjustments to the "work to do" calculation

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 698abb3 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 21, 2024
@markshannon
Copy link
Member Author

!buildbot Android

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 6d8a0d4 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

  • AMD64 Android PR
  • aarch64 Android PR

@markshannon
Copy link
Member Author

!buildbot Android

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit d9632c6 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

  • AMD64 Android PR
  • aarch64 Android PR

@markshannon
Copy link
Member Author

!buildbot Android

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 0702959 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

  • AMD64 Android PR
  • aarch64 Android PR

@markshannon
Copy link
Member Author

!buildbot iOS

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 0702959 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

  • iOS ARM64 Simulator PR

@markshannon
Copy link
Member Author

FTR:
I needed to increase the threshold from 25k to 27k when in test_gc.test_incremental_gc_handles_fast_cycle_creation for Android and iOS to pass.
To be sure that the GC can keep up, I increased the number of iterations from 20k to 100k. The tests still passed. The number of iterations has been reduced to 20k again, so the test doesn't take too long.

@markshannon
Copy link
Member Author

!buildbot Android
!buildbot iOS

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit eaea41e 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

  • AMD64 Android PR
  • aarch64 Android PR

@markshannon
Copy link
Member Author

!buildbot iOS

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit eaea41e 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

  • iOS ARM64 Simulator PR

@markshannon
Copy link
Member Author

!buildbot iOS

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 79ab26c 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

  • iOS ARM64 Simulator PR

@markshannon
Copy link
Member Author

!buildbot Android

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 79ab26c 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

  • AMD64 Android PR
  • aarch64 Android PR

@markshannon markshannon added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Nov 22, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 79ab26c 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Nov 22, 2024
@markshannon
Copy link
Member Author

Performance shows a ~3% speedup. The results appear noisier than usual, so might be worth repeating, but they are roughly inline with results for earlier versions of this PR.

@@ -329,6 +334,7 @@ struct _gc_runtime_state {
Py_ssize_t work_to_do;
/* Which of the old spaces is the visited space */
int visited_space;
int phase;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this can be rolled into the collecting field (like 0 for not collecting, 1 for MARK, 2 for COLLECT)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, but not very cleanly.
The phase depends on where we are in a full heap scavenge. The collecting flag will flip many times during a scavenge, for incremental collections.

@markshannon
Copy link
Member Author

markshannon commented Dec 2, 2024

All the buildbots that failed were failing on main due to unrelated issues (the Windows 10 bot may have been fixed since)

@markshannon markshannon merged commit a8dd821 into python:main Dec 2, 2024
165 of 176 checks passed
mpage added a commit to mpage/cpython that referenced this pull request Dec 3, 2024
mpage added a commit to mpage/cpython that referenced this pull request Dec 4, 2024
mpage added a commit to mpage/cpython that referenced this pull request Dec 4, 2024
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this pull request Jan 8, 2025
…ycle collection (pythonGH-127110)

* Mark almost all reachable objects before doing collection phase

* Add stats for objects marked

* Visit new frames before each increment

* Update docs

* Clearer calculation of work to do.
@markshannon markshannon deleted the mark-first-2 branch January 10, 2025 16:23
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
…ycle collection (pythonGH-127110)

* Mark almost all reachable objects before doing collection phase

* Add stats for objects marked

* Visit new frames before each increment

* Update docs

* Clearer calculation of work to do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants