Allow sanity check to pass without locking the cache #17434
Conversation
sbc100
force-pushed
the
sanity_no_lock
branch
2 times, most recently
from
a846583 to
052096d
Compare
| # Essentially the env. var EM_EXCLUSIVE_CACHE_ACCESS signals from parent to | ||
| # child process that the child can reuse the lock that the parent already has | ||
| # acquired. | ||
| EM_EXCLUSIVE_CACHE_ACCESS = int(os.environ.get('EM_EXCLUSIVE_CACHE_ACCESS', '0')) |
There was a problem hiding this comment.
Its not needed anymore, which I think is a nice feature of this change.
Basically, compile-only sub-processed should no longer require getting the cache lock, since they are not writing anything into the cache. Prior to this change compiler-subprocess could not be run while holding the lock since they would always try to acquire it during the sanity check.
Without this the new test I wrote will just always pass since the test runner is a parent process of the compiler processes it run.
There was a problem hiding this comment.
I don't follow. emcc -c -sUSE_SDL2 will fetch SDL2 and write it into the cache, even though this is a compile-only command - we need to fetch SDL2 to get its header in order to compile with them?
There was a problem hiding this comment.
I put some logic back in to catch any times a child does try to lock the cache when a parent holds the lock already... which should never happen, but if it does this will give a clear error.
There was a problem hiding this comment.
I don't follow.
emcc -c -sUSE_SDL2will fetch SDL2 and write it into the cache, even though this is a compile-only command - we need to fetch SDL2 to get its header in order to compile with them?
I that case the parent process is the only one that writes to the cache. The parent triggers N different compile jobs, then it run emar to combine them, but neither the compile or the emar need to lock the cache. Only the parent would lock the cache in this case.
Allowing the child process to modify the cache while the parent is hold the lock was always a little strange.
There was a problem hiding this comment.
In the past all the children would pointlessly/needlessly lock the cache just the run the sanity check.
sbc100
force-pushed
the
sanity_no_lock
branch
3 times, most recently
from
8702c8f to
5470ef3
Compare
sbc100
force-pushed
the
sanity_no_lock
branch
6 times, most recently
from
9782ac4 to
b8d9732
Compare
We check the sanity file once, without holding the lock, and if that fails we acquire the lock and check again. This means that if some other process is performing some long running cache operation (e.g building libc) it won't block all emscripten processes from running (e.g. compile processes won't be blocked by link process).
We check the sanity file once, without holding the lock, and if that
fails we acquire the lock and check again.
This means that if some other process is performing some long running
cache operation (e.g building libc) it won't block all emscripten
processes from running (e.g. compile processes won't be blocked by link
process).