Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental compilation on Windows is slow because of hard links #137560

Open
kryptan opened this issue Feb 24, 2025 · 5 comments
Open

Incremental compilation on Windows is slow because of hard links #137560

kryptan opened this issue Feb 24, 2025 · 5 comments
Labels
A-incr-comp Area: Incremental compilation I-compiletime Issue: Problems and improvements with respect to compile times. O-windows Operating system: Windows T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@kryptan
Copy link
Contributor

kryptan commented Feb 24, 2025 •
edited
Loading

Every time I recompile my Rust project after a change it takes 2 or 3 seconds. I was wondering where that time is spent. I used the measureme/summarize tool to profile the rust compiler and it showed me that most time is spent in the codegen_copy_artifacts_from_incr_cache stage:

+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| Item                                                   | Self time | % of total time | Time     | Item count | Incremental load time | Incremental result hashing time |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| codegen_copy_artifacts_from_incr_cache                 | 12.24s    | 83.333          | 12.24s   | 255        | 0.00ns                | 0.00ns                          |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| run_linker                                             | 448.51ms  | 3.054           | 448.51ms | 1          | 0.00ns                | 0.00ns                          |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| incr_comp_prepare_session_directory                    | 416.99ms  | 2.840           | 416.99ms | 1          | 0.00ns                | 0.00ns                          |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| LLVM_passes                                            | 414.15ms  | 2.820           | 414.15ms | 1          | 0.00ns                | 0.00ns                          |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| copy_all_cgu_workproducts_to_incr_comp_cache_dir       | 349.22ms  | 2.378           | 349.22ms | 1          | 0.00ns                | 0.00ns                          |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
| codegen_crate                                          | 264.25ms  | 1.800           | 375.35ms | 1          | 0.00ns                | 0.00ns                          |
+--------------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+---------------------------------+
...

The reported time is larger than the actual time, probably because of multithreading.

This stage involves creating hard links.

Process Monitor shows that there are many SetLinkInformationFile syscalls some of which take 100-200ms each:

Image

I have observed this behaviour on many different projects.

Steps to reproduce in any Rust project of non-trivial size are:

  1. cargo build
  2. Make any small change, like changing a string literal.
  3. cargo rustc -- -Z self-profile
  4. summarize summarize <filename>.mm_profdata

Looks like creating hard links is sometimes very slow on Windows. Maybe some other strategy can be used to avoid creating so many hard links?

rustc 1.87.0-nightly (f8a913b13 2025-02-23)
binary: rustc
commit-hash: f8a913b1381e90379c7ca63ac2b88b9518936628
commit-date: 2025-02-23
host: x86_64-pc-windows-msvc
release: 1.87.0-nightly
LLVM version: 20.1.0
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Feb 24, 2025
@ehuss
Copy link
Contributor

ehuss commented Feb 24, 2025

Thanks for the report! I think this is essentially a duplicate of #64291 (though more data and attention is good!).

@lolbinarycat lolbinarycat added O-windows Operating system: Windows I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-incr-comp Area: Incremental compilation labels Feb 24, 2025
@the8472
Copy link
Member

the8472 commented Feb 24, 2025

Could we try symlinking instead of hardlinking (which should at least work for people with developer mode enabled) or do those files have lifecyles where neither outlives the other?

@saethlin
Copy link
Member

Oh I have a mitigation for this already. Not a complete fix, but: #128320.

That entire region of code needs a lot of refactoring attention to make it easier to understand. I've basically avoided touching it because it is so hard to understand. The irony.

@saethlin saethlin mentioned this issue Feb 24, 2025
Merged
@frederikhors

This comment has been minimized.

Sign in to view
@saethlin
Copy link
Member

Can you help me with this?

No. The issue tracker is not a support forum. You are just reporting a duplicate.

@saethlin saethlin removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Feb 27, 2025
bors added a commit to rust-lang-ci/rust that referenced this issue Mar 21, 2025
@bors
Auto merge of rust-lang#128320 - saethlin:link-me-maybe, r=compiler-e…
48b36c9 
…rrors

Avoid no-op unlink+link dances in incr comp

Incremental compilation scales quite poorly with the number of CGUs. This PR improves one reason for that.

The incr comp process hard-links all the files from an old session into a new one, then it runs the backend, which may just hard-link the new session files into the output directory. Then codegen hard-links all the output files back to the new session directory.

This PR (perhaps unimaginatively) fixes the silliness that ensues in the last step. The old `link_or_copy` implementation would be passed pairs of paths which are already the same inode, then it would blindly delete the destination and re-create the hard-link that it just deleted. This PR lets us skip both those operations. We don't skip the other two hard-links.

`cargo +stage1 b && touch crates/core/main.rs && strace -cfw -elink,linkat,unlink,unlinkat cargo +stage1 b` before and then after on `ripgrep-13.0.0`:
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 52.56    0.024950          25       978       485 unlink
 34.38    0.016318          22       727           linkat
 13.06    0.006200          24       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.047467          24      1954       485 total
```
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 42.83    0.014521          57       252           unlink
 38.41    0.013021          26       486           linkat
 18.77    0.006362          25       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.033904          34       987           total
```

This reduces the number of hard-links that are causing perf troubles, noted in rust-lang#64291 and rust-lang#137560
bjorn3 pushed a commit to rust-lang/rustc_codegen_cranelift that referenced this issue Mar 23, 2025
@bors
Auto merge of #128320 - saethlin:link-me-maybe, r=compiler-errors
1986ad7 
Avoid no-op unlink+link dances in incr comp

Incremental compilation scales quite poorly with the number of CGUs. This PR improves one reason for that.

The incr comp process hard-links all the files from an old session into a new one, then it runs the backend, which may just hard-link the new session files into the output directory. Then codegen hard-links all the output files back to the new session directory.

This PR (perhaps unimaginatively) fixes the silliness that ensues in the last step. The old `link_or_copy` implementation would be passed pairs of paths which are already the same inode, then it would blindly delete the destination and re-create the hard-link that it just deleted. This PR lets us skip both those operations. We don't skip the other two hard-links.

`cargo +stage1 b && touch crates/core/main.rs && strace -cfw -elink,linkat,unlink,unlinkat cargo +stage1 b` before and then after on `ripgrep-13.0.0`:
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 52.56    0.024950          25       978       485 unlink
 34.38    0.016318          22       727           linkat
 13.06    0.006200          24       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.047467          24      1954       485 total
```
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 42.83    0.014521          57       252           unlink
 38.41    0.013021          26       486           linkat
 18.77    0.006362          25       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.033904          34       987           total
```

This reduces the number of hard-links that are causing perf troubles, noted in rust-lang/rust#64291 and rust-lang/rust#137560
github-actions bot pushed a commit to rust-lang/rustc-dev-guide that referenced this issue Mar 24, 2025
@bors
Auto merge of #128320 - saethlin:link-me-maybe, r=compiler-errors
e9b51bd 
Avoid no-op unlink+link dances in incr comp

Incremental compilation scales quite poorly with the number of CGUs. This PR improves one reason for that.

The incr comp process hard-links all the files from an old session into a new one, then it runs the backend, which may just hard-link the new session files into the output directory. Then codegen hard-links all the output files back to the new session directory.

This PR (perhaps unimaginatively) fixes the silliness that ensues in the last step. The old `link_or_copy` implementation would be passed pairs of paths which are already the same inode, then it would blindly delete the destination and re-create the hard-link that it just deleted. This PR lets us skip both those operations. We don't skip the other two hard-links.

`cargo +stage1 b && touch crates/core/main.rs && strace -cfw -elink,linkat,unlink,unlinkat cargo +stage1 b` before and then after on `ripgrep-13.0.0`:
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 52.56    0.024950          25       978       485 unlink
 34.38    0.016318          22       727           linkat
 13.06    0.006200          24       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.047467          24      1954       485 total
```
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 42.83    0.014521          57       252           unlink
 38.41    0.013021          26       486           linkat
 18.77    0.006362          25       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.033904          34       987           total
```

This reduces the number of hard-links that are causing perf troubles, noted in rust-lang/rust#64291 and rust-lang/rust#137560
lnicola pushed a commit to lnicola/rust-analyzer that referenced this issue Mar 24, 2025
@bors
Auto merge of #128320 - saethlin:link-me-maybe, r=compiler-errors
abd31ef 
Avoid no-op unlink+link dances in incr comp

Incremental compilation scales quite poorly with the number of CGUs. This PR improves one reason for that.

The incr comp process hard-links all the files from an old session into a new one, then it runs the backend, which may just hard-link the new session files into the output directory. Then codegen hard-links all the output files back to the new session directory.

This PR (perhaps unimaginatively) fixes the silliness that ensues in the last step. The old `link_or_copy` implementation would be passed pairs of paths which are already the same inode, then it would blindly delete the destination and re-create the hard-link that it just deleted. This PR lets us skip both those operations. We don't skip the other two hard-links.

`cargo +stage1 b && touch crates/core/main.rs && strace -cfw -elink,linkat,unlink,unlinkat cargo +stage1 b` before and then after on `ripgrep-13.0.0`:
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 52.56    0.024950          25       978       485 unlink
 34.38    0.016318          22       727           linkat
 13.06    0.006200          24       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.047467          24      1954       485 total
```
```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 42.83    0.014521          57       252           unlink
 38.41    0.013021          26       486           linkat
 18.77    0.006362          25       249           unlinkat
------ ----------- ----------- --------- --------- ----------------
100.00    0.033904          34       987           total
```

This reduces the number of hard-links that are causing perf troubles, noted in rust-lang/rust#64291 and rust-lang/rust#137560
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-incr-comp Area: Incremental compilation I-compiletime Issue: Problems and improvements with respect to compile times. O-windows Operating system: Windows T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ehuss @the8472 @kryptan @saethlin @lolbinarycat @frederikhors @rustbot