Add -Zsplit-metadata flag to reduce disk usage

[Kobzol]

Proposal

I would like to propose an unstable flag for rustc tentatively named -Zsplit-metadata. It would instruct the compiler to avoid storing metadata in rlib and dylib artifacts. Instead, it would only store a small metadata stub, and the full metadata would be kept only in the .rmeta file. When using such a compiled artifact as a dependency, the user would have to provide both the rlib/dylib and a corresponding .rmeta file (passed through --extern), which would contain the full metadata contents.

The main motivation for this flag is to reduce compiled artifact disk usage, and potentially also to improve compilation performance slightly when used with Cargo on systems with slow I/O/FS, by writing less data to disk (in exchange for reading a little bit more data, as the compiler would now have to read .rmeta files when linking the final leaf artifact).

Currently, when Cargo compiles a crate, by default it uses --emit=metadata for intermediate dependencies, which tells them to emit a .rmeta file, in addition to the main .rlib file. This allows Cargo to use pipelined compilation, which improves compilation speed. However, it also means that after the compilation, both the .rmeta and .rlib files are stored on the disk, and the crate metadata stored inside of them is thus duplicated. And since the metadata can be quite large, this can waste non-trivial amounts of disk space.

Preliminary results that show how much size could be saved by using this approach can be found here.

This idea has been floated around since a long time ago (rust-lang/rust#23366, rust-lang/rust#29511, rust-lang/rust#57076).

@bjorn3 has created an initial implementation, and under his guidance I cleaned it up a little bit and created an MVP in rust-lang/rust#137535.

If this gets accepted, the plan for landing this would be approximately this:

1. Land -Zsplit-metadata
2. Teach Cargo to use it (e.g. behind an unstable Cargo flag)
3. Dogfood -Zsplit-metadata for the compiler itself in bootstrap
4. Stabilize the flag

Mentors or Reviewers

@bjorn3, @petrochenkov

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

[rustbot]

Important

This issue is not meant to be used for technical discussion. There is a Zulip stream for that.
Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections can formally be registered here by adding a comment.

@rfcbot concern reason-for-concern
<description of the concern>

Concerns can be lifted with:

@rfcbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[petrochenkov]

@rustbot second