Big static array causes rlib to bloat in size

[nebulark]

A big static array inside a library causes the generated lib file to bloat in size, even if it is zero initialized.
I tried this code:

static BUFFER : [u8; 1 << 29] = unsafe { std::mem::zeroed() };

I would expect that this would not affect the size of library.
Instead this causes the library to include a LOT of 0s, increasing the size in this example to over 1GB.
The final executeable is unaffected, only the .rlib file bloats.

Meta

This happens in stable and nightly

rustc --version --verbose:

rustc 1.86.0-nightly (1e9b0177d 2025-01-24)
binary: rustc
commit-hash: 1e9b0177da38e3f421a3b9b1942f1777d166e06a
commit-date: 2025-01-24
host: x86_64-pc-windows-msvc
release: 1.86.0-nightly
LLVM version: 19.1.7

[workingjubilee]

why would you expect such a static to not be present in the binary, exactly?

[adwinwhite]

Because zeroed statics can be put at .bss section of object file which only records the sizes?

If we change the immutable static above into a mutable static, it gets to be stored in .bss section rather than .rodata.
Though the rmeta file still contains the big array.

[faptc]

What does gcc/clang do with static arrays? Are the arrays placed on .bss or .rodata section?

Edit: It seems both gcc and clang does place the static zeroed arrays on .bss section.
I agree rustc should do the same thing.

[workingjubilee]

@adwinwhite Thank you! That helps narrow down what I'm looking for.

For this C:

 char BUFFER[1 << 29] = {};
 
 int main() {
 
 }

I get this in the compiled object from clang:

         .type   BUFFER,@object                  # @BUFFER
         .bss
         .globl  BUFFER
         .p2align        4, 0x0
 BUFFER:
         .zero   536870912
         .size   BUFFER, 536870912

But with this code diff:

-char BUFFER[1 << 29] = {};
+const char BUFFER[1 << 29] = {};

I see this object diff:

         .type   BUFFER,@object                  # @BUFFER
-        .bss
+        .section        .rodata,"a",@progbits
         .globl  BUFFER
         .p2align        4, 0x0

That will have the matching effect, right?

So this is actually a consistent implementation between clang and rustc, it's just different defaults, right?

[workingjubilee]

We can put such a zeroed array in bss as an optimization, I suppose, but it seems we would be deviating from clang in doing so.

[hanna-kruppe]

I'm not sure if the comparison with assembly/object files produced by Clang is appropriate. The original issue said "the final executable is unaffected" so it seems that the final codegen is already as expected. But rlib files also contain a lot of metadata, not just LLVM IR and/or object files. In particular, it contains the values of constants/statics that may be needed by constant evaluation in dependent crates. I suspect that the large size of the rlib may be caused by this representation of the static, because there's probably no equivalent of .bss in rustc's representation of constant values.

[workingjubilee]

Then I suppose if I check the static mut version the rlib is going to also be huge.

[workingjubilee]

using a run-length encoding type of thing for static arrays with homogenous values seems reasonable, I suppose, but I wonder if it will cause some sort of linking errors...

[adwinwhite]

Then I suppose if I check the static mut version the rlib is going to also be huge.

Yes. The rlib contains a lib.rmeta file and a object file in this case.
I guess the buffer is stored in eval_static_initializer. But I'm not familiar enough with const eval allocation to hack on it.

[workingjubilee]

huh.

$ cargo build
   Compiling const-init-size v0.1.0 (/home/jubilee/rust/const-init-size)
warning: static `BUFFER` is never used
 --> src/lib.rs:1:8
  |
1 | static BUFFER: [u8; 1 << 29] = [0; 1 << 29];
  |        ^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default

ls  Building [                             ] 0/1: const-init-size                                                                                                                                                                                                                                                         
warning: `const-init-size` (lib) generated 1 warning
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.20s
$ ls -lh target/debug/
total 1.1G
drwxr-xr-x 2 jubilee jubilee 4.0K Feb  6 17:26 build
drwxr-xr-x 3 jubilee jubilee 4.0K Feb  6 20:42 deps
drwxr-xr-x 2 jubilee jubilee 4.0K Feb  6 17:26 examples
drwxr-xr-x 6 jubilee jubilee 4.0K Feb  6 20:41 incremental
-rw-r--r-- 1 jubilee jubilee  119 Feb  6 20:41 libconst_init_size.d
-rw-r--r-- 2 jubilee jubilee 1.1G Feb  6 20:42 libconst_init_size.rlib
$ helix .
$ cargo build
   Compiling const-init-size v0.1.0 (/home/jubilee/rust/const-init-size)
warning: static `BUFFER` is never used
 --> src/lib.rs:1:12
  |
1 | static mut BUFFER: [u8; 1 << 29] = [0; 1 << 29];
  |            ^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default

warning: `const-init-size` (lib) generated 1 warning
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.28s
$ ls -lh target/debug/
total 513M
drwxr-xr-x 2 jubilee jubilee 4.0K Feb  6 17:26 build
drwxr-xr-x 3 jubilee jubilee 4.0K Feb  6 20:43 deps
drwxr-xr-x 2 jubilee jubilee 4.0K Feb  6 17:26 examples
drwxr-xr-x 6 jubilee jubilee 4.0K Feb  6 20:41 incremental
-rw-r--r-- 1 jubilee jubilee  119 Feb  6 20:41 libconst_init_size.d
-rw-r--r-- 2 jubilee jubilee 513M Feb  6 20:43 libconst_init_size.rlib

I guess static mut has half the impact...? perhaps due to the .bss versus .rodata difference...?

[adwinwhite]

@workingjubilee indeed. You can ar x libname.rlib to unarchive the rlib file to see the inners.

[nebulark]

Sorry I probably have reduced that example too much. It was also pretty late, when I submitted this.
This is close to the thing I was trying to get to work (without bloating the lib).

struct Buff(std::cell::UnsafeCell<[u8;1 << 29]>);
unsafe  impl Send for Buff { }
unsafe  impl Sync for Buff { }
static BUF : Buff =  unsafe { std::mem::zeroed() };

Essentially this is about being able to "reserve" memory in the data section. One reason for why, is to have memory that is close addresswise to text/data. Useful when hotpatching as it allows to use 32bit relative jumps & refer to existing data symbols from hotpatched functions. Relocations targeting data symbols are generally 32 bit, at least on windows.

[workingjubilee]

I don't think there's any requirement that these things are mapped next to each other.

[workingjubilee]

Nonetheless,

@nebulark I am curious why the size in the rlib matters to you, given that it does not seem to affect your use case?

[nebulark]

For me personally it does not matter that much. It just takes some additional disk space needlessly, which it a very minor annoyance.
I mostly just wanted to submit the bug report so that this issue is known.

[workingjubilee]

Aha, okay! I mostly wanted to make sure I wasn't missing something important.

[saethlin]

because there's probably no equivalent of .bss in rustc's representation of constant values.

That can be fixed!

[saethlin]

Going to perf ideas here: #137152