rustup is slow

[erikschul]

Running rustup update and rustup target add x86_64-unknown-linux-gnu is very slow for me.

info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)
info: downloading component 'rust-src'
info: downloading component 'cargo'
  1.7 MiB /   7.7 MiB ( 22 %)  40.8 KiB/s in 41s ETA:  2m 29s 

It eventually fails. Sometimes rustup update works and downloads at high speed, e.g. 100Mbit+.
I suspect there's a problem with the mirrors/servers that rustup uses?
If I should take a wild guess, maybe there is DNS roundrobin load balancing, and the DNS response is then cached/pinned to a slow/unresponsive server, until the DNS cache expires.
I worry about downtime in production. There doesn't seem to be any way to choose between mirrors? Are there even any mirrors?

[Noratrieb]

I assume your internet speed is usually faster than 40KiB/s. Has this happened before, and can you reproduce it? Or did the slowness go away? Where in the world are you, roughly?

I created a topic on Zulip where the infra team resides, let's continue the discussion there: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/slow.20rustup.20downloads

[erikschul]

Thanks.
Yes, for other uses, normally gigabit speeds, i.e. 50-100 megabyte/s.
Perhaps it would be useful if rustup would print which mirror it uses, for future debugging.

[takkuumi]

Here is a 1000M bandwidth mirror, which was created by a team from ByteDance. rsproxy

[jdno]

We use two different Content Delivery Networks to serve Rust releases. There's no general problem with the download speeds and we haven't received any other reports over the past few weeks, so my guess is that this might be a peering issue with your personal internet connection and (one of) our CDNs.

You can test if this is the case by explicitly downloading from one of the two CDNs:

RUSTUP_DIST_SERVER=https://cloudfront-static.rust-lang.org rustup update
# or
RUSTUP_DIST_SERVER=https://fastly-static.rust-lang.org rustup update

(Please not that the URLs are implementation details and can change in the future.)

Does one work better than the other for you?

[erikschul]

@jdno It works correctly now, but it's good to have these commands to try with if the issue occurs again. Thanks a lot!

[erikschul]

@jdno FYI, I've had similar issues before with Fastly CDN (according to ip whois), when downloading Kubernetes binaries.
From two separate networks in two different countries, so I'm not sure it's a peering issue.
kubernetes-sigs/downloadkubernetes#508

[jdno]

Thanks for the update!

We have created a tracking issue to get a bit more visibility into the performance of the CDNs. But it's really difficult to debug from our perspective, since there are a lot of variables at play. Most importantly, the connection between the user and the closes point-of-presence (POP), whether the artifact is already in the POP's cache, and how far away the POP is from our S3 bucket if it's not. It's very hard to test this for all the different locations and ISPs in the world. 😞

Glad to hear that it is working now!

[erikschul]

@jdno I can imagine. My only advice would be to observe if ever a download transmits at <100 kB/s. That should be unusual on most modern connections. And with multiple datapoints, it might reveal when and where it happens.

[wmorgue]

@jdno thank you.
From 160 KiB/s to 30 MiB/s

[Abeautifulsnow]

Here is a 1000M bandwidth mirror, which was created by a team from ByteDance. rsproxy

Great! Thank u man. 👍

[erikschul]

@jdno Are the packages signed?
I assume using a third party mirror, such as rsproxy, is unsafe?

[JakkuSakura]

It used to work perfectly, but this happened to me suddenly today. Using the cloudfront server solves the issue.

[erikschul]

@jdno Does rustup verify signatures? If not, using a third party mirror like rsproxy could be dangerous.
(And even if it does validate signatures, a vulnerability might bypass it, for injecting malware into CICD pipelines)

[eproxus]

Can confirm that using the Cloudfront CDN works, but the default not (Fastly?). I'm in Germany on a Deutsche Telekom DSL connection (I think Deutsche Telekom are somewhat known for bad peering agreements).

[TheShiningBoots]

Yeah cloudfront is much much faster, went from 25.6kb to 38MiB.

[ENCRYPTEDFOREVER]

Using cloudfront CDN solved the issue for me too

[iirekm]

What was it closed? Problem still exists.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y takes about 30 minutes with a good internet connection where download speeds are normally >100 MiB/s. If cloudfront is faster, why won't you make it default?

[erikschul]

@iirekm I've reopened, but the issue is so old by now that I doubt it'll get any attention. And there are currently 10k open issues.

So the answer is probably: prioritization.

There is a workaround, so use it.

Do I hate it? Absolutely. But it's free!