Degraded performance with ghcr.io

[enescakir]

Select Topic Area

Question

Body

We run a number of self-hosted GitHub Actions runners in Europe. Today we've started seeing degraded performance with the GitHub Container Registry (ghcr.io). Image pulls are extremely slow or failing.

Is anyone else experiencing similar issues? We haven’t been able to find any active incidents

[SaTu07]

I can confirm! Since more than an hour I can see it's very slow.

[enescakir]

docker push to ghcr.io is fast, but docker pull from ghcr.io is extremely slow.

We have observed slow image download when tried to pull ghcr.io image from Hetzner Cloud Falkenstein and AWS eu-central-1 virtual machines, it takes at least 40 mins. When we tried downloading the same image from Hetzner Cloud Hillsboro or AWS east-us-1, it took 1-2 mins to pull.

[enescakir]

It looks like render.com is also affected by the ghcr.io issue.

https://status.render.com/incidents/3qvtpwf108k4

[fdr]

I've been testing this, after reproducing (from Hetzner), I've had three successful fast pulls. I will keep trying until about 5 PST before thinking it's surely over.

[Debaroon2000]

Seeing similar behavior as well, any fix from the team yet?

[fdr]

for what it's worth, the path that was affected for us was hetzner -> decix frankfurt -> fastly, probably in the digital realty campus, given all the relevant parties were present there. No transit provider was in-between.

Here's the MTR trace, redacting some source information. Nothing about the trace particularly abnormal, which is something I'd think would be more likely if some operations, like container push, were fast.

                                                                                                                Packets               Pings
 Host                                                                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
 3. AS24940  2a01:4f8::12:4                                                                                    0.0%    37    0.6   1.3   0.4   9.0   1.9
 4. AS24940  core21.fsn1.hetzner.com                                                                           0.0%    37    0.7   3.8   0.5  26.0   6.8
 5. AS24940  core0.fra.hetzner.com                                                                             0.0%    37    5.2   5.2   4.9   5.6   0.1
 6. AS???    fra2.decixfra.fastly.net                                                                          0.0%    37    5.6   5.6   5.4   6.9   0.2
 7. AS54113  2606:50c0:8001::154                                                                               0.0%    37    5.3   5.5   5.2   7.9   0.4

[enescakir]

GitHub’s support team pointed to an incident at Fastly as the root cause. They mentioned it was resolved at 18:00 UTC, but we continued experiencing issues until 21:00 UTC. Did anyone else notice problems last night or today?

There was a disruption with Fastly yesterday that affected some regions in Europe from 17:08 to 18:00 UTC
https://www.fastlystatus.com/incident/377915

Sometimes a pull from the GHCR issues a redirect to a partner CDN, like Fastly, via https://pkg-containers.githubusercontent.com/. In these instances, the actual download of the bytes would come from Fastly to serve and complete the request.

[philipp1992]

Issue still persists. Any update?

[angelo291196]

I'm facing very slow image pulls from Azure Kubernetes Service clusters in West Europe region

[henkhofs]

Same here, GitHub is having more and more issues lately

[Harishkrishna17]

We’re facing the same issue — our AKS cluster in West Europe is unable to pull images from the GitHub Container Registry (ghcr).

[BjoernRave]

Yes, facing the same issue, hetzner vps in germany using coolify and trying to pull container from ghcr. the download speed is super slow

[taieb-tk]

We have the same issue right now

[fdr]

this might be a new one.

stuff happens, but between the last one and this one, it looks like the github status page management bureaucracy is dysfunctional. A public advisory would be helpful.

[fredriknils]

We have been having the same issue with our AKS cluster in West Europe for the past few days. Some builds are randomly unable to pull images from ghcr.io, getting stuck during download.

[rickardp]

Same issue also exists with a similar AKS (Github ARC) setup in Azure swedencentral, so doesn't look isolated to westeurope only

[ssnielsen]

Starting to see the same exact issue in the same environments as last week. Anyone seeing the same thing? 🤔

[ssnielsen]

Seeing things going back to normal again 🤔

[enescakir]

We’ve started to observe it again.

[taieb-tk]

Same here

[parallelcreativity]

Slow AF

[parallelcreativity]

I have been trying to pull an image to runpod US-KS-1, too slow to operate, same speed on EU-RO-1, around 4-5MB/s. Painful.

[bhornett-phsa]

Seems to be very inconsistent. One day an image pull takes 10 seconds and the next day it's 15 minutes.

[marcodali]

it's happening again

[Papierkorb]

Still no change. It's extremely slow to obtain any image from ghcr.io. I'm in Switzerland with fast Internet. Other image hosters, like docker hub, don't have this issue.

[phil-klein-edclub]

Today for us in the Virginia / Maryland area, it is happening as for everyone above, nice and quick pushes, and pulls get maybe 70kb/s