[Incident] issues with Library Manager indexer infrastructure

[rsora]

Dear Libraries maintainers,
Starting from Sep 15, 2021 12:46:05 PM UTC, we are experiencing issues, probably related to hardware degradation, with our Library Manager indexer nodes.

Processes affected by the incident:

• New libraries submissions PRs, even if merged, will experience delays in being available for installation via Library Manager (i.e., not listed in library_index.json
• New releases for already submitted libraries will experience delays in being available for installation via Library Manager (i.e., not listed in library_index.json

Processes not affected by the incident:

• IDE/CLI Library manager: library_index.json is normally served by our CDN with no service degradation whatsoever, so the IDE and the CLI Library Manager can fetch correctly the list of the available libraries
• New libraries submission requests: submission requests can be opened and they will be validated by the Github Actions bot

The Arduino team will keep you posted here on the incident resolution.

this is fine

[rsora]

Dear Libraries maintainers,

Being the Hardware degradation declared not recoverable by the team, we migrated the indexing process to a new cloud VM and reconfigured the whole indexing pipeline in a new environment.

Starting from Sep 17, 2021 17:38:18 UTC, we restarted the indexing and updated the Library Manager with the latest submissions and new releases for already accepted libraries.

We now consider the incident as resolved, the team is going to continue the monitoring for the next couple of days before declaring it as closed, in case no issues are reported/observed.

Sorry for the inconvenience and thank you for your patience.

Please don't hesitate to reach out to the team, commenting this issue if you are experiencing problems related to the Library Manager update.

Happy coding!

[rsora]

Dear Libraries maintainers,

Starting from Sep 20, 2021 09:06:00 UTC, we consider the incident as closed

Again, sorry for the inconvenience and thank you for your patience.

Please don't hesitate to reach out to the team, opening an issue if you are experiencing problems related to the Library Manager update.

Happy coding!

[khoih-prog]

I wonder if the same issues came back today. This ESP32_PWM library has been submitted 17 hrs ago, but still not updated to Library Manager.

Could you please check,

Best Regards,

[per1234]

@khoih-prog thanks for your report. I just now checked and found it is available for me in Library Manager:

So I think everything is well.
I think the job that adds the library releases to the index runs hourly at this time but I believe there will be some additional time required for the index to propagate through the CDN cache and the IDE also caches the Library Manager index locally. So even after you see the indexing happened in the logs there will still be some additional time before the newly indexed releases are shown in Library Manager. We give a conservative estimate of one day but I'm sure you've noticed that it is often significantly less than that.

[khoih-prog]

Thanks,

I just check and everything is back to normal, even another new library submitted recently.

Thanks a lot,

Best,