Run stubtest on stubs on different platforms

[sobolevn]

Open problems:

• How to install deps on other platforms. brew works well on macos, but no idea how to use windows (I've never touched it since 2008)
• Listing platform specific packages. Which ones should we add?
• I still need to change daily.yml to do the same thing for all stubs
• How to test this? Not quite clear for now. I will need to tweak some values

Refs #8660

[sobolevn]

I have no idea how to fix this Windows problem:

Run STUBS=$(
ParserError: D:\a\_temp\4a20ead6-174f-4640-96c9-b47d87c97584.ps1:8
Line |
   8 |  if [ -n "$STUBS" ]; then
     |    ~
     | Missing '(' after 'if' in if statement.

It is valid sh, what more do you need? 🤣
Probably we need two different test runners: one for win and one for posix

[AlexWaygood]

It is valid sh, what more do you need? 🤣
Probably we need two different test runners: one for win and one for posix

The Windows shell is a whole different beast.

Can we use WSL in GitHub Actions? That might be an easy workaround.

[sobolevn]

Looks like it works! https://github.com/python/typeshed/actions/runs/3273273447/jobs/5385277526

[Avasam]

It is valid sh, what more do you need?

But not valid powershell ! Although you already figured out that you can specify the shell in github actions. Which is what I would've recommended over:

Can we use WSL in GitHub Actions? That might be an easy workaround.

---

How to install deps on other platforms

Chocolatey is what I see most often on windows (at least when developing with Node and Python).
Brew for MacOS.

---

Listing platform specific packages. Which ones should we add?

Whatever's needed as you develop/test this I guess?  
Taking a look at stubtest_allowlist.txt files, and skip=true in METADATA.toml, seems to be a good starting point to get an idea of which stubs have os-specific needs.

[Akuli]

You can use shell: bash on Windows github actions.

[sobolevn]

I already do that, thanks!

[AlexWaygood]

Could you make this change to the METADATA.toml file for D3DShot as part of this PR, to test that the Windows part of this is working?

  [tool.stubtest]
- # The library only works on Windows; we currently only run stubtest on Ubuntu for third-party stubs in CI.
- # See #8660
- skip = true
+ platform = ["win32"]

[Akuli]

Nit: I would prefer if the metadata key was named platforms instead of platform.

[AlexWaygood]

You might have to add a @tests/requirements-stubtest.txt file to D3DShot, and pin Pillow to a specific revision, in order to get it to install.

[sobolevn]

Output from windows daily run: https://github.com/sobolevn/typeshed/actions/runs/3280905801/jobs/5402262235

[sobolevn]

Since we now have platform-specific stubtest failures: https://github.com/sobolevn/typeshed/actions/runs/3283716889/jobs/5408788907

I will introduce platform-specific allowlists. Something similar to what we have in stdlib/

[AlexWaygood]

Looks great to me! Thanks for all the work you've done on this!

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[JelleZijlstra]

Slight wording suggestion because many stubs may be "supported" on other platforms, we just don't need to run stubtest multiple times.

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[AlexWaygood]

A few more tiny nits I spotted while looking over again

[sobolevn]

Suggested change

# TODO: re-enable

[AlexWaygood]

Wait, this TODO comment I did like -- it would be good if we could figure out how to enable stubtest for this package!

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[AlexWaygood]

Time for this to go in.

[Avasam]

@sobolevn & @AlexWaygood There's an unforseen issue: Now I can't run stubtest locally on any stub that doesn't explicitely specify my non-linux OS!

If no platform is specified in METADATA.toml, it should be allowed to be run on any OS, but the CI should only run linux.

[sobolevn]

I think we can add --local-run flag to be sure that it is indeed runing on a local env.

[Avasam]

Proposed solution: #9173