[windows][toolchain] Run non-executable Swift Runtime tests for aarch64 Android

[weliveindetail]

We cross-compile the Swift runtime libs for the Android SDKs in the Windows toolchain. This patch adds a build step that runs non-executable tests for them.

[weliveindetail]

@swift-ci please test Windows

[weliveindetail]

The build failed with an unrelated error and didn't reach the test stage. Investigating whether this is existing or due to my config adjustments for testing:

swift/Runtimes/Core/runtime/EnvironmentVariables.def:25:1: error: use of undeclared identifier 'parse_bool'
VARIABLE(SWIFT_DEBUG_ENABLE_METADATA_ALLOCATION_ITERATION, bool, false,
^
C:/Users/swift-ci/jenkins/workspace/swift-PR-windows/swift/Runtimes/Core/runtime/EnvironmentVariables.cpp:205:9: note: expanded from macro 'VARIABLE'
        parse_##type(#name, buffer, defaultValue);              \
        ^

[weliveindetail]

Rebased and skipped build of ExperimentalRuntime for Android for the moment. Let's see if that gets us to the test stage.

@swift-ci please test Windows

[weliveindetail]

The build failed because the NEW setting for CMake policy CMP0157 isn't compatible with Swift's legacy driver. This is a known issue. Let's try again with the OLD setting in the respective dependencies.

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test Windows

[weliveindetail]

With the CMake policy workaround we successfully avoided the CMP0157 issue. However, the build had test failures for Swift Windows and thus didn't reach the test stage for Android:

Failed Tests (2):
  Swift(windows-x86_64) :: Interop/Cxx/stdlib/use-std-span.swift
  Swift(windows-x86_64) :: Interop/Cxx/stdlib/use-std-string-view.swift

[weliveindetail]

@swift-ci please test Windows

[weliveindetail]

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test Windows

[weliveindetail]

The missing required modules error was fixed in tests for symbol-graph-extract, but there are 4 more cases in swift-synthesize-interface and swift-frontend use cases.

Overall pass-rate is now above 50% for android-aarch64

Total Discovered Tests: 11245
  Unsupported        : 4648 (41.33%)
  Passed             : 6016 (53.50%)
  Expectedly Failed  :   29 (0.26%)
  Failed             :  550 (4.89%)
  Unexpectedly Passed:    2 (0.02%)

[weliveindetail]

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test Windows

[weliveindetail]

We didn't reach the test phase due to an unrelated build error in swift-corelibs-foundation:

Sources\Foundation\NSCalendar.swift:45:13: error: switch must be exhaustive

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test Windows

[weliveindetail]

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test Windows

[weliveindetail]

Down to 187 failures

Testing Time: 961.20s

Total Discovered Tests: 11258
  Unsupported        : 4650 (41.30%)
  Passed             : 6390 (56.76%)
  Expectedly Failed  :   29 (0.26%)
  Failed             :  187 (1.66%)
  Unexpectedly Passed:    2 (0.02%)

[finagolfin]

Do you know why so many fewer tests are discovered on the Windows CI compared to the linux CI, around 8k less? I also see that 2k more tests are unsupported when cross-compiling for Android on the Windows CI, though that is true for the community Android CI which runs on linux also, ie almost 2k more are unsupported.

[weliveindetail]

Thanks for sharing your observation! After check-swift-all-only_non_executable had issues, because it loaded multiple lit.site.cfg files, e53973c was supposed to split off validation tests. But check-swift-only_non_executable never succeeded so that we didn't reach check-swift-validation-only_non_executable.

Checking again, it turns out that the latter includes the former and I switched to use that now (there is yet another target check-swift-only-validation-only_non_executable that only runs validation tests).

[weliveindetail]

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test Windows

[weliveindetail]

Yes, that worked. Getting closer now.

[finagolfin]

Nice work, that looks pretty similar to the community Android CI results, which are run on a linux host instead.

[weliveindetail]

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test

[weliveindetail]

Rebased, sorted out patches, trying again

swiftlang/swift-testing#944
swiftlang/swift-corelibs-foundation#5166
@swift-ci please test

[weliveindetail]

The issue on Windows remains. Let's track this down in a separate PR #79738

On Linux 3 tests failed. That's likely related to my changes:

Failed Tests (3):
  Swift(linux-x86_64) :: FixCode/fixits-apply-all.swift
  Swift(linux-x86_64) :: FixCode/fixits-if-else.swift
  Swift(linux-x86_64) :: FixCode/verify-fixits.swift

[weliveindetail]

Let's see if the above patch is effective as a workaround. Also, it appears that swift-testing has the same issue as swiftlang/swift-corelibs-foundation#5180 Let's see if we can reproduce it in CI.

@swift-ci please test Windows

[weliveindetail]

CI bot confused the ticket link with a PR link so checkout failed. Let's try again.

@swift-ci please test Windows

[weliveindetail]

Split off the target-specific substitutions changes into #79998 Once the test is green, I will remove them here together with the utility patches.

[weliveindetail]

Yet another unrelated error prevented the build from reaching the Android Swift Runtime tests..

When building swift-foundation with SPM for testing, we pull in a WinSDK header that defines 128-bit integers. Apparently, the (outdated?) Clang we ship can't process them. Error is:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\shared\intsafe.h:227:64: error: invalid suffix 'i128' on integer constant
 225 | #define DWORD64_MAX     0xffffffffffffffffui64
 226 | #define UINT64_MAX      0xffffffffffffffffui64
 227 | #define INT128_MAX      170141183460469231731687303715884105727i128
     |                                                                `- error: invalid suffix 'i128' on integer constant
 228 | #define UINT128_MAX     0xffffffffffffffffffffffffffffffffui128
 229 | 

The issue was discussed a while ago in https://forums.swift.org/t/128bit-types-on-windows/65013, but it appears that the resulting fix never landed https://reviews.llvm.org/D121497 @compnerd Do you remember if that was the last state of affairs here?

[weliveindetail]

Apparently a fix for this is in the works in LLVM upstream: llvm/llvm-project#130993

[weliveindetail]

swiftlang/swift-installer-scripts#400
swiftlang/llvm-project#10256
@swift-ci please test Windows

[weliveindetail]

swiftlang/swift-installer-scripts#400
@swift-ci please test Windows

[weliveindetail]

PR-test reached Android tests. It took 5hr 46min in total. Validation tests failed to resolve, because I missed this patch. Trying again.

swiftlang/swift-installer-scripts#400
@swift-ci please test Windows

[weliveindetail]

Much better now. PR-test took 5hr 38min in total.

Total Discovered Tests: 18647
  Unsupported        :  5474 (29.36%)
  Passed             : 12903 (69.20%)
  Expectedly Failed  :   113 (0.61%)
  Failed             :   149 (0.80%)
  Unexpectedly Passed:     8 (0.04%)

[weliveindetail]

@swift-ci please smoke test

[weliveindetail]

The x64 mainline bot is configured to run these tests as soon as they land. However, should wait until we got a successful build after #80007. In a next step we might want to enable them as pre-merge tests as well via #79997.

[weliveindetail]

@swift-ci please smoke test Linux

[shahmishal]

We are see test failures on nightly Windows bot:

https://ci-external.swift.org/job/swift-main-windows-toolchain/1155/

********************
Failed Tests (1):
  Swift(android-aarch64) :: ClangImporter/availability_custom_domains.swift

********************
Unexpectedly Passed Tests (3):
  Swift(android-aarch64) :: Driver/parseable_output.swift
  Swift(android-aarch64) :: Driver/parseable_output_unicode.swift
  Swift-validation(android-aarch64) :: compiler_crashers_fixed/28795-inprotocol-isrequirementsignaturecomputed-missing-signature.swift

[weliveindetail]

No, actually #80053 was effective and it's only the unexpectedly passing tests that fail the build. Apparently the test suite doesn't honour the LIT_FILTER_OUT environment variable.

I put a temporary workaround here #80148 and proposed a way forward here #80037