diff --git a/Documentation/ABI/TestContent.md b/Documentation/ABI/TestContent.md new file mode 100644 index 000000000..fd7b9f893 --- /dev/null +++ b/Documentation/ABI/TestContent.md @@ -0,0 +1,168 @@ +# Runtime-discoverable test content + + + +This document describes the format and location of test content that the testing +library emits at compile time and can discover at runtime. + +> [!WARNING] +> The content of this document is subject to change pending efforts to define a +> Swift-wide standard mechanism for runtime metadata emission and discovery. +> Treat the information in this document as experimental. + +## Basic format + +Swift Testing stores test content records in a dedicated platform-specific +section in built test products: + +| Platform | Binary Format | Section Name | +|-|:-:|-| +| macOS, iOS, watchOS, tvOS, visionOS | Mach-O | `__DATA_CONST,__swift5_tests` | +| Linux, FreeBSD, OpenBSD, Android | ELF | `swift5_tests` | +| WASI | WebAssembly | `swift5_tests` | +| Windows | PE/COFF | `.sw5test$B`[^windowsPadding] | + +[^windowsPadding]: On Windows, the Swift compiler [emits](https://github.com/swiftlang/swift/blob/main/stdlib/public/runtime/SwiftRT-COFF.cpp) + leading and trailing padding into this section, both zeroed and of size + `MemoryLayout.stride`. Code that walks this section must skip over this + padding. + +### Record layout + +Regardless of platform, all test content records created and discoverable by the +testing library have the following layout: + +```swift +typealias TestContentRecord = ( + kind: UInt32, + reserved1: UInt32, + accessor: (@convention(c) (_ outValue: UnsafeMutableRawPointer, _ hint: UnsafeRawPointer?) -> CBool)?, + context: UInt, + reserved2: UInt +) +``` + +This type has natural size, stride, and alignment. Its fields are native-endian. +If needed, this type can be represented in C as a structure: + +```c +struct SWTTestContentRecord { + uint32_t kind; + uint32_t reserved1; + bool (* _Nullable accessor)(void *outValue, const void *_Null_unspecified hint); + uintptr_t context; + uintptr_t reserved2; +}; +``` + +### Record content + +#### The kind field + +Each record's _kind_ determines how the record will be interpreted at runtime. A +record's kind is a 32-bit unsigned value. The following kinds are defined: + +| As Hexadecimal | As [FourCC](https://en.wikipedia.org/wiki/FourCC) | Interpretation | +|-:|:-:|-| +| `0x00000000` | – | Reserved (**do not use**) | +| `0x74657374` | `'test'` | Test or suite declaration | +| `0x65786974` | `'exit'` | Exit test | + + + +#### The accessor field + +The function `accessor` is a C function. When called, it initializes the memory +at its argument `outValue` to an instance of some Swift type and returns `true`, +or returns `false` if it could not generate the relevant content. On successful +return, the caller is responsible for deinitializing the memory at `outValue` +when done with it. + +If `accessor` is `nil`, the test content record is ignored. The testing library +may, in the future, define record kinds that do not provide an accessor function +(that is, they represent pure compile-time information only.) + +The second argument to this function, `hint`, is an optional input that can be +passed to help the accessor function determine if its corresponding test content +record matches what the caller is looking for. If the caller passes `nil` as the +`hint` argument, the accessor behaves as if it matched (that is, no additional +filtering is performed.) + +The concrete Swift type of the value written to `outValue` and the value pointed +to by `hint` depend on the kind of record: + +- For test or suite declarations (kind `0x74657374`), the accessor produces an + asynchronous Swift function that returns an instance of `Test`: + + ```swift + @Sendable () async -> Test + ``` + + This signature is not the signature of `accessor`, but of the Swift function + reference it writes to `outValue`. This level of indirection is necessary + because loading a test or suite declaration is an asynchronous operation, but + C functions cannot be `async`. + + Test content records of this kind do not specify a type for `hint`. Always + pass `nil`. + +- For exit test declarations (kind `0x65786974`), the accessor produces a + structure describing the exit test (of type `__ExitTest`.) + + Test content records of this kind accept a `hint` of type `SourceLocation`. + They only produce a result if they represent an exit test declared at the same + source location (or if the hint is `nil`.) + +#### The context field + +This field can be used by test content to store additional context for a test +content record that needs to be made available before the accessor is called: + +- For test or suite declarations (kind `0x74657374`), this field contains a bit + mask with the following flags currently defined: + + | Bit | Value | Description | + |-:|-:|-| + | `1 << 0` | `1` | This record contains a suite declaration | + | `1 << 1` | `2` | This record contains a parameterized test function declaration | + + Other bits are reserved for future use and must be set to `0`. + +- For exit test declarations (kind `0x65786974`), this field is reserved for + future use and must be set to `0`. + +#### The reserved1 and reserved2 fields + +These fields are reserved for future use. Always set them to `0`. + +## Third-party test content + +Testing tools may make use of the same storage and discovery mechanisms by +emitting their own test content records into the test record content section. + +Third-party test content should set the `kind` field to a unique value only used +by that tool, or used by that tool in collaboration with other compatible tools. +At runtime, Swift Testing ignores test content records with unrecognized `kind` +values. To reserve a new unique `kind` value, open a [GitHub issue](https://github.com/swiftlang/swift-testing/issues/new/choose) +against Swift Testing. + +The layout of third-party test content records must be compatible with that of +`TestContentRecord` as specified above. Third-party tools are ultimately +responsible for ensuring the values they emit into the test content section are +correctly aligned and have sufficient padding; failure to do so may render +downstream test code unusable. + + diff --git a/Package.swift b/Package.swift index a72f7086a..ab06f693d 100644 --- a/Package.swift +++ b/Package.swift @@ -67,7 +67,10 @@ let package = Package( "_Testing_CoreGraphics", "_Testing_Foundation", ], - swiftSettings: .packageSettings + swiftSettings: .packageSettings + [ + // For testing test content section discovery only + .enableExperimentalFeature("SymbolLinkageMarkers"), + ] ), .macro( diff --git a/Sources/Testing/CMakeLists.txt b/Sources/Testing/CMakeLists.txt index f7728ac49..f205561a8 100644 --- a/Sources/Testing/CMakeLists.txt +++ b/Sources/Testing/CMakeLists.txt @@ -81,10 +81,13 @@ add_library(Testing Support/Locked.swift Support/SystemError.swift Support/Versions.swift + Discovery.swift + Discovery+Platform.swift Test.ID.Selection.swift Test.ID.swift Test.swift Test+Discovery.swift + Test+Discovery+Legacy.swift Test+Macro.swift Traits/Bug.swift Traits/Comment.swift diff --git a/Sources/Testing/Discovery+Platform.swift b/Sources/Testing/Discovery+Platform.swift new file mode 100644 index 000000000..db5594703 --- /dev/null +++ b/Sources/Testing/Discovery+Platform.swift @@ -0,0 +1,217 @@ +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2023–2025 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// See https://swift.org/CONTRIBUTORS.txt for Swift project authors +// + +internal import _TestingInternals + +/// A structure describing the bounds of a Swift metadata section. +struct SectionBounds: Sendable { + /// The base address of the image containing the section, if known. + nonisolated(unsafe) var imageAddress: UnsafeRawPointer? + + /// The in-memory representation of the section. + nonisolated(unsafe) var buffer: UnsafeRawBufferPointer + + /// All test content section bounds found in the current process. + static var allTestContent: some RandomAccessCollection { + _testContentSectionBounds() + } +} + +#if !SWT_NO_DYNAMIC_LINKING +#if SWT_TARGET_OS_APPLE +// MARK: - Apple implementation + +/// An array containing all of the test content section bounds known to the +/// testing library. +private let _sectionBounds = Locked<[SectionBounds]>(rawValue: []) + +/// A call-once function that initializes `_sectionBounds` and starts listening +/// for loaded Mach headers. +private let _startCollectingSectionBounds: Void = { + // Ensure _sectionBounds is initialized before we touch libobjc or dyld. + _sectionBounds.withLock { sectionBounds in + sectionBounds.reserveCapacity(Int(_dyld_image_count())) + } + + func addSectionBounds(from mh: UnsafePointer) { +#if _pointerBitWidth(_64) + let mh = UnsafeRawPointer(mh).assumingMemoryBound(to: mach_header_64.self) +#endif + + // Ignore this Mach header if it is in the shared cache. On platforms that + // support it (Darwin), most system images are contained in this range. + // System images can be expected not to contain test declarations, so we + // don't need to walk them. + guard 0 == mh.pointee.flags & MH_DYLIB_IN_CACHE else { + return + } + + // If this image contains the Swift section we need, acquire the lock and + // store the section's bounds. + var size = CUnsignedLong(0) + if let start = getsectiondata(mh, "__DATA_CONST", "__swift5_tests", &size), size > 0 { + _sectionBounds.withLock { sectionBounds in + let buffer = UnsafeRawBufferPointer(start: start, count: Int(clamping: size)) + let sb = SectionBounds(imageAddress: mh, buffer: buffer) + sectionBounds.append(sb) + } + } + } + +#if _runtime(_ObjC) + objc_addLoadImageFunc { mh in + addSectionBounds(from: mh) + } +#else + _dyld_register_func_for_add_image { mh, _ in + addSectionBounds(from: mh) + } +#endif +}() + +/// The Apple-specific implementation of ``SectionBounds/all``. +/// +/// - Returns: An array of structures describing the bounds of all known test +/// content sections in the current process. +private func _testContentSectionBounds() -> [SectionBounds] { + _startCollectingSectionBounds + return _sectionBounds.rawValue +} + +#elseif os(Linux) || os(FreeBSD) || os(OpenBSD) || os(Android) +// MARK: - ELF implementation + +private import SwiftShims // For MetadataSections + +/// The ELF-specific implementation of ``SectionBounds/all``. +/// +/// - Returns: An array of structures describing the bounds of all known test +/// content sections in the current process. +private func _testContentSectionBounds() -> [SectionBounds] { + var result = [SectionBounds]() + + withUnsafeMutablePointer(to: &result) { result in + swift_enumerateAllMetadataSections({ sections, context in + let version = sections.load(as: UInt.self) + guard version >= 4 else { + // This structure is too old to contain the swift5_tests field. + return true + } + + let sections = sections.load(as: MetadataSections.self) + let result = context.assumingMemoryBound(to: [SectionBounds].self) + + let start = UnsafeRawPointer(bitPattern: sections.swift5_tests.start) + let size = Int(clamping: sections.swift5_tests.length) + if let start, size > 0 { + let buffer = UnsafeRawBufferPointer(start: start, count: size) + let sb = SectionBounds(imageAddress: sections.baseAddress, buffer: buffer) + result.pointee.append(sb) + } + + return true + }, result) + } + + return result +} + +#elseif os(Windows) +// MARK: - Windows implementation + +/// Find the section with the given name in the given module. +/// +/// - Parameters: +/// - sectionName: The name of the section to look for. Long section names are +/// not supported. +/// - hModule: The module to inspect. +/// +/// - Returns: A structure describing the given section, or `nil` if the section +/// could not be found. +private func _findSection(named sectionName: String, in hModule: HMODULE) -> SectionBounds? { + hModule.withNTHeader { ntHeader in + guard let ntHeader else { + return nil + } + + let sectionHeaders = UnsafeBufferPointer( + start: swt_IMAGE_FIRST_SECTION(ntHeader), + count: Int(clamping: max(0, ntHeader.pointee.FileHeader.NumberOfSections)) + ) + return sectionHeaders.lazy + .filter { sectionHeader in + // FIXME: Handle longer names ("/%u") from string table + withUnsafeBytes(of: sectionHeader.Name) { thisSectionName in + 0 == strncmp(sectionName, thisSectionName.baseAddress!, Int(IMAGE_SIZEOF_SHORT_NAME)) + } + }.compactMap { sectionHeader in + guard let virtualAddress = Int(exactly: sectionHeader.VirtualAddress), virtualAddress > 0 else { + return nil + } + + var buffer = UnsafeRawBufferPointer( + start: UnsafeRawPointer(hModule) + virtualAddress, + count: Int(clamping: min(max(0, sectionHeader.Misc.VirtualSize), max(0, sectionHeader.SizeOfRawData))) + ) + guard buffer.count > 2 * MemoryLayout.stride else { + return nil + } + + // Skip over the leading and trailing zeroed uintptr_t values. These + // values are always emitted by SwiftRT-COFF.cpp into all Swift images. +#if DEBUG + let firstPointerValue = buffer.baseAddress!.loadUnaligned(as: UInt.self) + assert(firstPointerValue == 0, "First pointer-width value in section '\(sectionName)' at \(buffer.baseAddress!) was expected to equal 0 (found \(firstPointerValue) instead)") + let lastPointerValue = ((buffer.baseAddress! + buffer.count) - MemoryLayout.stride).loadUnaligned(as: UInt.self) + assert(lastPointerValue == 0, "Last pointer-width value in section '\(sectionName)' at \(buffer.baseAddress!) was expected to equal 0 (found \(lastPointerValue) instead)") +#endif + buffer = UnsafeRawBufferPointer( + rebasing: buffer + .dropFirst(MemoryLayout.stride) + .dropLast(MemoryLayout.stride) + ) + + return SectionBounds(imageAddress: hModule, buffer: buffer) + }.first + } +} + +/// The Windows-specific implementation of ``SectionBounds/all``. +/// +/// - Returns: An array of structures describing the bounds of all known test +/// content sections in the current process. +private func _testContentSectionBounds() -> [SectionBounds] { + HMODULE.all.compactMap { _findSection(named: ".sw5test", in: $0) } +} +#else +/// The fallback implementation of ``SectionBounds/all`` for platforms that +/// support dynamic linking. +/// +/// - Returns: The empty array. +private func _testContentSectionBounds() -> [SectionBounds] { + #warning("Platform-specific implementation missing: Runtime test discovery unavailable (dynamic)") + return [] +} +#endif +#else +// MARK: - Statically-linked implementation + +/// The common implementation of ``SectionBounds/all`` for platforms that do not +/// support dynamic linking. +/// +/// - Returns: A structure describing the bounds of the test content section +/// contained in the same image as the testing library itself. +private func _testContentSectionBounds() -> CollectionOfOne { + let (sectionBegin, sectionEnd) = SWTTestContentSectionBounds + let buffer = UnsafeRawBufferPointer(start: n, count: max(0, sectionEnd - sectionBegin)) + let sb = SectionBounds(imageAddress: nil, buffer: buffer) + return CollectionOfOne(sb) +} +#endif diff --git a/Sources/Testing/Discovery.swift b/Sources/Testing/Discovery.swift new file mode 100644 index 000000000..b2fc7825c --- /dev/null +++ b/Sources/Testing/Discovery.swift @@ -0,0 +1,173 @@ +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2023–2025 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// See https://swift.org/CONTRIBUTORS.txt for Swift project authors +// + +private import _TestingInternals + +/// The content of a test content record. +/// +/// - Parameters: +/// - kind: The kind of this record. +/// - reserved1: Reserved for future use. +/// - accessor: A function which, when called, produces the test content. +/// - context: Kind-specific context for this record. +/// - reserved2: Reserved for future use. +/// +/// - Warning: This type is used to implement the `@Test` macro. Do not use it +/// directly. +public typealias __TestContentRecord = ( + kind: UInt32, + reserved1: UInt32, + accessor: (@convention(c) (_ outValue: UnsafeMutableRawPointer, _ hint: UnsafeRawPointer?) -> CBool)?, + context: UInt, + reserved2: UInt +) + +/// Resign any pointers in a test content record. +/// +/// - Parameters: +/// - record: The test content record to resign. +/// +/// - Returns: A copy of `record` with its pointers resigned. +/// +/// On platforms/architectures without pointer authentication, this function has +/// no effect. +private func _resign(_ record: __TestContentRecord) -> __TestContentRecord { + var record = record + record.accessor = record.accessor.map(swt_resign) + return record +} + +// MARK: - + +/// A protocol describing a type that can be stored as test content at compile +/// time and later discovered at runtime. +/// +/// This protocol is used to bring some Swift type safety to the ABI described +/// in `ABI/TestContent.md`. Refer to that document for more information about +/// this protocol's requirements. +/// +/// This protocol is not part of the public interface of the testing library. In +/// the future, we could make it public if we want to support runtime discovery +/// of test content by second- or third-party code. +protocol TestContent: ~Copyable { + /// The unique "kind" value associated with this type. + /// + /// The value of this property is reserved for each test content type. See + /// `ABI/TestContent.md` for a list of values and corresponding types. + static var testContentKind: UInt32 { get } + + /// The type of value returned by the test content accessor for this type. + /// + /// This type may or may not equal `Self` depending on the type's compile-time + /// and runtime requirements. If it does not equal `Self`, it should equal a + /// type whose instances can be converted to instances of `Self` (e.g. by + /// calling them if they are functions.) + associatedtype TestContentAccessorResult: ~Copyable + + /// A type of "hint" passed to ``discover(withHint:)`` to help the testing + /// library find the correct result. + /// + /// By default, this type equals `Never`, indicating that this type of test + /// content does not support hinting during discovery. + associatedtype TestContentAccessorHint: Sendable = Never +} + +extension TestContent where Self: ~Copyable { + /// Enumerate all test content records found in the given test content section + /// in the current process that match this ``TestContent`` type. + /// + /// - Parameters: + /// - sectionBounds: The bounds of the section to inspect. + /// + /// - Returns: A sequence of tuples. Each tuple contains an instance of + /// `__TestContentRecord` and the base address of the image containing that + /// test content record. Only test content records matching this + /// ``TestContent`` type's requirements are included in the sequence. + private static func _testContentRecords(in sectionBounds: SectionBounds) -> some Sequence<(imageAddress: UnsafeRawPointer?, record: __TestContentRecord)> { + sectionBounds.buffer.withMemoryRebound(to: __TestContentRecord.self) { records in + records.lazy + .filter { $0.kind == testContentKind } + .map(_resign) + .map { (sectionBounds.imageAddress, $0) } + } + } + + /// Call the given accessor function. + /// + /// - Parameters: + /// - accessor: The C accessor function of a test content record matching + /// this type. + /// - hint: A pointer to a kind-specific hint value. If not `nil`, this + /// value is passed to `accessor`, allowing that function to determine if + /// its record matches before initializing its out-result. + /// + /// - Returns: An instance of this type's accessor result or `nil` if an + /// instance could not be created (or if `hint` did not match.) + /// + /// The caller is responsible for ensuring that `accessor` corresponds to a + /// test content record of this type. + private static func _callAccessor(_ accessor: SWTTestContentAccessor, withHint hint: TestContentAccessorHint?) -> TestContentAccessorResult? { + withUnsafeTemporaryAllocation(of: TestContentAccessorResult.self, capacity: 1) { buffer in + let initialized = if let hint { + withUnsafePointer(to: hint) { hint in + accessor(buffer.baseAddress!, hint) + } + } else { + accessor(buffer.baseAddress!, nil) + } + guard initialized else { + return nil + } + return buffer.baseAddress!.move() + } + } + + /// The type of callback called by ``enumerateTestContent(withHint:_:)``. + /// + /// - Parameters: + /// - imageAddress: A pointer to the start of the image. This value is _not_ + /// equal to the value returned from `dlopen()`. On platforms that do not + /// support dynamic loading (and so do not have loadable images), the + /// value of this argument is unspecified. + /// - content: The value produced by the test content record's accessor. + /// - context: Context associated with `content`. The value of this argument + /// is dependent on the type of test content being enumerated. + /// - stop: An `inout` boolean variable indicating whether test content + /// enumeration should stop after the function returns. Set `stop` to + /// `true` to stop test content enumeration. + typealias TestContentEnumerator = (_ imageAddress: UnsafeRawPointer?, _ content: borrowing TestContentAccessorResult, _ context: UInt, _ stop: inout Bool) -> Void + + /// Enumerate all test content of this type known to Swift and found in the + /// current process. + /// + /// - Parameters: + /// - hint: An optional hint value. If not `nil`, this value is passed to + /// the accessor function of each test content record whose `kind` field + /// matches this type's ``testContentKind`` property. + /// - body: A function to invoke, once per matching test content record. + /// + /// This function uses a callback instead of producing a sequence because it + /// is used with move-only types (specifically ``ExitTest``) and + /// `Sequence.Element` must be copyable. + static func enumerateTestContent(withHint hint: TestContentAccessorHint? = nil, _ body: TestContentEnumerator) { + let testContentRecords = SectionBounds.allTestContent.lazy.flatMap(_testContentRecords(in:)) + + var stop = false + for (imageAddress, record) in testContentRecords { + if let accessor = record.accessor, let result = _callAccessor(accessor, withHint: hint) { + // Call the callback. + body(imageAddress, result, record.context, &stop) + if stop { + break + } + } + } + } +} diff --git a/Sources/Testing/ExitTests/ExitTest.swift b/Sources/Testing/ExitTests/ExitTest.swift index af7981297..01810a7ca 100644 --- a/Sources/Testing/ExitTests/ExitTest.swift +++ b/Sources/Testing/ExitTests/ExitTest.swift @@ -25,17 +25,50 @@ private import _TestingInternals /// A type describing an exit test. /// /// Instances of this type describe an exit test defined by the test author and -/// discovered or called at runtime. +/// discovered or called at runtime. Tools that implement custom exit test +/// handling will encounter instances of this type in two contexts: +/// +/// - When the current configuration's exit test handler, set with +/// ``Configuration/exitTestHandler``, is called; and +/// - When, in a child process, they need to look up the exit test to call. +/// +/// If you are writing tests, you don't usually need to interact directly with +/// an instance of this type. To create an exit test, use the +/// ``expect(exitsWith:_:sourceLocation:performing:)`` or +/// ``require(exitsWith:_:sourceLocation:performing:)`` macro. @_spi(Experimental) @_spi(ForToolsIntegrationOnly) #if SWT_NO_EXIT_TESTS @available(*, unavailable, message: "Exit tests are not available on this platform.") #endif -public struct ExitTest: Sendable, ~Copyable { +public typealias ExitTest = __ExitTest + +/// A type describing an exit test. +/// +/// - Warning: This type is used to implement the `#expect(exitsWith:)` macro. +/// Do not use it directly. Tools can use the SPI ``ExitTest`` typealias if +/// needed. +@_spi(Experimental) +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public struct __ExitTest: Sendable, ~Copyable { /// The expected exit condition of the exit test. + @_spi(ForToolsIntegrationOnly) public var expectedExitCondition: ExitCondition + /// The source location of the exit test. + /// + /// The source location is unique to each exit test and is consistent between + /// processes, so it can be used to uniquely identify an exit test at runtime. + @_spi(ForToolsIntegrationOnly) + public var sourceLocation: SourceLocation + /// The body closure of the exit test. - fileprivate var body: @Sendable () async throws -> Void = {} + /// + /// Do not invoke this closure directly. Instead, invoke ``callAsFunction()`` + /// to run the exit test. Running the exit test will always terminate the + /// current process. + fileprivate var body: @Sendable () async throws -> Void /// Storage for ``observedValues``. /// @@ -72,16 +105,25 @@ public struct ExitTest: Sendable, ~Copyable { } } - /// The source location of the exit test. + /// Initialize an exit test at runtime. /// - /// The source location is unique to each exit test and is consistent between - /// processes, so it can be used to uniquely identify an exit test at runtime. - public var sourceLocation: SourceLocation + /// - Warning: This initializer is used to implement the `#expect(exitsWith:)` + /// macro. Do not use it directly. + public init( + __expectedExitCondition expectedExitCondition: ExitCondition, + sourceLocation: SourceLocation, + body: @escaping @Sendable () async throws -> Void = {} + ) { + self.expectedExitCondition = expectedExitCondition + self.sourceLocation = sourceLocation + self.body = body + } } #if !SWT_NO_EXIT_TESTS // MARK: - Invocation +@_spi(Experimental) @_spi(ForToolsIntegrationOnly) extension ExitTest { /// Disable crash reporting, crash logging, or core dumps for the current /// process. @@ -174,28 +216,17 @@ extension ExitTest { // MARK: - Discovery -/// A protocol describing a type that contains an exit test. -/// -/// - Warning: This protocol is used to implement the `#expect(exitsWith:)` -/// macro. Do not use it directly. -@_alwaysEmitConformanceMetadata -@_spi(Experimental) -public protocol __ExitTestContainer { - /// The expected exit condition of the exit test. - static var __expectedExitCondition: ExitCondition { get } - - /// The source location of the exit test. - static var __sourceLocation: SourceLocation { get } +extension ExitTest: TestContent { + static var testContentKind: UInt32 { + 0x65786974 + } - /// The body function of the exit test. - static var __body: @Sendable () async throws -> Void { get } + typealias TestContentAccessorResult = Self + typealias TestContentAccessorHint = SourceLocation } +@_spi(Experimental) @_spi(ForToolsIntegrationOnly) extension ExitTest { - /// A string that appears within all auto-generated types conforming to the - /// `__ExitTestContainer` protocol. - private static let _exitTestContainerTypeNameMagic = "__🟠$exit_test_body__" - /// Find the exit test function at the given source location. /// /// - Parameters: @@ -206,17 +237,34 @@ extension ExitTest { public static func find(at sourceLocation: SourceLocation) -> Self? { var result: Self? - enumerateTypes(withNamesContaining: _exitTestContainerTypeNameMagic) { _, type, stop in - if let type = type as? any __ExitTestContainer.Type, type.__sourceLocation == sourceLocation { + enumerateTestContent(withHint: sourceLocation) { _, exitTest, _, stop in + if exitTest.sourceLocation == sourceLocation { result = ExitTest( - expectedExitCondition: type.__expectedExitCondition, - body: type.__body, - sourceLocation: type.__sourceLocation + __expectedExitCondition: exitTest.expectedExitCondition, + sourceLocation: exitTest.sourceLocation, + body: exitTest.body ) stop = true } } + if result == nil { + // Call the legacy lookup function that discovers tests embedded in types. + enumerateTypes(withNamesContaining: exitTestContainerTypeNameMagic) { _, type, stop in + guard let type = type as? any __ExitTestContainer.Type else { + return + } + if type.__sourceLocation == sourceLocation { + result = ExitTest( + __expectedExitCondition: type.__expectedExitCondition, + sourceLocation: type.__sourceLocation, + body: type.__body + ) + stop = true + } + } + } + return result } } @@ -259,7 +307,7 @@ func callExitTest( var result: ExitTestArtifacts do { - var exitTest = ExitTest(expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation) + var exitTest = ExitTest(__expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation) exitTest.observedValues = observedValues result = try await configuration.exitTestHandler(exitTest) @@ -312,6 +360,7 @@ func callExitTest( // MARK: - SwiftPM/tools integration +@_spi(Experimental) @_spi(ForToolsIntegrationOnly) extension ExitTest { /// A handler that is invoked when an exit test starts. /// diff --git a/Sources/Testing/Support/Additions/WinSDKAdditions.swift b/Sources/Testing/Support/Additions/WinSDKAdditions.swift index 9b902c5d1..18d08bfcd 100644 --- a/Sources/Testing/Support/Additions/WinSDKAdditions.swift +++ b/Sources/Testing/Support/Additions/WinSDKAdditions.swift @@ -101,5 +101,32 @@ extension HMODULE { return nil } } + + /// Get the NT header corresponding to this module. + /// + /// - Parameters: + /// - body: The function to invoke. A pointer to the module's NT header is + /// passed to this function, or `nil` if it could not be found. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`. + func withNTHeader(_ body: (UnsafePointer?) throws -> R) rethrows -> R { + // Get the DOS header (to which the HMODULE directly points, conveniently!) + // and check it's sufficiently valid for us to walk. The DOS header then + // tells us where to find the NT header. + try withMemoryRebound(to: IMAGE_DOS_HEADER.self, capacity: 1) { dosHeader in + guard dosHeader.pointee.e_magic == IMAGE_DOS_SIGNATURE, + let e_lfanew = Int(exactly: dosHeader.pointee.e_lfanew), e_lfanew > 0 else { + return try body(nil) + } + + let ntHeader = (UnsafeRawPointer(dosHeader) + e_lfanew).assumingMemoryBound(to: IMAGE_NT_HEADERS.self) + guard ntHeader.pointee.Signature == IMAGE_NT_SIGNATURE else { + return try body(nil) + } + return try body(ntHeader) + } + } } #endif diff --git a/Sources/Testing/Test+Discovery+Legacy.swift b/Sources/Testing/Test+Discovery+Legacy.swift new file mode 100644 index 000000000..746c4128f --- /dev/null +++ b/Sources/Testing/Test+Discovery+Legacy.swift @@ -0,0 +1,81 @@ +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2023 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// See https://swift.org/CONTRIBUTORS.txt for Swift project authors +// + +private import _TestingInternals + +/// A protocol describing a type that contains tests. +/// +/// - Warning: This protocol is used to implement the `@Test` macro. Do not use +/// it directly. +@_alwaysEmitConformanceMetadata +public protocol __TestContainer { + /// The set of tests contained by this type. + static var __tests: [Test] { get async } +} + +/// A string that appears within all auto-generated types conforming to the +/// `__TestContainer` protocol. +let testContainerTypeNameMagic = "__🟠$test_container__" + +/// A protocol describing a type that contains an exit test. +/// +/// - Warning: This protocol is used to implement the `#expect(exitsWith:)` +/// macro. Do not use it directly. +@_alwaysEmitConformanceMetadata +@_spi(Experimental) +public protocol __ExitTestContainer { + /// The expected exit condition of the exit test. + static var __expectedExitCondition: ExitCondition { get } + + /// The source location of the exit test. + static var __sourceLocation: SourceLocation { get } + + /// The body function of the exit test. + static var __body: @Sendable () async throws -> Void { get } +} + +/// A string that appears within all auto-generated types conforming to the +/// `__ExitTestContainer` protocol. +let exitTestContainerTypeNameMagic = "__🟠$exit_test_body__" + +// MARK: - + +/// The type of callback called by ``enumerateTypes(withNamesContaining:_:)``. +/// +/// - Parameters: +/// - imageAddress: A pointer to the start of the image. This value is _not_ +/// equal to the value returned from `dlopen()`. On platforms that do not +/// support dynamic loading (and so do not have loadable images), this +/// argument is unspecified. +/// - type: A Swift type. +/// - stop: An `inout` boolean variable indicating whether type enumeration +/// should stop after the function returns. Set `stop` to `true` to stop +/// type enumeration. +typealias TypeEnumerator = (_ imageAddress: UnsafeRawPointer?, _ type: Any.Type, _ stop: inout Bool) -> Void + +/// Enumerate all types known to Swift found in the current process whose names +/// contain a given substring. +/// +/// - Parameters: +/// - nameSubstring: A string which the names of matching classes all contain. +/// - body: A function to invoke, once per matching type. +func enumerateTypes(withNamesContaining nameSubstring: String, _ typeEnumerator: TypeEnumerator) { + withoutActuallyEscaping(typeEnumerator) { typeEnumerator in + withUnsafePointer(to: typeEnumerator) { context in + swt_enumerateTypes(withNamesContaining: nameSubstring, .init(mutating: context)) { imageAddress, type, stop, context in + let typeEnumerator = context!.load(as: TypeEnumerator.self) + let type = unsafeBitCast(type, to: Any.Type.self) + var stop2 = false + typeEnumerator(imageAddress, type, &stop2) + stop.pointee = stop2 + } + } + } +} diff --git a/Sources/Testing/Test+Discovery.swift b/Sources/Testing/Test+Discovery.swift index 389d4cc92..9a187c917 100644 --- a/Sources/Testing/Test+Discovery.swift +++ b/Sources/Testing/Test+Discovery.swift @@ -1,7 +1,7 @@ // // This source file is part of the Swift.org open source project // -// Copyright (c) 2023 Apple Inc. and the Swift project authors +// Copyright (c) 2023–2025 Apple Inc. and the Swift project authors // Licensed under Apache License v2.0 with Runtime Library Exception // // See https://swift.org/LICENSE.txt for license information @@ -10,71 +10,73 @@ private import _TestingInternals -/// A protocol describing a type that contains tests. -/// -/// - Warning: This protocol is used to implement the `@Test` macro. Do not use -/// it directly. -@_alwaysEmitConformanceMetadata -public protocol __TestContainer { - /// The set of tests contained by this type. - static var __tests: [Test] { get async } -} +extension Test: TestContent { + static var testContentKind: UInt32 { + 0x74657374 + } -extension Test { - /// A string that appears within all auto-generated types conforming to the - /// `__TestContainer` protocol. - private static let _testContainerTypeNameMagic = "__🟠$test_container__" + typealias TestContentAccessorResult = @Sendable () async -> Self /// All available ``Test`` instances in the process, according to the runtime. /// /// The order of values in this sequence is unspecified. static var all: some Sequence { get async { - await withTaskGroup(of: [Self].self) { taskGroup in - enumerateTypes(withNamesContaining: _testContainerTypeNameMagic) { _, type, _ in - if let type = type as? any __TestContainer.Type { - taskGroup.addTask { - await type.__tests - } - } - } + var generators = [@Sendable () async -> [Self]]() - return await taskGroup.reduce(into: [], +=) + // Figure out which discovery mechanism to use. By default, we'll use both + // the legacy and new mechanisms, but we can set an environment variable + // to explicitly select one or the other. When we remove legacy support, + // we can also remove this enumeration and environment variable check. + enum DiscoveryMode { + case tryBoth + case newOnly + case legacyOnly + } + let discoveryMode: DiscoveryMode = switch Environment.flag(named: "SWT_USE_LEGACY_TEST_DISCOVERY") { + case .none: + .tryBoth + case .some(true): + .legacyOnly + case .some(false): + .newOnly } - } - } -} -// MARK: - + // Walk all test content and gather generator functions. Note we don't + // actually call the generators yet because enumerating test content may + // involve holding some internal lock such as the ones in libobjc or + // dl_iterate_phdr(), and we don't want to accidentally deadlock if the + // user code we call ends up loading another image. + if discoveryMode != .legacyOnly { + enumerateTestContent { imageAddress, generator, _, _ in + generators.append { @Sendable in + await [generator()] + } + } + } -/// The type of callback called by ``enumerateTypes(withNamesContaining:_:)``. -/// -/// - Parameters: -/// - imageAddress: A pointer to the start of the image. This value is _not_ -/// equal to the value returned from `dlopen()`. On platforms that do not -/// support dynamic loading (and so do not have loadable images), this -/// argument is unspecified. -/// - type: A Swift type. -/// - stop: An `inout` boolean variable indicating whether type enumeration -/// should stop after the function returns. Set `stop` to `true` to stop -/// type enumeration. -typealias TypeEnumerator = (_ imageAddress: UnsafeRawPointer?, _ type: Any.Type, _ stop: inout Bool) -> Void + if discoveryMode != .newOnly && generators.isEmpty { + enumerateTypes(withNamesContaining: testContainerTypeNameMagic) { imageAddress, type, _ in + guard let type = type as? any __TestContainer.Type else { + return + } + generators.append { @Sendable in + await type.__tests + } + } + } -/// Enumerate all types known to Swift found in the current process whose names -/// contain a given substring. -/// -/// - Parameters: -/// - nameSubstring: A string which the names of matching classes all contain. -/// - body: A function to invoke, once per matching type. -func enumerateTypes(withNamesContaining nameSubstring: String, _ typeEnumerator: TypeEnumerator) { - withoutActuallyEscaping(typeEnumerator) { typeEnumerator in - withUnsafePointer(to: typeEnumerator) { context in - swt_enumerateTypes(withNamesContaining: nameSubstring, .init(mutating: context)) { imageAddress, type, stop, context in - let typeEnumerator = context!.load(as: TypeEnumerator.self) - let type = unsafeBitCast(type, to: Any.Type.self) - var stop2 = false - typeEnumerator(imageAddress, type, &stop2) - stop.pointee = stop2 + // *Now* we call all the generators and return their results. + // Reduce into a set rather than an array to deduplicate tests that were + // generated multiple times (e.g. from multiple discovery modes or from + // defective test records.) + return await withTaskGroup(of: [Self].self) { taskGroup in + for generator in generators { + taskGroup.addTask { + await generator() + } + } + return await taskGroup.reduce(into: Set()) { $0.formUnion($1) } } } } diff --git a/Sources/_TestingInternals/Discovery.cpp b/Sources/_TestingInternals/Discovery.cpp index baf4ebd90..8af5e1690 100644 --- a/Sources/_TestingInternals/Discovery.cpp +++ b/Sources/_TestingInternals/Discovery.cpp @@ -10,6 +10,31 @@ #include "Discovery.h" +#if defined(SWT_NO_DYNAMIC_LINKING) +#pragma mark - Statically-linked section bounds + +#if defined(__APPLE__) +extern "C" const char testContentSectionBegin __asm("section$start$__DATA_CONST$__swift5_tests"); +extern "C" const char testContentSectionEnd __asm("section$end$__DATA_CONST$__swift5_tests"); +#elif defined(__wasi__) +extern "C" const char testContentSectionBegin __asm__("__start_swift5_tests"); +extern "C" const char testContentSectionEnd __asm__("__stop_swift5_tests"); +#else +#warning Platform-specific implementation missing: Runtime test discovery unavailable (static) +static const char testContentSectionBegin = 0; +static const char& testContentSectionEnd = testContentSectionBegin; +#endif + +/// The bounds of the test content section statically linked into the image +/// containing Swift Testing. +const void *_Nonnull const SWTTestContentSectionBounds[2] = { + &testContentSectionBegin, + &testContentSectionEnd +}; +#endif + +#pragma mark - Legacy test discovery + #include #include #include diff --git a/Sources/_TestingInternals/include/Discovery.h b/Sources/_TestingInternals/include/Discovery.h index d12f623ee..9d7a5a6e9 100644 --- a/Sources/_TestingInternals/include/Discovery.h +++ b/Sources/_TestingInternals/include/Discovery.h @@ -16,6 +16,62 @@ SWT_ASSUME_NONNULL_BEGIN +#pragma mark - Test content records + +/// The type of a test content accessor. +/// +/// - Parameters: +/// - outValue: On successful return, initialized to the value of the +/// represented test content record. +/// - hint: A hint value whose type and meaning depend on the type of test +/// record being accessed. +/// +/// - Returns: Whether or not the test record was initialized at `outValue`. If +/// this function returns `true`, the caller is responsible for deinitializing +/// the memory at `outValue` when done. +typedef bool (* SWTTestContentAccessor)(void *outValue, const void *_Null_unspecified hint); + +/// Resign an accessor function from a test content record. +/// +/// - Parameters: +/// - accessor: The accessor function to resign. +/// +/// - Returns: A resigned copy of `accessor` on platforms that use pointer +/// authentication, and an exact copy of `accessor` elsewhere. +/// +/// - Bug: This C function is needed because Apple's pointer authentication +/// intrinsics are not available in Swift. ([141465242](rdar://141465242)) +SWT_SWIFT_NAME(swt_resign(_:)) +static SWTTestContentAccessor swt_resignTestContentAccessor(SWTTestContentAccessor accessor) { +#if defined(__APPLE__) && __has_include() + accessor = ptrauth_strip(accessor, ptrauth_key_function_pointer); + accessor = ptrauth_sign_unauthenticated(accessor, ptrauth_key_function_pointer, 0); +#endif + return accessor; +} + +#if defined(__ELF__) && defined(__swift__) +/// A function exported by the Swift runtime that enumerates all metadata +/// sections loaded into the current process. +/// +/// This function is needed on ELF-based platforms because they do not preserve +/// section information that we can discover at runtime. +SWT_IMPORT_FROM_STDLIB void swift_enumerateAllMetadataSections( + bool (* body)(const void *sections, void *context), + void *context +); +#endif + +#if defined(SWT_NO_DYNAMIC_LINKING) +#pragma mark - Statically-linked section bounds + +/// The bounds of the test content section statically linked into the image +/// containing Swift Testing. +SWT_EXTERN const void *_Nonnull const SWTTestContentSectionBounds[2]; +#endif + +#pragma mark - Legacy test discovery + /// The type of callback called by `swt_enumerateTypes()`. /// /// - Parameters: diff --git a/Sources/_TestingInternals/include/Includes.h b/Sources/_TestingInternals/include/Includes.h index b1f4c7973..dfcbf50f0 100644 --- a/Sources/_TestingInternals/include/Includes.h +++ b/Sources/_TestingInternals/include/Includes.h @@ -127,6 +127,10 @@ #if !SWT_NO_LIBDISPATCH #include #endif + +#if __has_include() +#include +#endif #endif #if defined(__FreeBSD__) diff --git a/Sources/_TestingInternals/include/Stubs.h b/Sources/_TestingInternals/include/Stubs.h index caeb7c493..303cf0c46 100644 --- a/Sources/_TestingInternals/include/Stubs.h +++ b/Sources/_TestingInternals/include/Stubs.h @@ -91,6 +91,14 @@ static LANGID swt_MAKELANGID(int p, int s) { static DWORD_PTR swt_PROC_THREAD_ATTRIBUTE_HANDLE_LIST(void) { return PROC_THREAD_ATTRIBUTE_HANDLE_LIST; } + +/// Get the first section in an NT image. +/// +/// This function is provided because `IMAGE_FIRST_SECTION()` is a complex macro +/// and cannot be imported directly into Swift. +static const IMAGE_SECTION_HEADER *_Null_unspecified swt_IMAGE_FIRST_SECTION(const IMAGE_NT_HEADERS *ntHeader) { + return IMAGE_FIRST_SECTION(ntHeader); +} #endif #if defined(__linux__) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__ANDROID__) diff --git a/Tests/TestingTests/ABIEntryPointTests.swift b/Tests/TestingTests/ABIEntryPointTests.swift index f259fc8cf..86ede749e 100644 --- a/Tests/TestingTests/ABIEntryPointTests.swift +++ b/Tests/TestingTests/ABIEntryPointTests.swift @@ -52,6 +52,7 @@ struct ABIEntryPointTests { passing arguments: __CommandLineArguments_v0, recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void = { _ in } ) async throws -> CInt { +#if !SWT_NO_DYNAMIC_LINKING // Get the ABI entry point by dynamically looking it up at runtime. let copyABIEntryPoint_v0 = try withTestingLibraryImageAddress { testingLibrary in try #require( @@ -60,6 +61,7 @@ struct ABIEntryPointTests { } ) } +#endif let abiEntryPoint = copyABIEntryPoint_v0().assumingMemoryBound(to: ABIEntryPoint_v0.self) defer { abiEntryPoint.deinitialize(count: 1) diff --git a/Tests/TestingTests/MiscellaneousTests.swift b/Tests/TestingTests/MiscellaneousTests.swift index 02f2cc768..3c987a9ad 100644 --- a/Tests/TestingTests/MiscellaneousTests.swift +++ b/Tests/TestingTests/MiscellaneousTests.swift @@ -9,6 +9,7 @@ // @testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing +private import _TestingInternals @Test(/* name unspecified */ .hidden) @Sendable func freeSyncFunction() {} @@ -569,4 +570,77 @@ struct MiscellaneousTests { } #expect(duration < .seconds(1)) } + +#if !SWT_NO_DYNAMIC_LINKING && hasFeature(SymbolLinkageMarkers) + struct DiscoverableTestContent: TestContent { + typealias TestContentAccessorHint = UInt32 + typealias TestContentAccessorResult = UInt32 + + static var testContentKind: UInt32 { + record.kind + } + + static var expectedHint: TestContentAccessorHint { + 0x01020304 + } + + static var expectedResult: TestContentAccessorResult { + 0xCAFEF00D + } + + static var expectedContext: UInt { + record.context + } + +#if os(macOS) || os(iOS) || os(watchOS) || os(tvOS) || os(visionOS) + @_section("__DATA_CONST,__swift5_tests") +#elseif os(Linux) || os(FreeBSD) || os(OpenBSD) || os(Android) || os(WASI) + @_section("swift5_tests") +#elseif os(Windows) + @_section(".sw5test$B") +#endif + @_used + private static let record: __TestContentRecord = ( + 0xABCD1234, + 0, + { outValue, hint in + if let hint, hint.loadUnaligned(as: TestContentAccessorHint.self) != expectedHint { + return false + } + _ = outValue.initializeMemory(as: TestContentAccessorResult.self, to: expectedResult) + return true + }, + UInt(UInt64(0x0204060801030507) & UInt64(UInt.max)), + 0 + ) + } + + @Test func testDiscovery() async { + await confirmation("Can find a single test record") { found in + DiscoverableTestContent.enumerateTestContent { _, value, context, _ in + if value == DiscoverableTestContent.expectedResult && context == DiscoverableTestContent.expectedContext { + found() + } + } + } + + await confirmation("Can find a test record with matching hint") { found in + let hint = DiscoverableTestContent.expectedHint + DiscoverableTestContent.enumerateTestContent(withHint: hint) { _, value, context, _ in + if value == DiscoverableTestContent.expectedResult && context == DiscoverableTestContent.expectedContext { + found() + } + } + } + + await confirmation("Doesn't find a test record with a mismatched hint", expectedCount: 0) { found in + let hint = ~DiscoverableTestContent.expectedHint + DiscoverableTestContent.enumerateTestContent(withHint: hint) { _, value, context, _ in + if value == DiscoverableTestContent.expectedResult && context == DiscoverableTestContent.expectedContext { + found() + } + } + } + } +#endif }