Skip to content

Latest commit

 

History

History
515 lines (427 loc) · 22.7 KB

0007-test-scoping-traits.md

File metadata and controls

515 lines (427 loc) · 22.7 KB

Test Scoping Traits

Note

This proposal was accepted before Swift Testing began using the Swift evolution review process. Its original identifier was SWT-0007.

Revision history

  • v1: Initial pitch.
  • v2: Dropped 'Custom' prefix from the proposed API names (although kept the word in certain documentation passages where it clarified behavior).
  • v3: Changed the Trait requirement from a property to a method which accepts the test and/or test case, and modify its default implementations such that custom behavior is either performed per-suite or per-test case by default.
  • v4: Renamed the APIs to use "scope" as the base verb instead of "execute".

Introduction

This introduces API which enables a Trait-conforming type to provide a custom execution scope for test functions and suites, including running code before or after them.

Motivation

One of the primary motivations for the trait system in Swift Testing, as described in the vision document, is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior, init and/or deinit (if applicable) can be used today. But if only some of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: the Trait protocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to.

Customizing a test's behavior typically means running code either before or after it runs, or both. Consolidating common set-up and tear-down logic allows each test function to be more succinct with less repetitive boilerplate so it can focus on what makes it unique.

Proposed solution

At a high level, this proposal entails adding API to the Trait protocol allowing a conforming type to opt-in to providing a custom execution scope for a test. We discuss how that capability should be exposed to trait types below.

Supporting scoped access

There are different approaches one could take to expose hooks for a trait to customize test behavior. To illustrate one of them, consider the following example of a @Test function with a custom trait whose purpose is to set mock API credentials for the duration of each test it's applied to:

@Test(.mockAPICredentials)
func example() {
  // ...
}

struct MockAPICredentialsTrait: TestTrait { ... }

extension Trait where Self == MockAPICredentialsTrait {
  static var mockAPICredentials: Self { ... }
}

In this hypothetical example, the current API credentials are stored via a static property on an APICredentials type which is part of the module being tested:

struct APICredentials {
  var apiKey: String

  static var shared: Self?
}

One way that this custom trait could customize the API credentials during each test is if the Trait protocol were to expose a pair of method requirements which were then called before and after the test, respectively:

public protocol Trait: Sendable {
  // ...
  func setUp() async throws
  func tearDown() async throws
}

extension Trait {
  // ...
  public func setUp() async throws { /* No-op */ }
  public func tearDown() async throws { /* No-op */ }
}

The custom trait type could adopt these using code such as the following:

extension MockAPICredentialsTrait {
  func setUp() {
    APICredentials.shared = .init(apiKey: "...")
  }

  func tearDown() {
    APICredentials.shared = nil
  }
}

Many testing systems use this pattern, including XCTest. However, this approach encourages the use of global mutable state such as the APICredentials.shared variable, and this limits the testing library's ability to parallelize test execution, which is another part of the Swift Testing vision.

The use of nonisolated static variables is generally discouraged now, and in Swift 6 the above APICredentials.shared property produces an error. One way to resolve that is to change it to a @TaskLocal variable, as this would be concurrency-safe and still allow tests accessing this state to run in parallel:

extension APICredentials {
  @TaskLocal static var current: Self?
}

Binding task local values requires using the scoped access TaskLocal.withValue() API though, and that would not be possible if Trait exposed separate methods like setUp() and tearDown().

For these reasons, I believe it's important to expose this trait capability using a single, scoped access-style API which accepts a closure. A simplified version of that idea might look like this:

public protocol Trait: Sendable {
  // ...

  // Simplified example, not the actual proposal
  func executeTest(_ body: @Sendable () async throws -> Void) async throws
}

extension MockAPICredentialsTrait {
  func executeTest(_ body: @Sendable () async throws -> Void) async throws {
    let mockCredentials = APICredentials(apiKey: "...")
    try await APICredentials.$current.withValue(mockCredentials) {
      try await body()
    }
  }
}

Avoiding unnecessarily lengthy backtraces

A scoped access-style API has some potential downsides. To apply this approach to a test function, the scoped call of a trait must wrap the invocation of that test function, and every other trait applied to that same test which offers custom behavior also must wrap the other traits' calls in a nesting fashion. To visualize this, imagine a test function with multiple traits:

@Test(.traitA, .traitB, .traitC)
func exampleTest() {
  // ...
}

If all three of those traits provide a custom scope for tests, then each of them needs to wrap the call to the next one, and the last trait needs to wrap the invocation of the test, illustrated by the following:

TraitA.executeTest {
  TraitB.executeTest {
    TraitC.executeTest {
      exampleTest()
    }
  }
}

Tests may have an arbitrary number of traits applied to them, including those inherited from containing suite types. A naïve implementation in which every trait is given the opportunity to customize test behavior by calling its scoped access API might cause unnecessarily lengthy backtraces that make debugging the body of tests more difficult. Or worse: if the number of traits is great enough, it could cause a stack overflow.

In practice, most traits probably do not need to provide a custom scope for the tests they're applied to, so to mitigate these downsides it's important that there be some way to distinguish traits which customize test behavior. That way, the testing library can limit these scoped access calls to only traits which need it.

Avoiding unnecessary (re-)execution

Traits can be applied to either test functions or suites, and traits applied to suites can optionally support inheritance by implementing the isRecursive property of the SuiteTrait protocol. When a trait is directly applied to a test function, if the trait customizes the behavior of tests it's applied to, it should be given the opportunity to perform its custom behavior once for every invocation of that test function. In particular, if the test function is parameterized and runs multiple times, then the trait applied to it should perform its custom behavior once for every invocation. This should not be surprising to users, since it's consistent with the behavior of init and deinit for an instance @Test method.

It may be useful for certain kinds of traits to perform custom logic once for all the invocations of a parameterized test. Although this should be possible, we believe it shouldn't be the default since it could lead to work being repeated multiple times needlessly, or unintentional state sharing across tests, unless the trait is implemented carefully to avoid those problems.

When a trait conforms to SuiteTrait and is applied to a suite, the question of when its custom scope (if any) should be applied is less obvious. Some suite traits support inheritance and are recursively applied to all the test functions they contain (including transitively, via sub-suites). Other suite traits don't support inheritance, and only affect the specific suite they're applied to. (It's also worth noting that a sub-suite can have the same non-recursive suite trait one of its ancestors has, as long as it's applied explicitly.)

As a general rule of thumb, we believe most traits will either want to perform custom logic once for all children or once for each child, not both. Therefore, when it comes to suite traits, the default behavior should depend on whether it supports inheritance: a recursive suite trait should by default perform custom logic before each test, and a non-recursive one per-suite. But the APIs should be flexible enough to support both, for advanced traits which need it.

Detailed design

I propose the following new APIs:

  • A new protocol TestScoping with a single required provideScope(...) method. This will be called to provide scope for a test, and allows the conforming type to perform custom logic before or after.
  • A new method scopeProvider(for:testCase:) on the Trait protocol whose result type is an Optional value of a type conforming to TestScoping. A nil value returned by this method will skip calling the provideScope(...) method.
  • A default implementation of Trait.scopeProvider(...) which returns nil.
  • A conditional implementation of Trait.scopeProvider(...) which returns self in the common case where the trait type conforms to TestScoping itself.

Since the scopeProvider(...) method's return type is optional and returns nil by default, the testing library cannot invoke the provideScope(...) method unless a trait customizes test behavior. This avoids the "unnecessarily lengthy backtraces" problem above.

Below are the proposed interfaces:

/// A protocol that allows providing a custom execution scope for a test
/// function (and each of its cases) or a test suite by performing custom code
/// before or after it runs.
///
/// Types conforming to this protocol may be used in conjunction with a
/// ``Trait``-conforming type by implementing the
/// ``Trait/scopeProvider(for:testCase:)-cjmg`` method, allowing custom traits
/// to provide custom scope for tests. Consolidating common set-up and tear-down
/// logic for tests which have similar needs allows each test function to be
/// more succinct with less repetitive boilerplate so it can focus on what makes
/// it unique.
public protocol TestScoping: Sendable {
  /// Provide custom execution scope for a function call which is related to the
  /// specified test and/or test case.
  ///
  /// - Parameters:
  ///   - test: The test under which `function` is being performed.
  ///   - testCase: The test case, if any, under which `function` is being
  ///     performed. When invoked on a suite, the value of this argument is
  ///     `nil`.
  ///   - function: The function to perform. If `test` represents a test suite,
  ///     this function encapsulates running all the tests in that suite. If
  ///     `test` represents a test function, this function is the body of that
  ///     test function (including all cases if it is parameterized.)
  ///
  /// - Throws: Whatever is thrown by `function`, or an error preventing this
  ///   type from providing a custom scope correctly. An error thrown from this
  ///   method is recorded as an issue associated with `test`. If an error is
  ///   thrown before `function` is called, the corresponding test will not run.
  ///
  /// When the testing library is preparing to run a test, it starts by finding
  /// all traits applied to that test, including those inherited from containing
  /// suites. It begins with inherited suite traits, sorting them
  /// outermost-to-innermost, and if the test is a function, it then adds all
  /// traits applied directly to that functions in the order they were applied
  /// (left-to-right). It then asks each trait for its scope provider (if any)
  /// by calling ``Trait/scopeProvider(for:testCase:)-cjmg``. Finally, it calls
  /// this method on all non-`nil` scope providers, giving each an opportunity
  /// to perform arbitrary work before or after invoking `function`.
  ///
  /// This method should either invoke `function` once before returning or throw
  /// an error if it is unable to provide a custom scope.
  ///
  /// Issues recorded by this method are associated with `test`.
  func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws
}

public protocol Trait: Sendable {
  // ...

  /// The type of the test scope provider for this trait.
  ///
  /// The default type is `Never`, which cannot be instantiated. The
  /// ``scopeProvider(for:testCase:)-cjmg`` method for any trait with this
  /// default type must return `nil`, meaning that trait will not provide a
  /// custom scope for the tests it's applied to.
  associatedtype TestScopeProvider: TestScoping = Never

  /// Get this trait's scope provider for the specified test and/or test case,
  /// if any.
  ///
  /// - Parameters:
  ///   - test: The test for which a scope provider is being requested.
  ///   - testCase: The test case for which a scope provider is being requested,
  ///     if any. When `test` represents a suite, the value of this argument is
  ///     `nil`.
  ///
  /// - Returns: A value conforming to ``Trait/TestScopeProvider`` which may be
  ///   used to provide custom scoping for `test` and/or `testCase`, or `nil` if
  ///   they should not have any custom scope.
  ///
  /// If this trait's type conforms to ``TestScoping``, the default value
  /// returned by this method depends on `test` and/or `testCase`:
  ///
  /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``.
  ///   If the value of this suite trait's ``SuiteTrait/isRecursive`` property
  ///   is `true`, then this method returns `nil`; otherwise, it returns `self`.
  ///   This means that by default, a suite trait will _either_ provide its
  ///   custom scope once for the entire suite, or once per-test function it
  ///   contains.
  /// - Otherwise `test` represents a test function. If `testCase` is `nil`,
  ///   this method returns `nil`; otherwise, it returns `self`. This means that
  ///   by default, a trait which is applied to or inherited by a test function
  ///   will provide its custom scope once for each of that function's cases.
  ///
  /// A trait may explicitly implement this method to further customize the
  /// default behaviors above. For example, if a trait should provide custom
  /// test scope both once per-suite and once per-test function in that suite,
  /// it may implement the method and return a non-`nil` scope provider under
  /// those conditions.
  ///
  /// A trait may also implement this method and return `nil` if it determines
  /// that it does not need to provide a custom scope for a particular test at
  /// runtime, even if the test has the trait applied. This can improve
  /// performance and make diagnostics clearer by avoiding an unnecessary call
  /// to ``TestScoping/provideScope(for:testCase:performing:)``.
  ///
  /// If this trait's type does not conform to ``TestScoping`` and its
  /// associated ``Trait/TestScopeProvider`` type is the default `Never`, then
  /// this method returns `nil` by default. This means that instances of this
  /// trait will not provide a custom scope for tests to which they're applied.
  func scopeProvider(for test: Test, testCase: Test.Case?) -> TestScopeProvider?
}

extension Trait where Self: TestScoping {
  // Returns `nil` if `testCase` is `nil`, else `self`.
  public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self?
}

extension SuiteTrait where Self: TestScoping {
  // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`.
  // Otherwise, `test` is a function and this returns `nil` if `testCase` is
  // `nil`, else `self`.
  public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self?
}

extension Trait where TestScopeProvider == Never {
  // Returns `nil`.
  public func scopeProvider(for test: Test, testCase: Test.Case?) -> Never?
}

extension Never: TestScoping {}

Here is a complete example of the usage scenario described earlier, showcasing the proposed APIs:

@Test(.mockAPICredentials)
func example() {
  // ...validate API usage, referencing `APICredentials.current`...
}

struct MockAPICredentialsTrait: TestTrait, TestScoping {
  func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws {
    let mockCredentials = APICredentials(apiKey: "...")
    try await APICredentials.$current.withValue(mockCredentials) {
      try await function()
    }
  }
}

extension Trait where Self == MockAPICredentialsTrait {
  static var mockAPICredentials: Self {
    Self()
  }
}

Source compatibility

The proposed APIs are purely additive.

This proposal will replace the existing CustomExecutionTrait SPI, and after further refactoring we anticipate it will obsolete the need for the SPIAwareTrait SPI as well.

Integration with supporting tools

Although some built-in traits are relevant to supporting tools (such as SourceKit-LSP statically discovering .tags traits), custom test behaviors are only relevant within the test executable process while tests are running. We don't anticipate any particular need for this feature to integrate with supporting tools.

Future directions

Access to suite type instances

Some test authors have expressed interest in allowing custom traits to access the instance of a suite type for @Test instance methods, so the trait could inspect or mutate the instance. Currently, only instance-level members of a suite type (including init, deinit, and the test function itself) can access self, so this would grant traits applied to an instance test method access to the instance as well. This is certainly interesting, but poses several technical challenges that puts it out of scope of this proposal.

Convenience trait for setting task locals

Some reviewers of this proposal pointed out that the hypothetical usage example shown earlier involving setting a task local value while a test is executing will likely become a common use of these APIs. To streamline that pattern, it would be very natural to add a built-in trait type which facilitates this. I have prototyped this idea and plan to add it once this new trait functionality lands.

Alternatives considered

Separate set up & tear down methods on Trait

This idea was discussed in Supporting scoped access above, and as mentioned there, the primary problem with this approach is that it cannot be used with scoped access-style APIs, including (importantly) TaskLocal.withValue(). For that reason, it prevents using that common Swift concurrency technique and reduces the potential for test parallelization.

Add provideScope(...) directly to the Trait protocol

The proposed provideScope(...) method could be added as a requirement of the Trait protocol instead of being part of a separate TestScoping protocol, and it could have a default implementation which directly invokes the passed-in closure. But this approach would suffer from the lengthy backtrace problem described above.

Extend the Trait protocol

The original, experimental implementation of this feature included a protocol namedCustomExecutionTrait which extended Trait and had roughly the same method requirement as the TestScoping protocol proposed above. This design worked, provided scoped access, and avoided the lengthy backtrace problem.

After evaluating the design and usage of this SPI though, it seemed unfortunate to structure it as a sub-protocol of Trait because it means that the full capabilities of the trait system are spread across multiple protocols. In the proposed design, the ability to return a test scoping provider is exposed via the main Trait protocol, and it relies on an associated type to conditionally opt-in to custom test behavior. In other words, the proposed design expresses custom test behavior as just a capability that a trait may have, rather than a distinct sub-type of trait.

Also, the implementation of this approach within the testing library was not ideal as it required a conditional trait as? CustomExecutionTrait downcast at runtime, in contrast to the simpler and more performant Optional property of the proposed API.

API names

We first considered "execute" as the base verb for the proposed new concept, but felt this wasn't appropriate since these trait types are not "the executor" of tests, they merely customize behavior and provide scope(s) for tests to run within. Also, the term "executor" has prior art in Swift Concurrency, and although that word is used in other contexts too, it may be helpful to avoid potential confusion with concurrency executors.

We also considered "run" as the base verb for the proposed new concept instead of "execute", which would imply the names TestRunning, TestRunner, runner(for:testCase), and run(_:for:testCase:). The word "run" is used in many other contexts related to testing though, such as the Runner SPI type and more casually to refer to a run which occurred of a test, in the past tense, so overloading this term again may cause confusion.

Acknowledgments

Thanks to Dennis Weissmann for originally implementing this as SPI, and for helping promote its usefulness.

Thanks to Jonathan Grynspan for exploring ideas to refine the API, and considering alternatives to avoid unnecessarily long backtraces.

Thanks to Brandon Williams for feedback on the Forum pitch thread which ultimately led to the refinements described in the "Avoiding unnecessary (re-)execution" section.